IoTMark™-Wi-Fi

An EEMBC Benchmark
« IoTMark Home · About · Run Rules · View Scores · Submit a Score
This benchmark will be released in Q2'2021

Overview

EEMBC's IoTMark-Wi-Fi is a benchmark that examines the energy efficiency of embedded platforms supporting the 802.11 radio protocol, commonly known as Wi-Fi. It does this by emulating the real-world behavior seen by battery-operated wireless devices. The benchmark is targeted at devices that are expected to have months, or more, of battery-life and reasonably low Wi-Fi throughput demands.

The score produced by IoTMark-Wi-Fi is different from simple datasheet electrical characteristics, which may provide power numbers for each individual component, but not comprehensive assessment of the entire platofrm in a real usage scenario. In this way, the score is a reasonable comparitive indicator of a platform's energy-efficiency.

The benchmark was created over the course of 18 months by a team EEMBC member companies, including: Dialog Semiconductor, Silicon Labs (plus Redpines), Infineon (formerly Cypress), Texas Instruments, Altran, and STMicroelectronics. The Wi-Fi Alliance provided input as well. During this period, dozens of ideas and issues were discussed and debated by 802.11 industry experts and principal engineers. It was no small feat, requiring that competitors work together to develop an industry-standard solution.

Behavioral Model

A behavioral model describes what the benchmark actually does. The execution of the profile must be tightly controlled and completely repeatable for the benchmark to be meaningful. It must also be clearly defined so that measurements are comparible across different vendor's devices. IoTMark-Wi-Fi's behavioral model is split into three phases, two of which are used for scoring. The third is offered for analysis purposes.

PhaseConfigurationLayersScored
Connected Idle (DTIM)Host AP:
  • DTIM = 1
Station:
  • Sleep interval = 10 beacons;
  • L2 keep-alive = 30 sec
Measurement:
  • STA enters connected standby for 330 sec;
  • Record median power for 315 sec sliding window in 333 ms steps
802.11 core MAC/PHY - Network Model L1+L2Yes
MQTT / ApplicationHost AP:
  • DTIM = 1
Station:
  • Sleep interval = 10 beacons;
  • L2 keep-alive = 30 sec;
  • MQTT keep-alive = 60 sec
Measurement:
  • STA connect to MQTTS server on AP;
  • STA subscribes two topics “command” and “response” & enable echo based on format;
  • STA enters low-power state for 330 sec;
  • AP sends two messages: one Rx only, one Rx with Tx echo, all 32 B;
  • Record median power for 315 sec sliding window in 333 ms steps
TCP/IP – Network model L3 + appYes
Connected active modePERF UDPTX (no provision for Rx implemented)App
Final scoring phaseCombine scores from phase 1 & phase 2 (TBD)
Table 1. Three phases of the behavioral model.

Phase One: Connected Idle

Low-power devices spend a significant portion of their time in a mode called connected idle. This is a mode where the DUT (station) indicates to the Access Point (AP) that it is still connected, but will be entering a low-power state and will not be alert for every 802.11 beacon. Instead, the AP and the DUT agree on a sleep interval ahead of time, indicating precisely when the DUT will check to see if it needs to fully power-on. By using what is known as a Delivery Traffic Indication Map, or DTIM, the AP can tell the DUT to wake up and receive its queued data. The DUT DTIM check occurs every sleep interval of beacons. If there is data waiting, then the DUT powers-up to receive and process this data, re-entering connecterd standby after it has completed.

Phase Two: Application Communication

In order to minimize energy, several low-power communication protocols have been developed to operate over TCP/IP on 802.11. MQTT is one such protocol: it is a publication/subscribe model, where a central server is responsible for displatching incoming messages to subscribers. The benchmark uses an MQTT server installed on the AP to communicate with the DUT in a tightly-controlled manner. During this phase of the benchmark, the host system instructs the the DUT to perform both an Rx-Tx and an Rx-only transaction using MQTTS. MQTTS is a TLS enabled mode of MQTT, which means the data packets transmitted are encrypted with AES.

Phase Three: Connected Active Mode

This portion of the benchmark exposes the energy costsd of high-bandwidth communication, and is not part of benchmark score. However, it does allow the user to investigate the real-world power of their device when sending large amounts of data via TCP or UDP. It utilizes an iperf server running on the AP.

Framework

What makes the benchmark effective is the EEMBC Benchmark Framework. The Framework is a combination of hardware, software, and firmware that controls when the device connects to the Access Point (AP), the connection parameters, and when the device sends and receives data using the MQTT communication protocol.

Hardware

The hardware component of the Framework consists of the following:

  • A host PC to run the Framework software
  • The device under test (DUT)
  • The 802.11 Access Point (AP)
  • An energy monitor, which supplies and measures power to the DUT
  • An IO Manager that acts as an electrically-isolated proxy between the host PC and the DUT

The diagram below illustrates how these components are connected. The User Guide for the benchmark contains a BOM list for sourcing parts from DigiKey or Farnell.

Figure 1. The hardware components used in the IoTMark-Wi-Fi framework.

Software

The software component is a Host UI Runner application. It runs on the host PC and connects to the devices through the USB interface. From the control panel in the GUI, the user has the choice of running the benchmark using the official settings, or adjusting them to perform experiments and gain insight into their platform.

Figure 2. Benchmark control panel.
Figure 3. Zoomed-in energy chart for an MQTT transfer.
Firmware

Since the goal of the benchmark is run the same behavioral profile on a variety of DUTs, tempate source-code is provided for interfacing with the framework. This template code allows the DUT to speak to the Host PC using a thin API. This API is implemented by a developer using their platform's Software Developer Kit (SDK). For example, the API provides a function called "th_wifi_connect", which takes an SSID and passkey. The developer then implements this function using their SDK. There are roughly a dozen functions in the API for this benchmark.

Generic functions
  • th_command_ready
  • th_timestamp
  • th_mstandby_start
  • th_gettime
  • th_settime
802.11 Functions
  • th_async_wifi_connect
  • th_async_wifi_disconnect
  • th_async_wifi_scan
  • th_get_mac
  • th_get_sleep_interval
  • th_set_sleep_interval
  • th_get_keepalive_interval
  • th_set_keepalive_interval
MQTT Functions
  • th_async_mqtt_connect
  • th_async_mqtt_disconnect
  • th_async_mqtt_publish
  • th_async_mqtt_subscribe
  • th_async_mqtt_unsubscribe
  • th_mqtt_get_keepalive_interval
  • th_mqtt_set_keepalive_interval
iPerf Functions (optional)
  • th_async_iperf_start

Scoring

TBD - This benchmark is pre-release and this has not been voted on yet.

Submitting Scores

TBD - This benchmark is pre-release and this has not been voted on yet.

Obtaining the Benchmark

In order to run the benchmark and submit scores, you must obtain a license. EEMBC is a US 501(c)6 non-profit, funded by membership dues and licensing fees. Please fill out an information request to obtain information on licensing.


Copyright © EEMBC

Note: This website only works on browsers that support ES6, such as Edge, Chrome, Firefox, Safari; IE11 and earlier are not supported.