EEMBC's IoTMark-Wi-Fi benchmark examines the energy efficiency of embedded platforms supporting the 802.11 radio protocol, commonly known as Wi-Fi. It does this by emulating the real-world behavior seen by battery-operated wireless devices. The insight provided by this benchmark is crucial for not only evaluating claims of battery-life, but for understanding and optimizing systems so that they can last longer on a single charge.
A behavioral model describes what the benchmark actually does. The execution of the profile must be tightly controlled and completely repeatable for the benchmark to be meaningful. It must also be clearly defined so that measurements are comparible across different vendor's devices. IoTMark-Wi-Fi's behavioral model is split into three phases, two of which are used for scoring. The third is offered for analysis purposes.
|Connected Idle (DTIM)||Host AP:||802.11 core MAC/PHY - Network Model L1+L2||Yes|
|MQTT / Application||Host AP:||TCP/IP – Network model L3 + app||Yes|
|Connected active mode||PERF UDPTX (no provision for Rx implemented)||App||No|
Low-power devices spend a significant portion of their time in a mode called connected idle. This is a mode where the DUT (station) indicates to the Access Point (AP) that it is still connected, but will be entering a low-power state and will not be alert for every 802.11 beacon. Instead, the AP and the DUT agree on a sleep interval ahead of time, indicating precisely when the DUT will check to see if it needs to fully power-on. By using what is known as a Delivery Traffic Indication Map, or DTIM, the AP can tell the DUT to wake up and receive its queued data. The DUT DTIM check occurs every sleep interval of beacons. If there is data waiting, then the DUT powers-up to receive and process this data, re-entering connecterd standby after it has completed.
In order to minimize energy, several low-power communication protocols have been developed to operate over TCP/IP on 802.11. MQTT is one such protocol: it is a publication/subscribe model, where a central server is responsible for displatching incoming messages to subscribers. The benchmark uses an MQTT server installed on the AP to communicate with the DUT in a tightly-controlled manner. During this phase of the benchmark, the host system instructs the the DUT to perform both an Rx-Tx and an Rx-only transaction using MQTTS. MQTTS is a TLS enabled mode of MQTT, which means the data packets transmitted are encrypted with AES.
This portion of the benchmark exposes the energy costsd of high-bandwidth communication, and is not part of benchmark score. However, it does allow the user to investigate the real-world power of their device when sending large amounts of data via TCP or UDP. It utilizes an iperf server running on the AP.
What makes the benchmark effective is the EEMBC Benchmark Framework. The Framework is a combination of hardware, software, and firmware that controls when the device connects to the Access Point (AP), the connection parameters, and when the device sends and receives data using the MQTT communication protocol.
The hardware component of the Framework consists of the following:
The diagram below illustrates how these components are connected. The User Guide for the benchmark contains a BOM list for sourcing parts from DigiKey or Farnell.
The software component is a Host UI Runner application. It runs on the host PC and connects to the devices through the USB interface. From the control panel in the GUI, the user has the choice of running the benchmark using the official settings, or adjusting them to perform experiments and gain insight into their platform.
Since the goal of the benchmark is run the same behavioral profile on a variety of DUTs, tempate source-code is provided for interfacing with the framework. This template code allows the DUT to speak to the Host PC using a thin API. This API is implemented by a developer using their platform's Software Developer Kit (SDK). For example, the API provides a function called "th_wifi_connect", which takes an SSID and passkey. The developer then implements this function using their SDK. There are roughly a dozen functions in the API for this benchmark.Generic functions
The score of the benchmark is computed as follows (power in Watts):
IoTMark-Wi-Fi = 0.217 / (2/3 * Phase-1 Power + 1/3 * Phase-2 Power)
As the power decreases, the score increases. The scale factor adjusts the inverted power into a value somewhere between 10 and 2000. The scalar value 0.217 is roughly the conversion factor for two ideal AA alkaline battery's total charge. As a result of the multiplication, the score somewhat reflects "days of battery life," however real-world batteries vary considerably with discharge curve, heat, manufacturing, and other considerations. So it is not accurate to claim this as the true days of battery life, instead, it is simply a "ballpark" conversion.
Scores may be submitted by any licensee. The Host GUI Runner program that comes with the benchmark will enable score submission if all of the basic validation checks succeed after a given run completes.
In order to run the benchmark and submit scores, you must obtain a license. EEMBC is a US 501(c)6 non-profit, funded by membership dues and licensing fees. Please fill out an information request to obtain information on licensing.