The ULPMark benchmark line is now adding Machine Learning inference. The effort to standardize what is known as "tinyML" energy is underway, with a dozen companies participating in EEMBC's effort. The goal of this benchmark is to create a standardized suite of tasks that will measure a device's inference energy-efficiency as a single figure of merit. More details will follow as the benchmark develops, see the "Workgroup Status" below for updates and how to join the group.
Ultra-low power optimization is critical in devices that require a decade of battery life, or rely on energy harvesting solutions from the environment.Ultra-Low Power (ULP) describes a major design challenge MCUs face today, where product expectations range from running 10 years on a single battery, to harvesting pico-Joules of energy from the environment, to reducing overall global energy demand.
MCU vendors deploy creative design techniques to reduce the energy demands of a device in its different operating modes, such as deep sleep, peripheral data transfer, and active computation. Developers cannot rely on a single datasheet number to explain the benefits of these techniques because applications require optimization tradeoffs between all three energy demands, and performance.
The EEMBC ULPMark benchmark quantifies these tradeoffs by constructing behavioral profiles that capture device operation better than a single synthetic number. And by providing a concise methodology, ULPMark reliably and equitably measures the multiple aspects of MCU energy efficiency.
Since its inception in 2012, EEMBC's ULPMark Working Group continues to develop new profiles for MCU power and energy analysis. There are currently three profiles: ULPMark-CoreMark which focuses on performance and energy efficiency of an active-mode benchmark, ULPMark-PeripheralProfile which analyzes the energy cost of MCU peripherals, and the original ULPMark-CoreProfile which characterizes the sleep and wakeup energy of an MCU.
| ULPMark variant | What it measures |
|---|---|
| ULPMark-CoreProfile | True energy cost of deep-sleep modes. |
| ULPMark-PeripheralProfile | Common peripherals' energy impact on deep-sleep. |
| ULPMark-CoreMark | Active power, using CoreMark as the workload. |
| ULPMark-ML (TBD!) | Active power, using tiny Machine Learning workloads |
ULPMark-CP and ULPMark-PP both focus on sleepy-node energy efficiency which often illustrates the true cost of low power not captured by a datasheet. On the flip side of sleep power is active power: how much power does a low-power device use while doing useful work? Often designers must choose between energy efficiency and performance. This tradeoff isn't always clear, so ULPMark-CM enables this analysis by providing both measurements simultaneously. The ULPMark-CM score is the number of CoreMark iterations a device can execute per milli-Joule. This number, when presented with the CoreMark iterations-per-second score, illustrates how the two opposing metrics are related.
There are three components to every ULPMark-CM configuration: The ULPMark-CM energy-efficiency score, the voltage the score was collected at, and its CoreMark performance score. This triad of numbers succinctly defines an operating point on a performance-versus-energy curve. The benchmark defines three operating-point configurations that the developer must accommodate:
Unlike the original CoreMark performance benchmark that had a single instance of the functions, inside ULPMark-CM the developer provides multiple versions of the CoreMark library, each one optimized for different hardware conditions. This is very different from CoreMark, where the firmware operated at one voltage and frequency. The new ULPMark-CM firmware also allows the developer to provide extensive hardware reconfiguration during the benchmark, without having to disconnect the device or flash new firmware between measurements.
The host GUI supports a "Benchmark Mode", where the user measures the three operating points mentioned above and submit those scores to the EEMBC website, or in "Experiment Mode", where the user may explore any additional configurations supplied by the developer.
ULPMark-PP focuses on the MCU's commonly used peripherals like pulse-width modulation (PWM), analog-to-digital conversion (ADC), the serial peripheral interface (SPI), and a real-time clock (RTC). This benchmark defines ten one-second activity slots each with variable usage of ADC, SPI, PWM, RTC, allowing the MCU and peripherals to sleep after their activities have completed. The following table gives an overview of the activity in each slot. As soon as the device finishes the peripheral operation for that slot it can enter sleep. This means faster peripherals will most likely score higher since they can remain off longer.
Peripheral Profile Slot Descriptions
| Slot | ADC | PWM | SPI | RTC |
|---|---|---|---|---|
| 1 | | |||
| 2 | | |||
| 3 | | |||
| 4 | | |||
| 5 | | |||
| 6 | | |||
| 7 | | |||
| 8 | | |||
| 9 | | |||
| 10 |
The Core Profile focuses on the MCU’s core, specifically the energy cost in sleep, and the transition to and from active mode. This benchmark utilizes a common set of workloads that are portable across 8-, 16-, and 32-bit microcontrollers. The Core Profile runs on a one-second duty cycle combining these workloads with an extended period of inactivity to enable the use of microcontroller low-power modes. Please refer to the FAQ for more information on the active workload.
While the active portion of the benchmark is only running for ~3% of the total runtime, it requires data to be saved during deep-sleep through the use of "retention RAM". Since it is rare that a sleepy edge node would clear its RAM after every sleep cyle, the exit- and enter-costs of retention RAM illustrate the true energy cost of sleep modes. There's more to it than just a datasheet number!
During the active portion of the test, the benchmark does the following:
ULPMark has been redesigned since it's first release in 2014. It now works with the EEMBC IoTConnect™ benchmark framework, the same one used by IoTMark and SecureMark, with a super-thin API that enables any MCU to execute next generation EEMBC benchmarks. The STMicroelectronics PowerShield provides the backbone of the framework's energy measurement, with sub-100nJ accuracy on your desktop for around US$100.
The IoTConnect framework used for ULPMark creates an extensible framework for probing an embedded system. Block diagram (left), actual implementation (right). Click to enlarge. Note that for ULPMark-CP and ULPMark-PP, the Radio Manager and IO Manager are not required.
We are currently developing ULPMark-ML and are in the workload-definition phase.
Chairperson
All versions of ULPMark are available for corporate and academic licensing. As always, member companies and licensees are continually uploading new scores.
Join the working group to help ensure a meaningful and fair representation for your company’s products.
Copyright © EEMBC
Note: This website only works on browsers that support ES6, such as Edge, Chrome, Firefox, Safari; IE11 and earlier are not supported.
