Industry-Standard Benchmarks for Embedded Systems
EEMBC, an industry alliance, develops benchmarks to help system designers select the optimal processors and understand the performance and energy characteristics of their systems. EEMBC has benchmark suites targeting cloud and big data, mobile devices (for phones and tablets), networking, ultra-low power microcontrollers, the Internet of Things (IoT), digital media, automotive, and other application areas. EEMBC also has benchmarks for general-purpose performance analysis including CoreMark, MultiBench (multicore), and FPMark (floating-point).


An EEMBC Benchmark 


How do I submit my ULPMark results? What are the run rules?

EEMBC has a user interface, similar to what we provide for CoreMark, that allows you to submit your results, as well as other data that is pertinent to reproducing your benchmarking environment (i.e. device information, compiler details, timer configuration, active mode configuration, inactive/sleep mode configuration, operating conditions).

- ULPMark must be ported to the device under test. For information on porting, see examples under the Platforms folder in the source code distribution. Pay careful attention to the sections of ULPMark that can be modified for porting, and for Core Profile, avoid changing any code not under the platforms folder.
- The device under test must be powered by the Energy Monitor hardware as the benchmark is running to generate a valid result. For best results, ensure that only the DUT energy is being measured.
- Ambient temperature during execution must be higher than 21degC.
- Wakeup timer accuracy must be appropriate to the profile. For Core Profile, once per second, based on external 32KHz crystal with accuracy of 50ppm or better.
- I/O indicators must provide the output appropriate to the profile. For Core Profile, refer to the documentation.

What does typical really mean and does ULPMark address this?

There is a huge difference between typical and worst-case power and energy numbers that vendors publish. For example, you might find one vendor's MCU with very low sleep current but typical current is 2 orders of magnitude better than worst case of another vendor's MCU.Typical power doesn't mean anything because every vendor uses the 'whatever' mode. The beauty of the ULPMark, a standard benchmark methodology, is that all vendors measure in the same way. While ULPMark allows vendors to set it up for whatever mode gives them best energy consumption, we don't have to go into the details or arguments about the plethora of power modes.

Is ULPMark a benchmark for 8-, 16-, or 32-bit architectures?

The benchmark is defined under the premise that it is architecture independent.

How do I calculate the EEMBC ULPMark-CP?

Since "bigger is always better" in benchmarking, the ULPMark-CP uses a formula that takes the reciprocal of the energy values. ULPMark-CP = 1000/(median of 5 times average energy per second for 10 ULPMark cycles). The EnergyMonitor GUI also allows you to view the energy values (in µJoules).

From a energy perspective, what's the difference between running with an internal RC oscillator and an external crystal oscillator?

Generally speaking, a crystal oscillator requires a higher minimum drive current in order to oscillate due to the higher load capacitance as well as the external pin capacitance and resistances. An internal RC oscillator can have a very small capacitance and resistance and so needs very little drive current.

Does ULPMark also calculate battery life values (uAs, mAh)?

The user would calculate this for now based on the energy used in 1-second and what battery they are using. Right now, this feature is not there but it could be something we add into the GUI in the future.

When building the code for ULPMark, is it allowed to apply compiler optimizations, such as optimize for code size or speed?

Yes. Any optimizations will affect the energy usage.

Can I increase the sample frequency?

This is a fixed functionality and cannot be changed by the user. The sampling frequency changes dynamically based on the system load. Since this is a DC-DC based system, the frequency changes based on how often it needs to recharge the capacitor.

What workload does ULPMark perform?

The ULPMark-CoreProfile performs a variety of functions commonly found in ULP applications; among them are memory and math operations, sorting, GPIO interaction, etc.

The main function of the benchmark is as follows:
a. Blink an indication port 20 times quickly ending with active indicator.
b. Run the math_int8, math_int16, and state_machine 24 times, using the cumulative result of the math functions as the input to the state function.
c. Run the LCD function using the output of the state function as input, with display size set to 8
d. Run the StringSearch function on a string chosen based on the the previous step. Test the output value.
e. Run the BytesCopy for StringLength over the size of the chosen string.
f. Run the bubblesort function on the result of the previous step, and test the result.
g. Run the permute function for 16 permutes
h. Indicate end of workload iteration by calling clear_Indication_Port

1. math_int16; Approximation of a filter. Description: This function performs a simple filter using 16b math. A routine to filter away the DC content from an AC mains waveform signal. It does this using a heavily damped integrator to estimate the DC level. The current DC level is then subtracted from the signal.
2. LCD function will write values to the buffer that will represent LCD characters. The values written are based on the value of the bin parameter, and actual value to write is picked from the lcd_7segment_ascii_table.
3. The permute function sets a particular bit. The value to modify in table is based on size_mask lower bits of the index. 4. The bit to modify is based on the 3 lower bits of the index. The bit is set if the value parameter is not 0, unset otherwise.
5. The sort implements a simple bubble sort for 8b data.
6. The state machine implements a simple state machine using a switch statement.

What supply voltage does EnergyMonitor use?

The EnergyMonitor specify 3V for Phase 1 (ULPMark-Core Profile) but we plan on other voltage points for the next phase. Although MCUs will consume less energy at 1.8V, the goal is to produce a benchmark that allows apples-to-apples comparisons (we picked the lowest common demoninator).

ULPMark targets battery powered applications. These applications are typically powered by Lithium batteries which have a supply voltage of ~ 3V. There is some distribution around this:
- some start at 3.3 and go down to 2.7V
- some are at 3.0 from the beginning and very stable for a long time till they are discharged.
- Alkaline: 2 also give the 3V

What is the minimum current that can be measured?


What is the resolution of the measurement?

Avg. sampling frequency is 4Khz. No duty from 50:50. Varying duty cycle.
Error is 500nA or +/-2% 

What are the advanced configuration options for Threshold and Window?

In the EnergyMonitor GUI, the definitions for Threshold and Window are as follows:

Window is the window size to look for a “step” that indicates start of an active cycle. For example, if you have window of 200, the logic will start looking for active state as soon as 200 samples have been collected.
Threshold is the value above which active power is assumed, and no cycle detection is active.

Threshold - Setup cycle detection threshold. Set to 50 for ULPMark Core Profile.
Window - Setup cycle detection window size. Set to 200 for ULPMark Core Profile.

What is the accuracy of the EnergyMonitor?

The typical accuracy when using a stable USB supply is +/-1% for currents < 1mA, and +/-2% for currents > 1mA. We have determined that when using a direct connection to the USB port on a laptop, results can vary by as much as 5%.

Having a self-powered USB hub is the main requirement for stable results. While we don't endorse any specific brand, we have tested and proven this with two hubs (listed below), but we believe that any respectable quality USB hub that is self-powered is good enough.

1. D-Link Hi-Speed USB 2.0 4-Port Powered Hub (DUB-H4)
2. AmazonBasics 7 Port USB 3.0 Hub with 12V/3A power adapter

What are the test points on the EnergyMonitor board?

Other than Vcc and GND, we don’t release the schematics for EnergyMonitor.

How can I recalibrate the target voltage?

The voltage is automatically calibrated every time it captures. It uses onboard precision resistor dividers to measure the target output voltage. Measure discrepancies could be because of the voltage ripple from the DC-DC. It is within the +/-1% tolerance of the VCC. A lower VCC might appear during calibration where there is no load applied yet.

What is the behavior if the current exceeds 28mA due to inrush or other short duration overloads?

When an over-current occurs, the voltage output of EnergyMonitor starts dropping and will affect your energy measurements.

Will the ULPMark results be profiled over temperature?

Not for the first phase. Ultimately, the manufacturer should use it to qualify at various temperatures. For now, we only specify room temp (25C) operation. There's a hidden problem here - vendors publishing results could bin their parts to show lowest power at specific temperature range. But we expect that unbiased individuals worldwide will publish ULPMark results, which will help to sort out any binning issues.

Is CoreMark used in ULPMark?

Although we would like to use CoreMark, since it is an industry-accepted standard, the energy demand of CoreMark is far beyond the typical workload of an ultra-low power application. Hence, the ULPMark working group has developed a different set of workloads that are more relevant and real-world.

What is the roadmap for the evolution of ULPMark?

Phase 1 of ULPMark, called the Core Profile, is primarily focused on the active or run mode (code execution). Other phases currently in development, will cover more complex applications to test other functions, such as smart peripherals and various power modes of the microcontroller.

EnergyMonitor and Windows 8 and Windows 10

In order to use EnergyMonitor with Windows 8.1 or Windows 10, you'll need to disable driver signature verification; it's a very easy process.

The device should show up as an MSP device. If you go to Devices > Devices and Printers and right click on the MSP device and select "Properties".

Select one of the devices and open the properties of that device, then select the "Driver" tab (the same one since windows 95") and pick Update Driver. You will then have the "Browse my computer..." option, and then you can navigate to the driver that comes with the distribution. You will then be able to "Let me select..." and you will see EEMBC devices. Each of the two devices associated with the MSP device will associate with one of two EEMBC devices.