Industry-Standard Benchmarks for Embedded Systems
EEMBC, an industry alliance, develops benchmarks to help system designers select the optimal processors and understand the performance and energy characteristics of their systems. EEMBC has benchmark suites targeting mobile devices (for phones and tablets), networking, ultra-low power microcontrollers, the Internet of Things (IoT), digital media, automotive, and other application areas. EEMBC also has benchmarks for general-purpose performance analysis including CoreMark, MultiBench (multicore), and FPMark (floating-point).

ULPMARK


An EEMBC Benchmark 

ULPMark™ FAQs

How do I submit my ULPMark results? What are the run rules?

Scores are submitted using the ULPMark host software's "Submit" function. Only license holders of the software may upload scores (this includes corporate and university licensees as well as EEMBC workgroup and board members). The following run-rules apply:

ULPMark-CP and ULPMark-PP

  1. The firmware must be ported to the device under test. Refer to the porting instructions in the respective user manuals. Note that some parts of the firmware are not allowed to be modified.
  2. The device under test must be powered by the official EEMBC Energy Monitor.
  3. Ambient temperature during execution must be higher than 21C.
  4. The device must use a 32kHz external oscillator (+/-50ppm tol.). If scores are reported with an internal oscillator, they must also include a score uploaded with an external. Certification is only done with an external oscillator.
  5. A wake-up frequency of 1Hz needs to be ensured with a tolerance of max. 1% (considering device variances, and run rule ranges for temperature and voltage).

ULPMark-PP

  1. The host software will perform some basic sanity checks: ensuring each slot ran for one-second, and that there were no errors in the power trace (samples outside the range of the hardware).
  2. The ADC sampling frequency may be 1,000 or 1,024 Hz, whichever is easier to implement, with a +/-1% tolerance
  3. The 1MHz PWM must be >950kHz, and the period must be updated every 10ms (+/-5%)

Once a score is submitted, one of two things happen:

  1. If the score was submitted by someone who is not employed by the device manufacturer (aka "cross-posting") they must also include the source code. If the device manufacturer is a member of EEMBC, then an email will be sent to that member alerting them of the upload. The member then has two weeks to review the source code to verify it is a valid project, if not the score will not be published. Otherwise after the review the score will be published. If the device vendor is not a member, the score will be published within two weeks.
  2. If the score was submitted by the manufacturer than the score will be published within two weeks.

A score submitted by a vendor always supercedes a score submitted by anyone else, even if the vendor score is lower.

Certification

Certification is a process that exposes the benchmark to additional scrutiny: EEMBC certification labs review the source code and verify the scores are reproducable. Companies that request certification must also make the source code available internally to EEMBC workgroup members, this adds an additional layer of transparecy.

As a member, request certificaiton by clicking "Certify this result" when submitting a score. You must then send TWO typical boards to EEMBC along with the source code and the electrical hookup instructions. Board modification that require soldering are not allowed. EEMBC certificaiton labs will then review the source code port to make sure core benchmark code was not modified, validate the electrical connectivity & GPIO, verify the score by running multiple times, and then mark the score as certified on the website.

Am I allowed to use ULPMark scores in my literature?

The ULPMark licensee publishing rights are determined by the ULPMark workgroup, and may differe from other EEMBC benchmarks' publishing rules.

A licensee may not publish ULPMark scores publicly unless the scores are visible (submitted and published) on the EEMBC ULPMark public website. If the scores are on the public website, and they are not certified, that must be indicated in said literatuure. 

A licensee may share unpublished scores under NDA, but publication status must be stated, as must certification status.

What does typical really mean and does ULPMark address this?

Typical can refer to the workload. There are many ways a vendor might choose to define typical based on their customer's usage models. The ULPMark benchmark standardizes workload profiles to remove this ambiguity by explicitly defining the operations to measure, allowing comparisons between different vendor's products.

Typical can also refer to a statistic distribution in the CPU die manufacturing varation, and power can be very sensitive to these systemic and stochastic variations. The most robust way to define a typical variation would be to measure ULPMark on tens of thousands of die. This is obviously prohibitive. It is entirely possible that a licensee running ULPMark on their board will see a difference in the measured vs reported power numbers if they did not purchase a "typical" part. While it is impossible to prevent this (as it is the nature of semiconductor manufacturing), EEMBC works to reduce this by requiring that vendors supply "typical" material for testing and certification as measured by what they consider the majority of silicon, or the most likely silicon a consumer would receive when purchasing a device off-the-shelf, as opposed to "cherry picking" the best scoring die.

Why doesn't Windows detect the Energy Monitor?

Be sure to install the STMicroelectronics Virtual Com Port driver first. This is required for Windows (but not Linux or macOS).

Is ULPMark a benchmark for 8-, 16-, or 32-bit architectures?

The benchmark is defined under the premise that it is architecture independent.

How is the ULPMark calculated?

Since "bigger is always better" in benchmarking, the ULPMark benchmarks use a formula that takes the reciprocal of the energy values.

ULPMark-CP = 1000 / median(5 runs of the benchmark for 10 seconds)

ULPMark-PP = 10000 / median(5 runs of the benchmark's 10 slots, e.g. 10 seconds

From an energy perspective, what's the difference between running with an internal RC oscillator and an external crystal oscillator?

Generally speaking, a crystal oscillator requires a higher minimum drive current in order to oscillate due to the higher load capacitance as well as the external pin capacitance and resistances. An internal RC oscillator can have a very small capacitance and resistance and so needs very little drive current.

Does the ULPMark benchmark also calculate battery life?

No. The user would need to calculate this based on the power reported from the Host UI and the "mAh" rating of the battery under consideration.

When building the code for ULPMark, is it allowed to apply compiler optimizations, such as optimize for code size or speed?

Yes. Any optimizations will affect the energy usage.

Can I increase the sample frequency?

No. This is fixed to 1 kHz by the host software. While the Energy Monitor supports up to 10 kHz for energy measurements, this does not improve accuracy since the samples are cumulative. 1 kHz is a reasonable tradeoff between visual inspection resolution and dataset size.

What are the workload profiles for the ULPMark benchmarks?

The ULPMark-CP (CoreProfile) performs a variety of functions commonly found in ULP applications; among them are memory and math operations, sorting, GPIO interaction, etc.

The main function of the benchmark, which runs in an infinite loop, consists of:

  1. Toggle a GPIO indication port 20 times quickly ending with active indicator.
  2. Run a math_int16 filter and state_machine 24 times, using the cumulative result of the math functions as the input to the state function.
  3. Run an LCD conversion function using the output of the state function as input, with display size set to 8
  4. Run a string-search function on a string chosen based on the the previous step. Test the output value.
  5. Run a byte copy over the size of the chosen string.
  6. Run a bubblesort function on the result of the previous step, and test the result.
  7. Run a permute function for 16 permutes
  8. Indicate end of workload iteration using the GPIO active indicator
  9. Go to sleep and wake up in 1 second

Details on the functions:

  1. The filter function performs a simple filter using 16b math. A routine to filter away the DC content from an AC mains waveform signal. It does this using a heavily damped integrator to estimate the DC level. The current DC level is then subtracted from the signal.
  2. The LCD function will write values to the buffer that will represent LCD characters. The values written are based on the value of the bin parameter, and actual value to write is picked from the lcd_7segment_ascii_table.
  3. The permute function sets a particular bit. The value to modify in table is based on size_mask lower bits of the index. 4. The bit to modify is based on the 3 lower bits of the index. The bit is set if the value parameter is not 0, unset otherwise.
  4. The sort implements a simple bubble sort for 8b data.
  5. The state machine implements a simple state machine using a switch statement.

The ULPMark-PP activates one or more of the following four peripherals in one of 10 fixed configurations, known as slots.

  1. Real Time Clock (RTC)
  2. Analog-to-Digital conversion (ADC) at 1 Hz and 1 kHz
  3. Pulse-width Modulation (PWM) at 32 kHz and 1 MHz
  4. Serial Peripheral Interface (SPI)

Please refer to the About section (http://www.eembc.org/ulpbench/about.php) for more details of the ULPMark-PP benchmark's slot definitions.

What is the minimum current that can be measured?

100nA

Where can I purchase an Energy Monitor (EMON)?

The Energy Monitor is manufacture by STMicroelectronics, but EEMBC provides the benchmarking software. Refer to this link for ordering information:

http://www.st.com/en/evaluation-tools/x-nucleo-lpm01a.html

What is the voltage range of the Energy Monitor?

The energy monitor (EMON) can supply 1.8 to 3.3 Volts at a maximum of 50 mA.

What happens when the maximum energy monitor current is exceeded?

If the device under test draws more than 50 mA, the energy monitor status LED will turn red and the supply voltage will be disabled. You will need to reset the energy monitor.

Is CoreMark used in ULPMark?

Although we would like to use CoreMark, since it is an industry-accepted standard, the energy demand of CoreMark is far beyond the typical workload of an ultra-low power application. Hence, the ULPMark working group has developed a different set of workloads that are more relevant and real-world.

What is the roadmap for the evolution of the benchmark?

Phase 1 of the benchmark is called ULPMark-CP, for Core Profile, and is primarily focused on the active or run mode (code execution). Other phases currently in development, will cover more complex applications to test other functions, such as smart peripherals and various power modes of the microcontroller.

Phase 2 is called called ULPMark-PP, for Peripheral Profile, and its focus is on four common peripherals.

Future variations of the benchmark will focus on different low-power CPU and peripheral configurations.