EEMBC & CoreMark Blog

CoreMark run rules allow for any compiler optimizations. Unlike Dhrystone, CoreMark relies on a design that forces any computation to happen at compile time by tracing any computation chain from a value that is not available at compile time and ends with an output.

While compilers can find more efficient ways of implementing those computations, the computations cannot be done at compile time, and thus actual operation cannot be “optimized away”. Dhrystone for example was split to multiple files since the advent of the technique called inlining allowed compilers to avoid performing the very tasks the benchmark was trying to analyze. Since compilers can now analyze the program even when it is split to multiple files, this cure did not work…

In fact, many modern architectures rely on the compiler to create code that can make efficient use of hardware resources, and restricting the compiler from optimizing is not a reasonable restriction. CoreMark does not attempt to force a specific number of branches or loads, rather only that all computations are performed at run time. Complex architectures are going to find different ways of doing those computations (e.g. SIMD, predicated execution and more), while simple architectures are going to perform the operations head on. As long as the operation is performed, the benchmark is useful to analyze the performance potential of the core.

Recently Van Smith of Canalabs submitted several scores to the CoreMark website. We asked him about his choices for run parameters…

[NOTE: To put this blog into context, refer to the scores submitted on April 12 for  Intel Atom N450, VIA Nano L3050, AMD Mobile Athlon XP-M (Barton), and Freescale i.MX515 at http://coremark.org/benchmark/index.php?pg=benchmark)

Why did you use FORK rather then PTHREADS?

Answer: I used the same set of CoreMark compiler flags and settings as I had come up with several months  ago after experimenting on an AMD Phenom II system. However, these settings came directly from the best performing Intel system on CoreMark website. Of course, forking is the most reliable way of getting multicore scaling across a broad range of platforms, which is my goal. Threading can lead to headaches in those situations.

*NOTE from Shay Gal-On: CoreMark actually contains 3 separate methods to use concurrency. All of them have been thoroughly validated, so feel free to pick any method you wish and tell us why you picked it…

Why did you choose to oversubscribe the cores with 4 threads?

Answer: I wanted to use exactly the same settings across all platforms under test. The only platform in my ARM versus x86 report to benefit from forking was the Intel Atom; if all CPUs had been single-core without HyperThreading, I would have only used one thread.  As you know, threading/forking places greater pressure on  caches and memory and I wanted to keep this consistent across all systems under test.

* We like this approach, though what is a fair comparison? As mentioned in previous posts, CoreMark is not really focused on multiple cores. We highly recommend using EEMBC MultiBench when trying to evaluate performance in a MultiCore environment.

Why did you under-clock the N450 to 1GHz when it normally runs at 1.66GHz? Did it change the core/bus ratio?

Answer: The ARM Cortex-A8 system ran at 800MHz and I wanted to have all of the platforms operate at this speed for a fair IPC performance comparison.

Unfortunately, the Atom N450 can only be downclocked to 1GHz, so that’s what I was forced to use.  Only the multiplier was changed on the Atom and the VIA Nano L3050; adjusting the bus clock / system clock should always be avoided in performance benchmarking for many, many obvious reasons.

* For processors with even a small cache, CoreMark will operate entirely inside the cache, and will not be affected by memory performance at all. This is a critical concern for small embedded devices though, as mentioned in a previous blog post.

All in all, nice to see that industry analysts are picking up CoreMark and using it to test processors at any level. Check the news section to see the latest…

Two CoreMark scores for the TI Stellaris were submitted recently. It is interesting to note that while the only difference between the submissions is the frequency, the CoreMark/MHz has changed (1.9 at 50MHz vs. 1.6 at 80MHz; a 16% drop). Since the device does not have cache, the CPU frequency to memory frequency ratio may come into effect, and indeed we find that the flash used on the device can only scale 1:1 with the CPU frequency up to 50MHz. Once frequency goes above 50MHz, the memory frequency scales 1:2 with the CPU.

The memory to CPU frequency ratio is a common limitation, and various technological solutions are available. Cache is one answer, but expensive in terms of silicon area and resulting cost for the end product, especially critical in low-end microcontrollers. Other solutions may have wide reads (e.g. NXP ARM7 parts read 128 bits at a time) which will speed up execution of serial blocks of code, or more advanced techniques such as the “Enhanced Flash Memory Accelerator”  (see NXP LPC1759 CPU).

In general, performance will not increase linearly with frequency if the code and/or data the program needs resides in memory that cannot scale at the same ratio. This will be true in benchmarks and in ‘real’ life. Does it matter to your application?

Shay Gal-On always wins when it comes to wrestling with EEMBC benchmarks. But he’s no match for sumo wrestler.

http://www.youtube.com/watch?v=njBGUzAExo4

Although the current run rules for CoreMark don’t allow messing with the code, it’s an interesting exercise to see how much concurrency can be extracted from this relatively sequential benchmark code. CriticalBlue took this on with its Prism tool, writing up an interesting article on the process and results. Check it out

CoreMark includes 3 major algorithms looking at core characteristics:

1. List manipulation – pointers and data access through pointers.

2. Matrix manipulation – serial data access and potentially using ILP.

3. Simple state machine – FSM that excercises the branch unit in the pipeline.

The source code for each of those is in a single file. The other files take care of setting up the data for the benchmark, measuring the time, and spitting out a report with the results.

The platform specific code is in a special folder, and there are several sample ports for self hosted platforms using gcc – linux, linux64, cygwin and simple.

Check the documentation for more info, and check back next week for some interesting analysis…