People often ask about the applicability of CoreMark for 8-, 16-, and 32-bit processors. They wonder if it provides a realistic measure of performance for an 8-bit micro when it does calculations based on 32-bit data (and vice versa). CoreMark will work on any architecture, though 8b handling is most efficient on 8b processors, 16b data types are handled optimally on 16b processors, and similarly 32b processors are the best at handling 32b data.
Realistically though, all of those data types are commonly used in most C code. The compiler is in charge of making the best use of the processor resources, and making sure the end result is correct.
CoreMark intrinsically uses several integer data types:
8b – used as data for the state machine (mostly in compares).
16b – used as data and status info during the list processing (read/write and bit manipulation) as well as input data for the matrix operations (computations), and for crc.
32b – used as result data type for matrix multiply operations.
This introduces a mix of the common programming data types and covers core integer functionality in the processor.
Hi Shay,
I think Coremark is a good effort toward “modern” benchmarking.
However, concerning data types, it suffers from a great problem: The arithmetic used to measure the “performance” of a compiler/target is 32-bit! This means that performance (speed) figures obtained on 8 or 16-bit processors are meaningless: Adding 2 32-bit objetcts is one instruction on a 32-bit processor, but can be a dozen on a 8-bitter.
I know, the benchmark tries to mimick some real-world applications, which require 32-bit arithmetics. However, lots of real-world applications do NOT require 32 and can live well with 16-bits arithmetic (think about all-software MP3 players).
When using the CoreMark results, engineers may get fooled by results and pick a 32-bit processor where an 8-bit would have been enough.
My proposal would be:
- Rewrite (just a little bit) the Coremark benchmark so that it has a meaningful 16-bit version.
- Separate the results of 8/16-bit processors from the 32-bit ones on the results page, so the figures are more meaningful.
Thanks for your excellent work anyway,
Bruno
Comment by Bruno Richard — February 17, 2010 @ 00:52
Bruno is absolutely correct.
1- The bench is heavily weighted towards 32-bit processing. We measured that 65% of the instructions generated by the compilation of the Coremark bench were of the Long type (32-bit). ALL processing instructions (ADD, SUB, MULT) in the bench work on 32-bit data.
2- They are thousands of applications where you will never instantiate 32-bit data. Think about:
- remote controls
- most appliances
- electronic light dimmers
- Thermostats
- seat positioning, window lift and mirror control in automotive.
The Coremark bench, by implicitly assuming that processing is always performed on 32-bit data (and not being very upfront about it), favors 32-bit architectures over 8- and 16-bit ones.
My recommendations are:
- be more upfront about what Coremark actually benches
- provide versions for 32-bit (which we have) and 8/16-bit
In any case this is already much better than the old Drystone bench.
Denis
Comment by Denis Cabrol — February 17, 2010 @ 09:15
The reverse is true. 32-bit processors do best with all int/unsigned local variables (globals and statics do not matter much). If you use char or short locals, you impact performance because they have to sign-extend or unsign-extend over and over.
Comment by Paul Kimelman — March 8, 2010 @ 16:11