RSS feed for comments on this post.
Line and paragraph breaks automatic, e-mail address never displayed, HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>
0.129 Powered by WordPress
Hi Shay,
I think Coremark is a good effort toward “modern” benchmarking.
However, concerning data types, it suffers from a great problem: The arithmetic used to measure the “performance” of a compiler/target is 32-bit! This means that performance (speed) figures obtained on 8 or 16-bit processors are meaningless: Adding 2 32-bit objetcts is one instruction on a 32-bit processor, but can be a dozen on a 8-bitter.
I know, the benchmark tries to mimick some real-world applications, which require 32-bit arithmetics. However, lots of real-world applications do NOT require 32 and can live well with 16-bits arithmetic (think about all-software MP3 players).
When using the CoreMark results, engineers may get fooled by results and pick a 32-bit processor where an 8-bit would have been enough.
My proposal would be:
- Rewrite (just a little bit) the Coremark benchmark so that it has a meaningful 16-bit version.
- Separate the results of 8/16-bit processors from the 32-bit ones on the results page, so the figures are more meaningful.
Thanks for your excellent work anyway,
Bruno
Comment by Bruno Richard — February 17, 2010 @ 00:52
Bruno is absolutely correct.
1- The bench is heavily weighted towards 32-bit processing. We measured that 65% of the instructions generated by the compilation of the Coremark bench were of the Long type (32-bit). ALL processing instructions (ADD, SUB, MULT) in the bench work on 32-bit data.
2- They are thousands of applications where you will never instantiate 32-bit data. Think about:
- remote controls
- most appliances
- electronic light dimmers
- Thermostats
- seat positioning, window lift and mirror control in automotive.
The Coremark bench, by implicitly assuming that processing is always performed on 32-bit data (and not being very upfront about it), favors 32-bit architectures over 8- and 16-bit ones.
My recommendations are:
- be more upfront about what Coremark actually benches
- provide versions for 32-bit (which we have) and 8/16-bit
In any case this is already much better than the old Drystone bench.
Denis
Comment by Denis Cabrol — February 17, 2010 @ 09:15
The reverse is true. 32-bit processors do best with all int/unsigned local variables (globals and statics do not matter much). If you use char or short locals, you impact performance because they have to sign-extend or unsign-extend over and over.
Comment by Paul Kimelman — March 8, 2010 @ 16:11