The Certification Process
The EEMBC Test Harness (TH)
Q: What is EEMBC's mission?
A: The Embedded Microprocessor Benchmark Consortium (EEMBC - pronounced embassy) is a non-profit organization that works collaboratively with member companies to develop performance benchmarks that target key applications of embedded systems. EEMBC activities include the EEMBC Technology Center, which delivers technical support to members and licensees, certifies benchmark scores, and oversees the development of benchmark software.
Q: Who are EEMBC's members?
A: EEMBC members include semiconductor manufacturers,
intellectual property providers, compiler and RTOS vendors,
Q: Why aren't benchmark scores
available for all embedded processors?
A: Members must agree to publish their benchmark
scores. If a processor score is not available on the EEMBC
website, you may request that score on line or directly with
the corresponding processor vendor. Customer requests for
scores on specific processors are a key mechanism by which
more scores are published, and EEMBC encourages you to request
certified scores for any processor you are considering for
Q: What steps does EEMBC take to
ensure that the published benchmark results are accurate and
A: Only scores that have been verified by the EEMBC Technology Center may be publishedfollowing a rigorous testing process to ensure that benchmark scores are repeatable, obtained fairly, and according to the rules of EEMBC. The EEMBC Technology Center is also responsible for verifying the exact environment under which each test was run; this includes processor frequency, wait states, and compiler version.
Q: What role does EDN play in the
A: EEMBC traces its origins to a "hands-on project" conducted by Markus Levy at EDN Magazine in early 1996, and since that time EDN has provided EEMBC with the same gracious support it extends to other non-profit organizations in the semiconductor industry. Otherwise, there is no ownership or organizational relationship between EEMBC and EDN.
Q: How do I qualify to be an EEMBC
A: You must be a designer or manufacturer for which
the end products are microprocessors, microcontrollers or
DSPs. Alternatively, you must be a designer or manufacturer
of microprocessor development tools or manufacturer of products
for which the primary end-product includes hardware, software,
or firmware associated with microprocessors.
Q: What are the main steps in
becoming a member of EEMBC?
A: First, please fill out the form you’ll find
on the EEMBC Membership page and submit the requested information.
We will contact you with full details regarding the process
of becoming a member. Member companies are required to execute
a set of standard agreements, including an Application for
Membership, the EEMBC Bylaws, and a Member License Agreement.
Membership dues are typically paid by check or wire transfer.
Q: Is my whole company included
in our EEMBC membership? Can more than one person in a member
company access the benchmark source code?
A: Once you have become an EEMBC member, as many
people in your company as you like may download the EEMBC
benchmark source code, but each such individual must be advised
of the EEMBC rules and the provisions of the EEMBC Member
License Agreement. Moreover, each person accessing the members-only
area of the EEMBC Web site to download source code or obtain
other information must register individually. You are also
responsible for seeing that any employees who leave your company
do not take the source code outside of your company.
Q: What sort of person should be
the representative from our company to the EEMBC Board of
A: EEMBC Board of Directors meetings combine both
technical and marketing-oriented discussions, and our member
representatives include individuals whose roles in their respective
companies are purely technical, purely marketing, and everything
in between. In general, the most technical discussions in
EEMBC take place on the subcommittee level by teleconference
outside of Board meetings. Once your company has joined the
Consortium, you are welcome to review the minutes of past
Board meetings (posted in the members-only area of the EEMBC
Web site) as a way of understanding the issues that are regularly
discussed and decided by the Board.
Q: Do we own the benchmark source code once we have
become a member?
A: No, EEMBC owns the source code; your membership
makes you a licensee.
Q: What are the fees for joining
A: The fees for a new Board of Directors member are
$30,000 for the first year and $16,000 for each subsequent
year. However, if you want to join but don't want to be a
Board member, you can join individual subcommittees (application
area) where the cost ranges between $7,500-$12,000 for the
first year ($6,000 for subsequent years) for each technical
subcommittee that you participate on. Note that from time
to time, dues may be raised or reduced as the Board members
Q: How are membership dues spent?
A: Your member dues go towards the development, maintenance, and marketing of the benchmarks as well as funding certifications and technical support from the EEMBC Technology Center.
Q: What's the difference between
a board member and a technical subcommittee member?
A: A board member has full voting rights on all the
major elements of the consortium. A board member has full
access to all of the EEMBC benchmarks. A subcommittee member
(application area) can only help determine the technical direction
of the subcommittee(s) for which it has paid a membership
fee. A subcommittee member has access to the benchmarks in
the application area addressed by the subcommittee.
Q: Does EEMBC have a university
program or special rate for academia?
A: Yes. For more information, visit the “Academic
Members” area under the “Membership” section
of the EEMBC Web site.
Q: Can I just "audit"
A: No, you must be an EEMBC member.
Q: Does EEMBC have benchmarks for
A: Yes. Our Digital Entertainment and Telecom benchmark
suites contain a number of kernels and applications that are
highly relevant for DSPs. Furthermore, EEMBC’s rules
allow benchmark code optimizations to promote support for
specialized DSP instructions and architectures. For more information,
visit the Digital Entertainment and Telecom areas on the EEMBC
Q: EEMBC has such a variety of
benchmarks, how do I determine which one to use?
A: Focus on those parameters that most closely match
your application needs. A designer typically uses the benchmark
scores as a criterion to pick the best processor for the application.
For example, if the application does many table look-ups,
then that parameter (in the Automotive/industrial benchmark
suite) is most important and the designer will like give the
most weight to that benchmark. On the other hand, if the application
is a modem, then the benchmark scores in the Telecomm suite
become the most important parameters.
Q: What process does EEMBC use
to select specific benchmark algorithms?
A: Algorithms are selected by discussion, consensus,
and ultimately a vote of the member companies. Non-members
are also invited to suggest benchmark algorithms that you
would like to see the consortium develop.
Q: Does EEMBC perform board-level
or system-level benchmarking?
A: The Version 1.0 benchmarks are processor benchmarks.
However, the board and/or system can make a significant difference
in processor performance (for example, the memory subsystem
is one aspect that can dictate performance). EEMBC’s
second-generation benchmarks, specifically for Networking
and Digital Entertainment are system-level benchmarks. The
same is true for EEMBC’s Java benchmarks which can be
run directly on a mobile phone platform.
Understanding EEMBC Benchmark Scores
Q: What scoring methods does EEMBC
A: EEMBC allows two scoring methods: Out-of-the-box
and Full-Fury. Out-of-the-box is where you use the compiler
of your choice along with any compiler switches, but you cannot
modify the original EEMBC source code. Full-Fury is the optimized
version where you can hand tune the code to take advantage
of any peripherals or hardware accelerators, special libraries,
or assembly language. Several members have provided different
levels of optimized scores.
Q: In out-of-the-box benchmarking,
a vendor can use any compiler, but cannot alter the source
code in any way. Does the compiler have a significant effect
on the benchmark score, even in out-of-the-box scores?
A: Absolutely. Feedback from several processor vendors
indicates that they have seen performance differences upwards
of 40% simply as a result of using a different compiler. This
is why vendors such as Green Hills, IAR, and Wind River are
actively involved in the EEMBC process.
Q: What do the out of box scores
reveal about compiler performance and/or processor performance?
A: Over time, as more processor vendors submit scores
from a variety of compilers running on the same processor,
this will provide very valuable information. Also, if you
held the compiler family constant and varied the processor,
this would be a great way to compare a processor's C friendliness.
Q: Must a compiler vendor be an
EEMBC member to have scores posted on the EEMBC web site?
A: A compiler vendor's results on the web site are
not related to whether or not that vendor is an EEMBC member.
The processor vendor member typically makes that decision,
and they usually publish scores from the compiler that yields
the best scores.
Q: EEMBC benchmarks attempt to
resemble real-world applications by allowing members to optimize
the source code to take advantage of their architectures.
How do you define the optimization guidelines so customers
can still make apples to apples comparisons?
A: EEMBC-allowed optimizations are decided on a benchmark-by-benchmark
basis. There are some fundamental restrictions in place to
ensure fair comparison. EEMBC allows optimizations that change
the source code to take advantage of libraries or hardware
accelerators. These restrictions will allow you to demonstrate
an architecture's capability and facilitate practical comparisons.
You can also use intrinsics ( i.e. C code modification), or
change the algorithm in some cases. For example, if EEMBC
has a radix-4 implementation of an FFT, we will allow it to
change to radix-2. So it is still an FFT, but on some DSPs
that makes a big difference.
Q: What are the restrictions on
information found on EEMBC's web site? How can I use the benchmark
scores for various processors?
A: This topic is treated in full in the document
Publishing Your Benchmark Scores: Questions and Answers on
Procedures and Marketing Opportunities, available from the
Members-only section of the EEMBC Web site. If you are not
an EEMBC member, please direct your questions to one of the
contacts listed on the "Press & Events" page
on the EEMBC Web site.
The Certification Process
Q: What are the main steps in obtaining
certified benchmark scores for my processor?
A: The major steps are listed here:
||If this is your first certification, read the EEMBC Operations Guide, which is posted in the Members-only area of the EEMBC Web site.
||Make a Certification Reservation in the Members-only area of the EEMBC Web site.
||If you haven’t done so before, download the benchmark source code. Read the Porting Guide.
||Port the benchmarks and Test Harness to your board or platform. For a fee, the EEMBC Technology Center can do this for you.
||Run the benchmarks according to the rules listed in the EEMBC Operations Guide.
||Complete a Disclosure Workbook and send it, along with other important data, to the EEMBC Technology Center for certification.
||Review certification report from the EEMBC Technology Center.
||If you want to publish the scores, sign the required disclosure form and return it to EEMBC.
Q: Are any additional tools needed
to run the benchmarks?
A: No. You need to download the benchmark source code and Test Harness and port these to your board or platform. For a fee, the EEMBC Technology Center can do this for you.
Q: What information goes into
the Disclosure Workbook?
A: The Disclosure Workbook contains all the information
necessary to recreate the benchmark environment. In addition
to all the benchmark scores, the Disclosure Workbook contains
board-level information (memory configuration, etc.), compiler
information (settings, libraries, etc.), and processor information
(core and bus speeds, cache sizes, on-chip peripherals, etc.).
It also contains details on the types of optimizations that
were used for a Full-Fury score. After certification is complete,
and the scores are published, the associated Disclosure Workbook
is publicly available alongside the benchmark scores.
Q: What does the certification
A: As an EEMBC Board member, you are entitled to two free certifications per year. Certifications are available to subcommittee members on a fee basis.
The EEMBC Test Harness
Q: What is the motivation behind
the Test Harness (TH)?
A: The Test Harness (TH) is a necessity to maintain
extremely portable, honest benchmarks. The TH provides a standardized
interface to each kernel so that you do not have to port each
kernel, just the TH. This provides a standard API for dealing
with timers and the serial port host-target interface, and
for file I/O. Parenthetically, the TH has been ported to more
than three dozen architectures so far and has proven a reliable
method of doing industry-standard benchmarking. It is analogous
to SPEC's test harness (although designed quite differently),
BAPCo's test setup, and every other benchmark suite. The TH
is one of the key components that allows us to conduct certification
work inexpensively, passing that savings on to each vendor,
and ensures that EEMBC's test results are truthful and accurate.
Q: What is the basic Test Harness
A: The TH is divided into two parts: the functional
layer (in the src directory) and the platform-dependent portion
(in the al, or adaptation layer). The TH assumes that you
have a small number of C library functions. When the TH was
created, all EEMBC members had C libraries, including all
the DSP vendors.
Q: Does the Test Harness affect
the benchmark score?
A: It is impossible to ensure that the TH and communications
with the host have no impact on the results. It is only possible
to ensure that it has an inconsequential impact. The issue
is the relatively small amount of time it takes for th_signal_start()
and th_signal_finished() to execute, and for the '>>
START!' and '>> FINSIHED!' messages to be transmitted
to the host and seen by the Host Control Program. If tests
are only executed for very short time periods, then the communications
latency between the target and the host would skew the results.
Benchmarks need to run for a relatively long period of time,
a minimum of two seconds, but preferably up to thirty seconds.
Q: Should the Test Harness use
the target or host timer?
A: Better accuracy is obtained if a TH port implements
the target timer. But even this method is problematic. For
example, if we mandated that tests run for a short period
of time, one second for instance, then we would need a timer
accuracy and resolution of at least 1ms to get good results.
Running a timer interrupt at 1ms skews the results! In this
case, it can actually produce more accurate results by letting
the host time the test. Many systems do not have high-accuracy,
low-overhead timers. In fact, many target platforms do not
have on board timers. This is exactly why the TH supports
host-based timing and why we have the Host Control Program.
Q: Why does EEMBC have an FFT filter
benchmark in both the Telecommunications and Automotive/Industrial
A: FFTs are used more often in Telecomm applications,
but they are becoming increasingly important in automotive
and industrial applications. Sensors used in engine knock
detection, vehicle stability control, occupant safety systems
and flow control, all require sophisticated filtering of the
incoming signal to determine what is really happening among
a lot of noise.
Q: Why don't the Telecomm benchmarks
use floating point?
A: All of the kernels (Bit Allocation, AutoCorrelation,
FFT/iFFT, Viterbi, Biquad IIR, Convolutional Encoder) in the
suite are intended to be run (for comparison purposes) using
16-bit fixed point math. There are FP versions of some of
the kernels which were used to generate the golden data file
sets. At this time, we have no plans to extend the "Out
of the Box" code to require/use FP math. You may re-code
the kernels, using FP for an optimized version of the benchmark
if you wish. At this time, we do not have access to any floating
point data sets. The data sets which accompany the AutoCorr
kernel are in 16-bit (1.15) fixed point format (that is: 1
sign bit, 15 bits of data) to reduce the precision of the
data, you can remove (or zero) bits from the LSBs of each