The rules contain two sections: model equivalency rules, which specify which transformations may be applied to the fp32 TFLite base-models; and electrical run rules, which are relevant for submitting energy scores.
The model equivalency rules were developed by MLCommons and can be found on the MLPerf Tiny GitHub repositor, Section 8. The rules below are the energy collection rules developed by EEMBC.
| Run Rule Type | # | Rule | Justification |
|---|---|---|---|
| General | 1.1 | Scores are only valid if collected with the EEMBC Host Runner software. | To ensure the test was run according to specification. |
| General | 1.2 | Energy, performance, and verification (EPV) must use the same firmware. | To ensure the three different scores are consistent. |
| General | 1.3 | Verification score must be within Accuracy % and AUC margins-of-error found here. | To ensure optimizations have not degraded the model's accuracy. |
| General | 1.4 | The DUT must be "typical" power SKU, e.g. a median of a large sample of publicly available parts. | To avoid picking ultra-rare parts for higher scores. |
| General | 1.5 | The DUT hardware must be publicly available. | To ensure that the hardware is available to anyone. |
| Electrical & Environmental | 2.1 | Only one power supply is allowed into the DUT for energy mode. | To prevent escapee current paths that would artificially reduce power. |
| Electrical & Environmental | 2.2 | Any energy used during the benchmark must be drawn from the energy monitor. | To prevent attempts at hiding the amount of energy used (e.g., via supercapacitor). |
| Electrical & Environmental | 2.3 | Minimum 21C ambient temperature. | To prevent thermal-related optimizations (e.g., leakage sensitivity). |
| Electrical & Environmental | 2.4 | The board may be modified by cutting traces, removing jumpers, or desoldering bridges to remove ancillary components that may increase power. | To allow the user to remove unused platform hardware to obtain a more realistic score. |
| Firmware | 3.1 | Only "th_" functions may be modified in the firmware’s source code. | To prevent the user from modifying the behavior of the benchmark. |