Engineering
How we benchmarked neuromorphic inference, and what we measured
Measured with a power meter, not derived from operation counts. The protocol, our numbers, the caveats, and why you should benchmark your own model instead.
Why we measured instead of calculating
Energy figures for low-power inference are usually derived rather than observed. You count synaptic operations, multiply by a published energy-per-operation, and report a ratio. The result is reproducible, useful for comparing architectures on paper, and tells you very little about what a device will draw on a bench.
Under NAVIR, a research project funded through the dAIEDGE network, we built an audio-visual speech recognition system that runs as spiking neural networks on a neuromorphic accelerator and drives a robot arm. We wanted to know what our models actually cost to run, so we put a meter on the demonstrator.
This post is mostly about method. The numbers are ours, they apply to our models on our setup, and the protocol is the part we would want anyone to copy.
The protocol
The demonstrator is a Raspberry Pi 5 with a BrainChip AKD1000 accelerator. Measurement was a FNIRSI FNB58 USB power meter on the supply.
- Repeated calls on fixed inputs. The same pre-computed lip crops and log-mel spectrograms, called in a loop for five minutes, so the average settles.
- Idle subtracted. Model loaded but not called, measured separately. The CPU backend idled at about 390 mWh over five minutes, and that is the baseline for every inference-attributable figure below.
- The meter's own draw accounted for. The Pi with no script ran 370.92 mWh, the unpowered meter alone 120.58 mWh, so the instrument is inside the numbers rather than beside them.
- Throughput recorded alongside energy. Energy per inference alone is easy to game by running slowly.
We also ran the same models on a laptop GPU for context. That comparison is the weakest thing in the study and we would rather say so than let it stand unqualified: GPU power came from polling nvidia-smi, which has known precision limits, and it measures the card while the Raspberry Pi figures are whole-system. The run was also longer and normalised back to the five-minute window. Different instrument, different scope, different duration. It is a rough sighting shot, not a like-for-like result.
What we measured
| Backend | Max throughput | Energy per inference |
|---|---|---|
| AKD1000, hardware | 14.55 inferences/s | 0.0165 mWh |
| Same board, CPU, Keras | 1.10 inferences/s | 0.4306 mWh |
| Same board, CPU, Akida software | 15.55 inferences/s | 0.0810 mWh |
| Laptop GPU, Keras (see caveats) | 2.54 inferences/s | 1.691 mWh |
For our audio-video model the accelerator drew 0.0894 mWh per inference at 2.61 inferences per second, against 0.7381 mWh on the same board's CPU.
These are measurements of our models, in our configuration, on one workload. They are not a benchmark of any product, and we would not present them as one. Anyone choosing hardware should measure their own model, which is the entire point of the post.
What the numbers say is genuinely promising
Taken on their own terms, the results are encouraging, and we say that as the people who went looking for reasons to doubt them.
The accelerator ran our video model at 14.55 inferences per second, which is comfortably above real time for this task, while drawing a small fraction of the energy of every other backend we tried on the same board. Both of those at once is the part that matters. Efficiency that costs you throughput is a trade, and this was not a trade. The audio-video model at 2.61 inferences per second is slower but still ample for speech, at well under a tenth of the CPU's per-inference energy.
For a device that runs continuously on a battery or on harvested power, in an enclosure with no fan, that difference is not an optimisation. It decides whether the product exists. We are actively looking for the next project on this class of hardware, because the ceiling on what it makes possible is clearly higher than what we reached in one research cycle.
Designing the model for the target
The accelerator executes a defined set of operations. Three-dimensional convolutions, recurrent connections and attention are outside it, which means a model built from those cannot be converted, only replaced. That is a design brief, not a complaint, and working within it was most of the engineering in this project.
What we built factorises spatial and temporal encoding into stages the chip executes natively: a per-frame image encoder, a temporal video encoder over short overlapping windows, a separate spectrogram encoder for audio, and a predictor head, with per-window outputs aggregated by a beam search constrained to the known grammar of the command set. According to the project's own review of the field, it is one of the first demonstrations of audio-visual speech recognition on this class of processor.
It also performs less well than unconstrained research models. Our lip-reading component reaches 34.0% word error rate on the GRID unseen-speaker split, where published conventional models reach 9.7% to 10.21% using spatiotemporal convolutions, attention decoders and large-scale pretraining. We chose a different design because we were targeting a power budget, and closing that gap is open work rather than a settled limit.
How to benchmark low-power inference properly
- Measure the supply, not the simulation. A USB meter costs very little against the cost of being wrong about a power budget.
- Subtract idle and publish the idle figure. Half the comparisons in this field are total-system draw against accelerator-only draw, and the ratio changes enormously.
- Report throughput next to energy. Both, always. Either alone can be made to look good.
- Run long enough to settle. Five minutes was enough for us. Ten seconds is not enough for anyone.
- Keep the scope of every number explicit. Whole system or one component. Same instrument or two. Say which.
- Benchmark your own model. Ours mapped onto the chip differently between two variants, so even our own numbers do not transfer between our own models.
- Record the accuracy alongside. An efficiency figure without the task performance next to it is not a result.
Something we have not explained
The same image encoder architecture, in two of our models, was partitioned onto the chip very differently: 75 neural processors in a single sequence for the video-only model, 348 across nine sequences for the audio-video one. The only difference between them is the trained weights. Total passes per inference went from 5 to 22, and throughput fell from 14.55 to 2.61 inferences per second.
We do not yet know why. It may be something in how we structured the audio-video model, something about how our weights interact with the partitioning step, or something we have not thought of. We are reporting it because it is the kind of result that costs somebody else a week, and because further optimisation of our model is clearly available.
Where this sits on the readiness scale
This was a research project and the outcome is a validated prototype, not a product. It ran on a bench and on a demonstrator, not in a plant, and we have not operated it over months or across a fleet. Anyone who tells you a neuromorphic AVSR pipeline is production-ready is ahead of the evidence, ours included.
What does transfer is the method. The benchmarking protocol, the model-design discipline of building for a constrained operator set, and the habit of reporting accuracy next to energy are all directly reusable, and they are what we would bring to the next project on this class of hardware. We are looking for that project.
My take
The useful output of this project was not a number, it was a protocol. We now know how to establish, in about two weeks, what a given model costs on a given piece of edge silicon, and how much task performance that costs. Before NAVIR we would have estimated it, and our estimate would have been wrong in both directions.
I am sceptical of efficiency claims in this field, including ones I would be tempted to make, and that scepticism is about measurement rather than about the technology. Idle draw, measurement scope, instrument, run length and whose model is running all move the answer by more than the differences people argue about. Two labs can measure the same board honestly and disagree by an order of magnitude.
The technology itself impressed us. Getting real-time throughput and an order-of-magnitude energy reduction simultaneously, on a board you can hold in one hand, is the sort of result that opens product categories rather than improving existing ones. We want to work on this again, and we think the interesting applications are the ones nobody is attempting yet because the power budget rules them out on conventional hardware.
The discipline I would insist on is narrow and unglamorous. Measure your own model, on your own target, with a meter, and report accuracy beside energy. We did that, the results were good, and they were good in a way we can defend line by line. What I will not do is tell you what a chip will do for your workload, because our own two models disagreed with each other on the same board. That is an argument for benchmarking, not an argument against the hardware.
The application these models serve is described in voice control that survives a factory floor, and the deployment step produced a quantisation result we did not expect. For conventional edge hardware sizing, see this post, or read about the capability itself.