Engineering
Quantisation made our model better, not worse. We did not expect that
Word error rate fell from 16.6% to 14.0% after quantisation, for the fused model only. The other three got worse. Here is the whole table and our best guess.
Quantisation is supposed to cost you something
The standard account is a trade. You reduce numerical precision, the model gets smaller and faster, and you give up a little accuracy. Quantisation-aware training recovers most of the loss. The remaining gap is the price of deployment.
On the NAVIR project we quantised an audio-visual speech recognition model for neuromorphic hardware, with a hybrid scheme of 8/4/4 bits for the input encoders and 4/4/4 for the intermediate modules, then fine-tuned with quantisation-aware training. We expected the usual small loss.
The fused model got better. Word error rate on the GRID benchmark under noise went from 16.6% before quantisation to 14.0% after, a 16% relative improvement in the wrong direction.
The whole table, because the headline is misleading on its own
If we stopped there it would be a nice result and a dishonest one. Quantisation did not improve the model. It improved one model and degraded three.
| Trained on | Before | After | Change |
|---|---|---|---|
| Noisy audio only | 21.0% | 22.5% | worse |
| Video only | 34.0% | 35.3% | worse |
| Clean audio only | 79.9% | 77.3% | marginal |
| Noisy audio and video | 16.6% | 14.0% | better |
Each single-modality model behaved exactly as the textbook says. Only the fused model improved, and it improved by more than the others lost.
Our best explanation, offered as a hypothesis
We think this is a regularisation effect that lands specifically on fusion, and we want to be clear that this is a hypothesis fitted to one result rather than something we set out to test.
A fused model has a way to cheat. When one modality is easier on the training distribution, the predictor head can lean on it and under-use the other. On noisy training audio the two channels are unevenly informative from moment to moment, which is exactly the condition under which a high-capacity head learns a lazy weighting.
Reducing precision to four bits in the intermediate modules removes the capacity required to represent that lazy weighting precisely. The head is pushed towards using both channels more evenly, which is the behaviour you actually want at inference time when you do not know in advance which modality will be degraded.
A single-modality model has no such shortcut available, so quantisation is pure loss for it. That is what the table shows.
We cannot prove this from one experiment. What we can say is that the improvement is consistent across both our datasets: on the task-specific command set, the fused model also held up under quantisation while the audio-only model collapsed entirely.
What we would do differently, and what we would tell you to check
- Quantise early, not last. We treated it as a deployment step. It changed model ranking, which means it belonged in the experimental loop.
- Re-rank your candidates after quantisation. The best floating-point model is not necessarily the best deployed model, and that is the only comparison that matters.
- Report the whole table. One row improving is a finding. One row reported alone is marketing.
- Watch the QAT budget. We used 15 epochs on GRID and 200 on the smaller task-specific set. That is a large difference in how much recovery the model gets, and it is rarely reported by anyone.
- Test fusion models under asymmetric degradation. Degrade one modality at inference and see whether performance falls off a cliff. That tells you whether the head learned a lazy weighting.
- Do not generalise from our result. Four-bit intermediate precision on a specific accelerator with a specific architecture. Your mileage will differ, which is the point of measuring.
My take
The tempting version of this post is the headline alone: quantisation improved our model by 16% relative, neuromorphic deployment is free. I have seen that post written, from weaker evidence than we have, and it is how a field builds a reputation for overclaiming.
The honest version is more useful to anyone actually building something. Quantisation did what it usually does to three of our four models. The fourth improved, we have a plausible mechanism, and we cannot prove it from one project.
What I have taken from this operationally is narrower and I am fairly confident about it: quantisation is not a post-processing step and treating it as one is a methodological error. We nearly shipped the wrong model, because we selected on floating-point performance and the ranking changed underneath us. If the deployed artefact is quantised, the quantised number is the only one that should influence a decision, and every architecture comparison made before that point is provisional.
The wider lesson is about fusion rather than about precision. A model with two inputs has an incentive to use one of them, and most evaluation protocols will never notice, because they test on data where both channels are healthy. Degrade one on purpose. It is a ten-minute experiment and it tells you whether you have built a fused model or an audio model with a camera attached.
The system these numbers come from is described in voice control that survives a factory floor, and the hardware constraints that forced the quantisation are in what neuromorphic inference costs you in architecture. Our broader MLOps work is largely about catching exactly this class of methodological error before it ships.