Case study
Voice control that survives a factory floor: what lip reading adds
An audio-only model at 3.7% word error rate returns 79.9% once the room is loud. Adding lip motion takes it to 16.6%. Measured numbers from a funded project.
Speech recognition is benchmarked in the wrong room
Almost every speech recognition figure you will see was measured on clean audio. Factory floors, workshops and warehouses are not clean audio. They are compressors, extraction fans, impact tools and people shouting over all three.
We built an audio-visual speech recognition system for exactly this case under NAVIR, a project funded through the dAIEDGE network, to give an operator voice control of a robot arm in a noisy environment. The first thing we measured was how badly conventional audio-only recognition degrades. The answer is worse than most people assume.
A model trained on clean audio does not degrade. It collapses.
These are word error rates on the GRID benchmark, unseen speakers, with noise drawn from the mechanical sounds of UrbanSound8K at signal-to-noise ratios between 0 and minus 15 decibels.
| Trained on | Clean audio | Noisy audio |
|---|---|---|
| Clean audio only | 3.7% | 79.9% |
| Noisy audio only | 4.4% | 21.0% |
| Video only (lip reading) | 34.0% | 34.0% |
| Noisy audio and video | 7.7% | 16.6% |
Read the first row twice. A model that reaches 3.7% word error rate on clean speech returns 79.9% when the room is loud. That is not a degraded system, it is a broken one, and it is the system most people deploy because it is the one that demos beautifully in a quiet office.
Noise augmentation during training recovers most of the loss, from 79.9% to 21.0%. Adding the visual channel takes it to 16.6%, a further 21% relative improvement over the best audio-only model.
Why lips help, and when they do not
Lip motion carries a signal that is entirely uncorrelated with acoustic noise. A compressor cycling does not change the shape of a speaker's mouth. So when the audio channel degrades, the visual channel is unaffected, and a fused model has somewhere to fall back to.
The third row of the table is the honest limit. Lip reading alone gives 34.0% word error rate on unseen speakers. That is not good, and we are not going to pretend otherwise: the best published non-neuromorphic lip-reading models on the same split reach 9.7% to 10.21%. They do it with full three-dimensional spatiotemporal convolutions, attention-augmented decoders and large-scale pretraining, none of which our target hardware supports. That trade is the subject of its own post.
Video is not a better modality than audio. It is an uncorrelated one, and that is what makes fusion work. The fused model beats both of its inputs precisely because their failure modes do not overlap.
What this means if you are specifying a voice interface
- Ask for the error rate at your noise floor, not theirs. A single word error rate figure is almost certainly a clean-audio number.
- Noise-augment the training, always. It is the cheapest single intervention available and it moved us from 79.9% to 21.0%.
- Constrain the vocabulary. We decode against the known grammar of the command set. A general dictation model solving a control problem is doing unnecessary work badly.
- Measure command accuracy, not just word error rate. Getting one word wrong in a command either does or does not change the action. Those are different failures.
- Decide what happens on a low-confidence result. For a robot arm the answer is do nothing and ask again, which is a design decision rather than a default.
- Check whether a camera is acceptable. A mouth-facing camera raises questions in some workplaces that a microphone does not.
On a real command set, the picture is starker
GRID is a research benchmark. Our own task set is a constrained vocabulary of robot commands, which is what an industrial deployment actually looks like. Measured as word error rate and command accuracy:
| Trained on | Clean audio | Noisy audio |
|---|---|---|
| Clean audio only | 6.6% WER, 91.5% commands | 98.7% WER, 0% commands |
| Video only | 0.7% WER, 100% commands | 0.7% WER, 100% commands |
| Noisy audio and video | 0.6% WER, 98.6% commands | 1.5% WER, 98.6% commands |
An audio-only system trained on clean speech executes zero commands correctly once the room is noisy. Not a degraded fraction. Zero. The fused system holds 98.6%.
The gap between the two tables is worth understanding. A constrained command set is a far easier problem than open-vocabulary sentence recognition, which is why the numbers are so much better. If somebody quotes you performance on their own command set, that is the correct thing for them to quote, and it tells you nothing about how the system handles a vocabulary it was not built for.
My take
The industry benchmarks speech recognition in a recording booth and then sells it to a factory. Everybody involved knows this and the practice continues, because the clean number is the one that fits on a slide and the noisy number is nobody's responsibility until deployment.
What surprised me in our own data was not that audio degrades in noise. It was the shape of the degradation. I expected a slope. What we got was a cliff: 3.7% to 79.9%, and on the command task a drop to literally zero commands executed. There is no graceful failure region to engineer around. The model is either in its training distribution or it is producing confident nonsense.
That is the real argument for fusion, and it is not about accuracy. It is that a second uncorrelated modality converts a cliff into a slope. I would take a fused system at 16.6% over an audio-only system at 21.0% even if the averages were identical, because I know what the fused one does when conditions get worse and I do not know that about the other.
If you are specifying a voice interface for a real industrial environment, the question to ask is not what the word error rate is. It is what happens at minus 10 decibels, and whether anyone has measured it.
The hardware side of this project has its own write-up: what the neuromorphic accelerator cost us in architecture, and why quantisation improved the fused model. For the capability in general, see neuromorphic and low-power edge AI.