Field notes
What a 16% false-positive rate actually costs a farm
We missed one of eight KPIs on a livestock project. Here is the number, what it means at flock scale, and why accuracy is the wrong thing to ask about.
The number we missed, and why we are publishing it
ELDER is an edge AI system that scores sheep for lameness from video as they walk past a camera. It was cascade-funded under the AGRARIAN programme, and like every funded project it declared its targets before it started. Eight of them. We beat six.
The one worth writing about is the one we missed. We targeted a false-positive rate below 10%. We finished at 16%.
| Measure | Target | Final | Result |
|---|---|---|---|
| Classification accuracy | 80% or better | 86.49% | Exceeded |
| False-negative rate | under 10% | 8.33% | Achieved |
| False-positive rate | under 10% | 16% | Missed |
| Data reduction vs raw video | 80% or better | 99% | Exceeded |
| Inference latency, 7-frame segment | under 500ms | 492.25ms | Achieved |
| Uptime during field testing | over 95% | 100% | Achieved |
| Unique animals tracked | 20 or more | 98 | Exceeded |
An accuracy figure of 86.49% would look fine on a slide. It is also the least useful number in the table.
What 16% costs at flock scale
A false-positive rate of 16% means that for every 100 healthy sheep that walk past the camera, 16 generate a lameness alert.
Put that in a commercial flock of 500 animals passing a race once a day. That is roughly 80 alerts a day for animals that are fine. A shepherd who checks the first twenty and finds them all sound will stop checking. Then the system's real output is not 80 false alerts, it is zero alerts, because nobody reads them any more.
This is the failure mode that kills monitoring systems, and it does not show up in any accuracy metric. The model keeps performing exactly as measured. The humans stop responding.
False positives and false negatives are not opposites
They are usually discussed as a trade-off along one axis, as though the only question were where to put the threshold. In a livestock context they have completely different shapes.
A false negative is a lame ewe that goes untreated for another day. That is a real cost, it compounds, and foot rot spreads through a flock. Our false-negative rate finished at 8.33%, meaning the system found roughly 92 of every 100 lame animals. For a screening tool that is the number that matters most, and it is why we treated it as the safety metric rather than accuracy.
A false positive costs a walk to the pen and a look at four feet. Individually trivial. The damage is cumulative and psychological rather than per-incident, and it lands on the one resource a farm cannot buy more of, which is attention.
So the two errors do not trade off cleanly. One has a direct animal-welfare cost per event. The other has almost no cost per event and destroys the system entirely once it crosses a threshold nobody can specify in advance.
What produced our false positives, in order of impact
- Sheep do not walk like test subjects. Head-dropping while walking, raising and lowering the head mid-stride, reads to a gait model as the compensatory head bob of a limp.
- Floors get washed. Sanitation leaves the passage slippery, and a healthy animal adjusting its footing on a wet floor produces exactly the asymmetry the model is looking for.
- Crowding. Animals pushing past each other occlude legs and distort the gait cycle.
- Running. An animal that trots or bolts through the race presents a gait the model was not trained on.
- The threshold was set for the lab. We started at a 50% lameness score and raised it to 70% during the field phase, which cut false alarms at the cost of sensitivity. That is a business decision dressed up as a hyperparameter.
Two honest ways to respond
Design for human-in-the-loop
- Every alert carries a snapshot, so verification takes seconds not a trip
- Alerts are ranked by score, not delivered as a flat list
- Low, moderate and high confidence bands let the farmer triage
- Keeps the low false-negative rate, which is the welfare-critical one
- Accepts that the system is a screening tool, not a diagnosis
Raise the threshold until alerts look clean
- Immediately satisfying, and the demo looks better
- Trades directly against the false-negative rate
- Pushes missed lame animals back onto the shepherd
- Hides the problem rather than measuring it
- The number that degrades is the one nobody is watching
We took the first route, and said so in the final report: a 16% false-positive rate necessitates a human-in-the-loop workflow, where the farmer verifies the alert from the attached image before acting. That reduces the labour saving. It does not remove the value, because the system still finds 92 of every 100 lame animals and it never gets tired.
My take
Every buyer I have met asks about accuracy in the first meeting. Almost none ask about the false-positive rate, and none at all ask who is going to read the alerts.
Those two questions determine whether a monitoring system is still switched on a year later. Accuracy determines almost nothing, because it can be inflated by class balance, by a favourable test set, or by a threshold quietly moved the week before the demo. A system can be 95% accurate and useless if the 5% arrives as eighty alerts a day.
We published the 16% because we would rather have the argument now than have a customer discover it in month three. The uncomfortable part is that I do not think we were unusually bad. I think most systems in this category have a false-positive rate in this range, and most of them do not report it, because the KPI was never declared in advance and the evaluation happened after the threshold was chosen.
Ask your vendor what their false-positive rate is at the threshold they intend to ship. If the answer is a single accuracy figure, you have learned something anyway.
The related failure in this project was not in the model at all, it was in the ground truth: annotators were labelling two different conditions as one. For the connectivity side of the same deployment, see how we got video down to 200 bytes over a satellite link. If you are scoping animal monitoring, this is the work we do.