Field notes
The labels were the problem: lameness, or just long hooves?
Our lameness classifier kept failing until we looked at the ground truth. Annotators were labelling two different conditions as one. The fix cost no compute.
The model looked broken
Partway through ELDER, our sheep lameness classifier stopped making sense. Not in the way a bad model usually fails, where accuracy is uniformly poor and you go looking for a bug. This was worse. The pipeline would classify one animal confidently and correctly, then classify a visually similar animal the other way, with equal confidence.
The obvious suspects were all wrong. The detection and pose stages were producing good keypoints. The temporal model was training normally, and loss was converging. More data made it slightly worse.
The problem was in the ground truth, and it had been there from the start.
Two conditions, one label
Our annotators were labelling sheep as lame or not lame. Reasonable instruction, and one that conceals a clinical distinction.
A sheep can walk badly because it is clinically lame, with foot rot, scald, an injury or an infection. A sheep can also walk badly because its hooves are overgrown and have not been trimmed. To a camera, and to a person watching for thirty seconds, both produce an irregular gait. To a farmer they are entirely different events: one needs a vet and isolation, the other needs a routine job doing.
Both were going into the training set under the same label, inconsistently, depending on who annotated that clip and what they were thinking about that day. The model was not confused. The model had correctly learned a boundary that did not exist.
An ambiguous label does not add noise evenly. It teaches the model to be confident about a distinction the data cannot support, which is why the failures looked arbitrary rather than merely weak.
The fix was a written annotation protocol that separated pathological lameness from maintenance-related gait irregularity, and a dataset expanded under those stricter rules. No architecture change. The pipeline that had looked unsalvageable became the one that finished at 86.49% accuracy with a false-negative rate of 8.33%.
How to find a label problem before it costs you a quarter
- Have two people label the same 100 examples, blind. Measure how often they agree. Do this before any model work, not after the model disappoints.
- Read the disagreements one by one. The pattern in them is your real problem definition. Ours was entirely one category of case.
- Ask what action follows the label. If two labels lead to different actions, they are different classes. If one label leads to two different actions, it is two classes wearing one name.
- Write the protocol down before scaling annotation. Including edge cases and worked examples. Verbal instructions do not survive contact with a second annotator.
- Suspect the labels when confidence and correctness are uncorrelated. A model that is confidently wrong on easy-looking cases is usually reporting an inconsistency in the training data faithfully.
- Re-check after any change of annotator. Label drift between people is as real as data drift over time, and far less monitored.
Your accuracy ceiling is annotator agreement
This is the part that generalises beyond livestock. If two competent people disagree about 15% of your examples, no model will reliably exceed 85% agreement with either of them, because there is no function that maps the input to both answers. You can spend a quarter on architecture and get nothing.
We make this point on our computer vision page, where the same thing happens with visual quality inspection: two inspectors disagree about what counts as a defect, and the model inherits the disagreement as an accuracy ceiling. ELDER is the version of that story where we had to live it ourselves.
Where to spend the next month
Fix the labels
- Raises the ceiling for every future model, not just this one
- Costs annotation time, not research time
- Forces a clear problem definition, which the customer benefits from too
- Produces a dataset that is reusable and can be handed over
- Usually finishes in weeks
Collect more data, or try a bigger model
- Feels like progress and is easy to justify in a status report
- More data with the same ambiguity makes the boundary worse, not better
- A larger model fits the inconsistency more confidently
- Costs compute, time and often new hardware
- Leaves the underlying defect in place for the next project
My take
Most things presented to me as model problems are label problems. I would put it at more than half, and the proportion rises the more specialised the domain is, because that is where the people writing the annotation instructions and the people who understand the distinction are least likely to be the same person.
It is an unpopular diagnosis. Fixing labels is unglamorous, it is not what anyone hired a machine learning team to do, and it implies the last few months of modelling were partly wasted. Suggesting a bigger model is always easier to say in a meeting.
But the economics are not close. A two-person labelling agreement check on 100 examples takes an afternoon. Not doing it cost us a chunk of ELDER, and we knew this failure mode already. The reason we did not catch it sooner is that the labels came from people who knew sheep far better than we did, and it did not occur to us that expertise and consistency are different things.
If you take one operational habit from this project, take that one. Before the modelling starts, have two experts label the same hundred examples and read every disagreement yourself.
The same project produced a false-positive rate we missed our target on, and a connectivity architecture that sent 200-byte messages over satellite. If you are building an inspection or monitoring model and the results look arbitrary, book a discovery call and bring a hundred labelled examples.