Field notes
Your edge AI pilot worked. Why it still will not scale
A pilot proves the model works at one site. Scaling to forty breaks on updates, connectivity, drift and ownership. What to agree before the pilot starts.
The pilot proves the model. It proves nothing else.
A pilot is a controlled experiment. One line, one barn, one array, one site. The camera is where you put it, the network is the one you tested on, and for eight weeks somebody who built the system is watching it daily. Under those conditions a decent model hits its target.
Then the pilot report says it works, and the proposal to roll it out across forty sites lands on a different desk with a different set of questions. Almost none of those questions are about accuracy, and almost none of them were answered by the pilot.
This is the most common way edge AI money gets wasted. Not a failed pilot. A successful pilot that was never designed to tell you anything about the fleet.
Five things that break between one site and forty
Each of these is invisible at n=1 and unavoidable at n=40.
- Model updates. Updating one device is a person with a laptop. Updating forty across three countries is a deployment pipeline, a rollback path, and a way to know which version is running where. If the pilot updated by hand, the roll-out has no update mechanism at all.
- Connectivity you did not test. The pilot site had the good link. Some fraction of the fleet will be on a link that drops for hours. A system that assumes an uplink degrades from useful to silent, and nobody finds out for weeks.
- Site-to-site variation. Different lighting, different camera mounts, different equipment vintages, different operators. A model tuned on one site's distribution meets forty slightly different distributions, and the accuracy you measured is the ceiling rather than the average.
- Drift with nobody watching. During the pilot an engineer noticed anomalies. In production nobody is looking, so accuracy decays quietly until a person downstream stops trusting the output and starts ignoring it. Trust, once lost, does not come back with a retrained model.
- Ownership. The pilot was owned by whoever wanted it. The fleet has to be owned by whoever is on call. If that handover was never planned, the system becomes orphaned infrastructure the first time its champion changes jobs.
What to agree before the pilot starts, not after
- Which sites are in scope eventually. Pick the pilot site to be representative of the hard case, not the easy one. A pilot at your best-connected, best-lit site tells you the least.
- How a model reaches a device. Even in the pilot, deploy through the mechanism you intend to use at scale. Deploying by hand once teaches you nothing you will reuse.
- What happens when the link is down. Define the degraded mode explicitly: buffer, queue, act locally, or stop. This is a business decision, not an engineering default.
- Who is told when accuracy drops, and how. Name the person and the channel. A monitoring dashboard nobody has agreed to read is not monitoring.
- What the retraining trigger is. A condition, not a calendar. Retraining on a schedule when nothing changed wastes money; retraining only after a complaint is too late.
- What handover means. Which documents, which runbook, which training, and the date your team takes it over. Write it into the pilot scope.
- What would make you stop. The condition under which you abandon the project. Deciding it while the outcome is still unknown is far cheaper than deciding it afterwards.
Two ways to run the same pilot
The difference is not budget. It is which questions the eight weeks are spent answering.
A pilot that tells you about the fleet
- Runs at a representative or difficult site
- Deploys through the pipeline you will use at scale
- Tests the degraded mode deliberately, by pulling the link
- Reports accuracy alongside latency, uptime and review load
- Ends with a runbook and a named owner
- States in advance what result would stop the project
A pilot that only tells you about itself
- Runs where conditions are most favourable
- Deploys by hand, once, by the person who built it
- Never loses connectivity during the test window
- Reports a single accuracy figure
- Ends with a slide deck
- Has no defined failure condition, so it cannot fail
Who owns it at three in the morning
Ask this question in the first meeting and watch the room. It is the fastest diagnostic available for whether a project will survive its first year.
An edge AI system is industrial infrastructure. It sits next to equipment that runs unattended, and it fails in ways the people around it have never seen before: not a broken machine, but a model quietly becoming less right. Somebody has to own that failure mode, and that somebody is usually neither the data science team nor the maintenance team, because it falls between them.
The answer does not have to be sophisticated. It has to exist, and it has to be a name.
My take
The accuracy target is the least interesting number in a pilot contract, and it is the one everyone negotiates hardest.
I understand why. It is the number that feels objective, it is the one a vendor can be held to, and it is the one that fits in the business case. But I have not seen a project die because the model reached 91% instead of 94%. I have seen several die because nobody could update forty devices, because the alerts went to an inbox nobody read, or because the one person who understood the system left.
So when a client wants to spend the pilot squeezing out another two points of accuracy, I usually argue for spending it somewhere else: deploy through the real pipeline, cut the link on purpose and see what happens, hand the runbook to someone who was not involved and watch them try to use it. Those results are less satisfying to put in a report and they are what determine whether there is a fleet in two years.
A pilot is not a demonstration that the model works. It is a rehearsal for operating it. Most are run as the first thing and then quoted as if they were the second.
Two of the failure modes above have their own posts: designing for the link that drops, and sizing the hardware without guessing. If a pilot of yours has stalled between one site and the rest, our MLOps work is mostly this problem, and you can book a discovery call.