Engineering
Sizing edge hardware for a vision model, without guessing
Most teams choose an edge device before optimising the model, then pay for that choice at every site. How to size hardware after optimisation, not before.
The sizing question is usually asked backwards
The normal sequence is: train a model, measure how much compute it needs, choose a device that comfortably runs it, multiply by the number of sites, put that number in the business case. It is a reasonable-looking sequence and it produces a hardware bill that is often an order of magnitude too large.
The reason is that the model you measured is not the model you will deploy. A vision model straight out of training is carrying weights at full precision, layers that contribute almost nothing, and capacity sized for the training distribution rather than for your line. Optimisation routinely removes most of that. Size the hardware first and you have bought headroom for a model that no longer exists.
The inverse mistake is cheaper but more embarrassing: optimise aggressively, buy the smallest device that passes on the bench, then discover the enclosure sits in direct sun in July and the board throttles every afternoon.
What you are actually sizing for
Compute is one of five constraints, and rarely the binding one.
- Latency budget. Not "as fast as possible". The real number comes from the process: how long after the part passes the camera must the decision exist for something to act on it. A reject gate 400mm downstream on a moving line gives you a hard figure. Most people never ask for it and then over-engineer against an imagined one.
- Throughput. Frames per second across all cameras on the device, at the resolution the defect actually requires, which is usually lower than the camera's maximum.
- Thermal envelope. Ambient temperature in the real enclosure, not the lab. Sustained throughput on a passively cooled board in a hot cabinet is well below its benchmark figure.
- Power and its delivery. Available supply at the point of installation, and what happens on a brownout. On remote sites this can dominate everything else.
- Physical and electrical fit. DIN rail or not, ingress protection, vibration, the cable run to the camera, and whether maintenance can reach it without stopping the line.
Measure these before you choose a device
- The decision deadline, in milliseconds, taken from the process. Ask what acts on the output and how long it has.
- The lowest resolution at which the defect is still detectable. Test it. Halving resolution usually cuts compute by four and often costs nothing in accuracy.
- Inference cost after quantisation, not before. Measure the optimised model, on the target architecture, not a desktop GPU.
- Sustained throughput at the enclosure's real ambient temperature. Run it for an hour, not for ten seconds.
- What else the device must do. Video decode, pre-processing, buffering and the operating system all take a share.
- Headroom for one model generation, not five. Budgeting for imaginary future models is how single-site costs become fleet-wide costs.
- The cost of being wrong in each direction. Over-spec is money per site. Under-spec is a site visit per device. Know which one your fleet size makes worse.
What optimisation actually buys
Three techniques do most of the work, and they compose.
Quantisation reduces numerical precision, most commonly to 8-bit integers. It is the highest-return step and usually the first one to try, because inference hardware is built for it. Quantisation-aware training recovers most of whatever accuracy post-training quantisation costs.
Pruning removes weights or whole channels that contribute little. Structured pruning is the one that matters in practice, because unstructured sparsity needs hardware support to turn into real speed rather than theoretical FLOP reduction.
Distillation trains a smaller model to reproduce the larger one's behaviour. It is the slowest of the three to apply and often the most effective, because it changes the architecture rather than compressing it.
The gains are not additive in a simple way and they are task-dependent, so anybody quoting you a universal compression factor is selling something. What is reliable is the direction: measure after optimisation and the device you need is usually smaller and cheaper than the one you would have bought.
Two device classes, and when each one wins
Accelerated edge modules (Jetson class)
- Best performance per watt for neural inference
- Handles multiple camera streams on one unit
- Mature tooling for quantised deployment
- Compact, suitable where power is limited
- Right choice when vision is the main workload
Industrial x86 gateways
- Familiar to the people who maintain the plant
- Runs your existing industrial software alongside the model
- Easier integration with fieldbus and existing control systems
- Wider operating temperature range in industrial variants
- Right choice when inference is one job among several
Neither column is a recommendation. The deciding question is usually not performance but who maintains the box and what else it has to do.
My take
Hardware sizing is the one place where a decision made in an afternoon gets multiplied by the size of the fleet and then paid for every year. It deserves more care than it gets, and it usually gets none, because it arrives disguised as a procurement question rather than an engineering one.
My rule is simple and unpopular with schedules. Do not choose the device until the model has been through at least quantisation, and until somebody has asked the process what the actual latency deadline is. Both take days. Both routinely change the answer by a factor that dwarfs anything else in the project.
The second thing I would push back on is buying headroom for models you have not built. It feels prudent. It is, in practice, paying a premium at every site for a capability with no specification, and when the future model finally arrives it needs a different accelerator anyway. Buy for the model you have plus one generation, and put the savings into the deployment pipeline, which is what will actually be underfunded.
Sizing is the easy half. The harder half is what happens once forty of these are installed: see why pilots do not reach the fleet and designing for the link that drops. Our computer vision work covers the vision side end to end, and you can book a discovery call to talk through a specific line.