Engineering
Why OCR alone fails on mill test certificates
OCR reads the characters and loses the rest: structure, units, heat scope. Why extraction needs a language model, validation rules and confidence scoring.
What a mill test certificate looks like to a machine
To a quality engineer, a mill test certificate is a familiar page. To a piece of software, it is one of several thousand incompatible layouts that happen to contain the same twenty facts.
A processor buying from a dozen mills receives a dozen templates, revised whenever a mill updates its system. Some arrive as native PDFs with a clean text layer. Many arrive as scans, occasionally as a photograph of a scan. Chemical composition might be a wide table with element symbols as headers, or a narrow one with a row per element, or a run of text. Heat numbers appear under half a dozen labels in at least four languages.
None of that is unusual for document processing. It is only unusual in what happens next, because the extracted number ends up on a certificate that somebody signs.
The five ways pure OCR fails
Optical character recognition converts pixels to characters. It does that well now. The failures are not in the characters, they are in everything the characters were sitting inside.
- Structure is lost. OCR returns text with coordinates, not a table. Knowing that 0.17 appeared on the page is useless without knowing it was carbon content and not manganese.
- Layout changes break templates. Template-based extraction works until a mill moves a column. Then it fails silently, which is worse than failing loudly.
- Ambiguous characters land in numeric fields. A scanned 0 read as 8, a 1 as 7, a misplaced decimal separator. In prose a human corrects this instinctively. In a yield strength value nobody notices.
- Units and conventions vary. Composition in percent or in parts per million, strength in MPa or N/mm², comma or point as the decimal mark, sometimes on the same page.
- Multi-page and multi-heat documents confuse scope. One PDF may cover several heats. Extracting values without tracking which heat they belong to produces a certificate that is precisely wrong.
Every one of these produces a plausible-looking number. That is the whole problem. A failure that produced obvious nonsense would be safe.
What separates a demo from a production pipeline
- It handles a mill it has never seen. No new parser, no configuration project, just lower confidence on the first few documents.
- It knows which heat each value belongs to. Scope tracking across pages, not a flat list of extracted numbers.
- It normalises units before comparison. Percent against parts per million, MPa against N/mm², resolved at extraction rather than at review.
- It checks values against the grade. Carbon outside the range for the declared steel grade is caught by a rule, not by a person at the end of the month.
- It scores its own uncertainty per field. Not a document-level score, a field-level one, because one bad cell is enough to invalidate a certificate.
- It refuses to guess. Below threshold, the field goes to an operator instead of into the output.
- It shows the source. Every extracted value traceable back to the region of the page it came from, so review takes seconds rather than a full re-read.
What a language model adds, and what it does not
A language model reading the OCR output alongside the page layout solves the structural problem, and it solves it in the way template parsers never could. It can tell that a column headed Rm holds tensile strength, that a block of figures under an unfamiliar heading is chemical composition, and that a number labelled in Italian is the same field as one labelled in German. Templates cannot generalise. Models can.
What a language model does not do is know whether the number is right. It will read 0.17 correctly, and it will read 0.77 correctly, and it has no independent opinion about which one belonged on the page. If the OCR layer produced a wrong character, a fluent model will happily build a fluent, wrong answer around it.
So the model is necessary and not sufficient. It converts a layout problem into a data problem. The data problem is the one that still has to be solved.
Validation rules are the part nobody demos
The unglamorous layer is where correctness actually comes from. Composition checked against the declared grade. Mechanical properties checked against the range the standard permits for that grade and product form. Heat numbers checked for format and for existence in the order. Cross-checks between the tensile report and the mill certificate where both exist.
None of this is clever. All of it catches the failures that matter, because the errors that survive OCR and survive a language model are exactly the errors that are individually plausible and collectively impossible.
Accuracy is the wrong metric
Vendors quote extraction accuracy. It is the wrong number to buy on, and this is worth being precise about.
Consider two pipelines. The first extracts 97% of fields correctly and reports nothing about the other 3%. The second extracts 88% correctly, and flags 15% of fields as uncertain, catching most of its own errors in the process.
On the headline metric the first wins comfortably. In a quality department the second is the only one you can use. The first pipeline puts roughly three wrong values in every hundred into signed certificates, invisibly, and you find them when a customer's incoming inspection does. The second puts a queue in front of an operator, and the certificates that come out the other side are ones somebody looked at.
Extraction with confidence scoring and rules
- Errors surface before the certificate exists
- Review effort concentrates on the uncertain minority of fields
- Unfamiliar mill layouts degrade visibly rather than silently
- Every value is traceable back to its place on the page
- The audit trail shows what was checked and by whom
Extraction optimised for a headline accuracy figure
- The remaining errors are indistinguishable from correct output
- Reviewing everything defeats the automation, reviewing nothing is unsafe
- A layout change degrades quality with no signal
- Errors are discovered by the customer, at the worst moment
- Trust collapses after the first incident and rarely returns
My take
If I could change one habit in how this software is bought, it would be the question asked in the first meeting. Everyone asks how accurate the extraction is. Almost nobody asks what happens when it is wrong.
The second question is the one that determines whether the system is usable. A pipeline that knows where it is unsure is a colleague. A pipeline that is confidently wrong 3% of the time is a liability with a subscription attached, and the liability is worse than the manual process it replaced, because the manual process at least had somebody reading the page.
The uncomfortable consequence is that a good system will look worse in a demo. It will stop and ask. It will send fields to review that a competitor's demo sails through. That hesitation is the feature. When we run our own pipeline against certificates it has never seen, the number I look at is not how many fields it got right, it is how many of the ones it got wrong it had already flagged.
Ask any vendor for that second number. The answer, or the absence of one, tells you what you need to know.
Our page on EN 10204 certificate automation shows the full pipeline, from ingestion to a generated 3.1. If you are working out which certificate type you owe in the first place, start with 3.1 or 2.2, and for the regulatory horizon see the Digital Product Passport for steel. To see it run on your documents rather than ours, book a discovery call.