Measure Extraction Accuracy Before You Promise Anything

When a company asks us whether AI can read their incoming requests and turn them into orders, quotes or tickets, the honest first answer is: probably, but we do not know yet. We do not know how well, on which fields, or how often it will be quietly wrong. Neither does anyone else who has not looked at your messages.

So the first step is never code. It is a measurement.

Two hundred real messages and a spreadsheet

We take a random sample of real messages from the last month or two. Random matters here. If the sample is picked by the person who knows the inbox best, it will be the tidy ones. Two hundred is usually enough to see patterns without turning the exercise into a project of its own.

Then someone who knows the process writes down, for every message, what the correct values are: customer, product, quantity, material, delivery date, whatever the next step needs. This is the slow, unglamorous part, and it is where most of the learning happens. People discover that a good share of requests do not contain a delivery date at all, or that "the usual" means something different for every customer.

Only then do we run the extraction and compare.

Measure per field, not per document

A single accuracy number is close to useless. What you want is a small table: for each field, how often the model got it right, how often it got it wrong, and how often it said it was not sure.

• Right: the extracted value matches what your colleague wrote down.
• Wrong: the model gave a confident value that does not match.
• Flagged: the model marked the value as uncertain or missing.

The "wrong" column is the one that matters. Flagged values cost a little review time. Confidently wrong values cost you a wrong order, a wrong price, or a customer who receives something they never asked for. A setup with a lower headline accuracy but almost nothing in the wrong column is often the better system.

The numbers change the conversation

Before the measurement, discussions tend to be about whether AI "works". After it, they are about specific things. Material codes are reliable. Delivery dates are not, because customers write them in six different ways, so we extract them but always show them for confirmation. Quantities are fine, except when the message refers to an attached drawing, so those go to review.

That is a much better conversation to have with an operations manager. It is concrete, it can be tested, and it does not require anyone to believe a promise.

We do not promise before we measure. It sounds like caution. In practice it is what makes the project move faster.

Keep measuring after go-live

The sample is not a one-off. Every correction a person makes after go-live is a new data point, so keep them. Re-run the measurement when the model or the prompt changes, and when the business changes: a new product line, a large new customer with their own way of writing. Accuracy that held in March can drift by September without anyone noticing, unless someone is counting.

None of this is sophisticated. It is close to what a quality engineer does with a new measuring instrument: calibrate it against known values before relying on it, and check it again every so often. That is precisely the point.