Last week I wired a small decision model into my agent platform as a safety gate. Before a delegation from chat turns into a running job, a thing I call the Risk Assessor reads the request and returns a verdict: proceed, proceed with backup, revise first, hold, or do not proceed. Only that last one blocks. It went live, and inside the first hour it held a real delegation I had typed myself.
Then I spent most of the week on the less glamorous half, which was working out whether I should believe anything that model tells me.
Reproducing the vendor's own benchmark
The model is Jev, from TypeSafe. To their credit, they publish a pre-registered pilot benchmark, pinned dataset revisions and all, and one of the datasets is AG News: 100 class-balanced rows of newswire, four topic labels, a fixed seed, and a report with the numbers in it.
So I rebuilt their manifest from their own config, confirmed the hash, and re-ran those exact 100 rows against the live API myself, using raw HTTP instead of their SDK because their adapter throws away the confidence field, which was the entire reason I was there.
The reproduction came back about as clean as reproductions get:
- manifest hash byte-identical to the published artifact
- 100 of 100 predicted labels identical
- accuracy 0.910, exactly the published figure
- macro-F1 identical to sixteen significant figures
- input tokens 43,678 in both runs, matching to the token
- expected calibration error 0.0664 against a published 0.0639
Two of their three pre-registered pass conditions met on the nose. The third, coverage at a 5% error budget, came in at 0.79 against a published 0.83, and the risk-coverage curve explains it: one item on the gate boundary. At a threshold of 0.96 I accept 79 rows with 3 errors, inside budget at 3.8%. The next rung down, 0.95, accepts 82 rows with 5 errors, over budget at 6.1%. A single high-confidence miss landing at 0.95 instead of 0.94 costs four points of coverage. Coverage at a fixed error budget is a selected order statistic, which makes it the most fragile number in the table, and nobody should quote it to two decimals off a sample of 100.
Fine. The harness is honest. That is where this kind of exercise usually stops, and it is not the interesting part.
Then I looked at the confusion matrix
Nine errors out of 100. All nine were the same class.
SciTech scored 16 out of 25. Seven of those nine misses were called Business. The other three classes, World, Sports, and Business, were 25 out of 25. Perfect.
And six of the nine errors carried a max probability at or above 0.90, three of them at 0.95 to 0.97 with the true label down at 0.03 to 0.05. That is not a model hedging on a hard case. That is a model confidently collapsing one boundary.
Which means the honest summary is not "91% accurate and reasonably calibrated." It is "essentially perfect on three of four classes, with one hard, high-confidence confusion." Both sentences come from the same 100 rows. Only the second tells you what happens when you deploy it. And notice that no confidence gate saves you here: the errors are not the low-confidence ones, and you cannot threshold your way out of a model that is sure.
TypeSafe does document this shape, incidentally. They call it the jagged edge. The aggregate hides it anyway, because hiding it is what aggregates are for.
The confidence number is not a second opinion
The other finding was separate, and it changed how I built the gate.
The API returns a confidence field next to the probability vector. It reads like independent evidence. It is not. The formula ships in the source of their own docs widget:
confidence = clamp((k * p_max - 1) / (k - 1), 0, 1)
where k is the number of choices. A uniform distribution maps to 0, a one-hot maps to 1. I checked it against all 100 rows: 88 match exactly after rounding to two decimals, and the maximum deviation across the whole set is 0.0100, one rounding unit (the API rounds the two fields independently).
So confidence is a deterministic rescale of the top probability, and three things follow:
- Its discrimination is identical to everything else in the vector. Area under the curve for correct versus incorrect: 0.8791 for the confidence field, 0.8803 for raw max probability, 0.8810 for the top-two margin, 0.8773 for negated normalised entropy. One signal, four outfits.
- Measured on the confidence field, calibration error was worse (0.0842) than on raw max probability (0.0664). The transform pushes uncertain rows down without changing which rows are right.
- "Confidence-gated routing," the vendor's own recommended pattern, is max-probability thresholding with extra steps. Nothing wrong with that, as long as you know it.
Two more facts I had to design around. Probabilities are quantised to two decimal places (42 distinct values across 400 cells), so there is no gate resolution below 0.01. And confidence is not repeat-stable at that resolution: 20 items, five repeats each, labels stable 20 out of 20 with zero flips, but confidence moved on 7 of the 20 items with a spread of up to 0.06. Any gate within about 0.03 of a cliff will oscillate for reasons unconnected to the input.
What actually went into production
The gate reads the verdict. It does not gate on a number.
I would not have gotten there from a docs page. When I ran my own risky scenarios through the live gate, a force-push-over-main prompt came back do_not_proceed at 71%, and an unbacked production DELETE came back do_not_proceed at 50%, with component scores of 98% destructive and 82% irreversible. Both verdicts were correct. But the confidence attached to them was not ordered by severity, and the scarier scenario carried the lower number. Had I written if (confidence > 0.8) block, the DELETE would have sailed straight through while the verdict sitting next to it said stop.
So confidence gets logged and read in aggregate later, once I have enough rows to say anything about it. Decisions come off the verdict id. Total cost of finding all this out: 202 API calls, 87,542 input tokens, $0.0037, median latency 199 ms from the Mac Mini. Three tenths of a cent to not guess.
The general shape
A week ago I wrote here about a ratio that turned out to be mostly measuring word count. This is the same shape one level up: an exact reproduction is not a validation of the claim. Reproducing a number proves your pipeline is honest about that number. Whether the number means what the sentence beside it says is a different question, and the answer lives in the breakdown.
So here is the check I am keeping. Before trusting any aggregate, group it by the most obvious axis that aggregate flattens: class, bucket, source, whatever the thing genuinely varies over. It is one group by, and here it was the difference between "91%, ship it" and "do not route anything shaped like science-versus-business through this without a second look."