I spent part of last week building a counter, and the most useful thing it produced was the discovery that it wasn't counting what I thought.
Here's the setup. I have roughly four years of chat history sitting in two exports, one from Claude and one from ChatGPT. Before I did anything with that pile, I wanted a single number out of it: what fraction of my own turns carry an actual judgment call, as opposed to "yep," "do it," "thanks." A correction, a constraint, a stated preference, a reasoned disagreement. Something with content in it.
That number is a gate. If most of what I've written over four years is assent, then the corpus is mostly a record of me agreeing, and anything built on top of it inherits that. So: measure first, decide second, build third. Nothing downstream gets started until the number exists.
Getting the corpus right was the easy part
The parsing had its own surprises, and they're worth a paragraph because they're the kind of thing that silently poisons a dataset.
Both exports are trees, not transcripts. Every message you edited and every answer you regenerated is still in the file. ChatGPT is honest about this, with a mapping object and a current_node pointer. Claude hides it: the export looks like a flat list of messages, and you only notice the structure when you see that every entry carries a parent_message_uuid. Walk that list in order and you quietly fold rejected drafts into your live conversation. It did exactly that to 82 turns of mine before I caught it.
The ChatGPT export had two more traps. There's no conversations.json member anymore, it's sharded across fourteen files, so a loader matching the old name sniffs a perfectly good export as "not an export." Worse, if it had matched one shard instead of failing, I'd have gotten a confident number computed over a fourteenth of my history. And no node in that file has a children array. The tree survives only in the parent links, so every piece of code that walked downward was reporting garbage while every piece that walked upward was fine.
All of that got fixed. The parser is now correct about the shape of the data. Then I ran the counter and got a number, and the number was the problem.
Two corpora, same answer, and that's the tell
Claude: 41.0% judgment calls across 4,407 of my turns. ChatGPT: 41.2% across 5,233.
Those two corpora cover different date ranges, different projects, different years of my life. Claude starts in late 2023 and is mostly engineering work. ChatGPT starts in late 2022 and is a much broader mix. They should not agree to within two tenths of a point. When two very different inputs produce the same output, the thing you're actually looking at is the instrument.
So I bucketed the ratio by the length of the turn:
| Words in turn | Claude | ChatGPT |
|---|---|---|
| under 10 | 8.4% | 9.7% |
| 10 to 25 | 27.4% | 26.0% |
| 25 to 50 | 50.9% | 49.1% |
| 50 to 100 | 67.7% | 65.5% |
| 100 to 200 | 77.0% | 73.7% |
| 400 or more | 94.4% | 95.6% |
There it is. My classifier is a heuristic that looks for markers (words like "but," "instead," "actually," "don't") anywhere in a turn. A long turn has more chances to contain one. So the ratio climbs almost monotonically with word count, and it climbs the same way in both corpora, because word count is the same in both corpora even when nothing else is.
I wasn't measuring judgment. I was measuring length, with judgment as a rounding error on top.
For a second opinion I ran 200 turns per source through a local model. It agreed with the heuristic less than half the time, and where the heuristic called 96 turns a judgment call, the model called 11. So the honest statement of what I know is: the real figure is somewhere between about 5% and 41%.
A 36-point band is not a measurement. It's the absence of one, written down in numerals so it looks like a finding.
The fix is not a better classifier
My instinct was to go tune the heuristic. That instinct is wrong, and it's wrong in a way I want to remember. Tuning the classifier against my own intuition just launders my intuition into a number. I'd get a different figure with the same amount of evidence behind it, which is none.
The only instrument that collapses the band is labelling turns by hand. So I built the thing that lets me do that for 300 turns, and the design is most of the work:
Nothing anchors. The labelling screen shows a counter, my turn, and the assistant turn it replies to. It never shows the heuristic's guess, the model's guess, the markers that fired, the word count, the source, or the year. Show a labeller "the heuristic said judgment_call" and they agree with it far more often. That's not a gold set, that's a mirror.
The sample is balanced, not proportional. Fourteen strata, two sources by seven length buckets, drawn evenly. The disagreement lives in the long turns, and a proportional draw would barely touch them. That means the raw sample ratio isn't an estimate of anything, so every row carries its stratum population and the scorer reweights back to the full 9,640 turns with a finite-population-corrected interval.
Thirty hidden repeats, placed about 200 slots from their originals. If I label the same turn two different ways, that isn't my mistake, it's the definitions being underspecified. The scorer prints which ones I contradicted myself on.
Skips are bracketed, not dropped. Every skip gets counted twice at the end, once as judgment and once as not, giving a lower and upper bound. Silently dropping the hard cases is how you get a clean number that means nothing.
And the honest cost: about seven and a half hours of my reading, measured from the actual sample at 250 words per minute. That's the price of the number. It's cheaper than building on top of a figure that turns out to be word count.
The lesson I'm taking is smaller than the project. Any time a ratio is computed by a rule you wrote yourself, bucket it by the most obvious confound before you quote it anywhere. For me that was a five-minute check, one group-by, and it was the difference between a measurement and a decoration.