On July 28th one of my apps told me a job had failed. It said so twice, in a modal, in plain language: "Generation timed out. Please try again."
The job had not failed. The job had not even started. It was sitting in a queue, healthy, attempt count zero, waiting its turn behind a long-running local task. The only thing that had actually expired was a setInterval in a browser tab that had been counting to five minutes and had no idea what it was counting.
I spent the last week fixing this, and it turned into three separate changes. Two of them were systems work. The third one was the one that mattered, and it consisted mostly of deleting code.
What the queue was actually doing
The setup: I run a Mac Mini at home that executes AI jobs for three different front ends. My site builder enqueues layout, theme, SEO, legal, and logo generation. My writing app enqueues book-builder passes. My agent enqueues whatever I have asked it to do that day. All three land in the same jobs table, and a worker process claims from it.
The worker claimed one job at a time, strictly FIFO, ordered by created_at. That is the most obvious possible policy and it is fine right up until the moment the three classes of work have wildly different latency budgets.
A layout suggestion takes about a minute and a human is staring at a spinner the entire time. A book-builder pass can hold the slot for forty minutes. A local agent job doing data work on my Colorado business-filings database routinely runs past thirty. FIFO does not know any of that. FIFO just knows who got there first.
So on the 28th, a local project job claimed the slot at 14:43:34 and was still going at 15:02. The layout job I triggered arrived a couple of minutes later and did exactly what it was supposed to do: nothing, quietly, in line. And the modal, which polls for five minutes and then gives up, announced a failure.
Then I clicked retry, and it got worse. The retry hit a duplicate guard on the server, correctly re-attached to the same pending job, and started polling it again from zero. From where I sat, the button did nothing. Twice.
Nothing on the Mini looked broken because nothing was broken. This was starvation, not a fault, and the UI had no vocabulary for the difference.
Fix one: stop being FIFO
The first change was the cheap one. Priority-ordered claim, still one job at a time, no schema change: site-builder jobs first, then writing-app jobs, then local agent jobs. Interactive work jumps the line.
That fixed the queue. It could not fix the case that actually bit me, because ordering only decides who goes next. It cannot touch a job that is already running. A forty-minute pass still blocks everything behind it for forty minutes.
Fix two: a slot that is always free
The real fix was concurrency two, with one slot effectively reserved for jobs that did not originate on the Mini. If a local job is running, the worker will only claim external work. Two local jobs never run together. Interactive work always has a lane.
There was one thing standing in the way, and it is my favorite part of this whole week. My own CLAUDE.md had a warning in it, written months ago, saying not to run concurrent claude --print calls because it can invalidate the machine's login session. That warning is exactly what made a second slot look risky.
While doing recon I noticed the warning was already being violated, constantly, by my own code. Tier-1 chat spawns claude --print from the web server every time I type a message, concurrently with whatever the worker is doing. Plus a login probe every five minutes. Same login, no invalidations, for months.
So I ran the experiment before writing the feature. Three rounds of deliberately concurrent runs, one of them a triple that included a real production job, watching the health-check log the whole time. All clean. The warning was stale, and it had been quietly shaping my architecture decisions from a file nobody had reread.
I deleted it and built the slot. The end-to-end test on the Mini: a long local job running, an external job claimed six seconds later through the reserved-slot filter, both finishing clean, zero stray tmux sessions.
Fix three: stop lying
The third change is the one I keep thinking about, because it is the only one that addressed the actual defect.
Here is what I found when I went through the five modals. Each had reimplemented its own polling loop. Four of them had a five-minute client-side timeout measured with Date.now() in the browser. One had no timeout at all and would spin forever. None of them ever asked the worker what the state of the job was, even though a perfectly good status endpoint had existed the whole time and one other feature was already using it.
Meanwhile the system's real timeout, the one keyed on actual progress, was sitting in the worker: an hourly sweep that fails rows with no movement in over an hour. That one is honest. It measures the thing it claims to measure.
So the five-minute browser timer went away entirely. In its place: one shared polling hook, and while the job is non-terminal the poll also asks the worker where the job stands. If it is queued, the modal says "Waiting for an earlier job to finish." No progress bar, no invented percentage, no countdown to a fake death. If it is running, you get progress. If it genuinely failed, you hear about it immediately instead of at the five-minute mark.
After a couple of minutes in line, the message becomes a version of "this keeps running in the background, you can close this window." Because it does, and you can. And when you reopen and the duplicate guard re-attaches, the modal now says so instead of pretending to start fresh.
I also found, during that same pass, that one of the five features had no duplicate guard at all. Every click was enqueuing a whole new job. That bug had been there for months and was invisible precisely because the UI was already lying about everything else.
The thing I want to remember
The first two fixes made the system faster. Neither would have prevented what I experienced on the 28th, which was being told something false. The layout job would have run sooner, sure. But the class of bug, a client-side timer asserting a failure it has no evidence for, was still there and would still fire the next time the queue got deep.
A progress indicator is a claim about system state. If it is not derived from system state, it is not a progress indicator. It is decoration that occasionally accuses your backend of crimes it did not commit.
The honest version was less code than the dishonest one.