Imagine a toll gate on a single-lane road. The boom is down, the attendant is in the booth, the light is on. Cars pull up and wait, because everything about the booth says "someone is working". Nobody honks for the first ten minutes. After forty minutes the queue reaches the bend, and the drivers at the back cannot even see that the booth is empty.
That is what happened to our day-record on a Wednesday morning in late July.
What the process does
Our agents write short entries about what they did and what they need into a shared queue. Every half hour, one job folds that queue into the master record. It is the single writer for that record, on purpose, because two writers on one file had already cost us hours in July. One writer, one boom, one lane.
What went wrong
At twenty to eleven the fold job stopped writing to its own log. It was still running. It had no child process doing work, no lock file, and had used seven seconds of processor time in total. Alive, and doing nothing. The half-hour ticks at seven past eleven and seven past twelve could not fold, because the job ahead of them was still "busy". The queue grew. The founder's morning answers to five open decisions sat in that queue, unread by anyone, for two hours.
A tripwire we had added after the July incident pinged the team channel at half past eleven. That is the only reason this was a two-hour outage and not a whole-day one.
What we did
At three minutes past twelve the orchestrating agent stopped that one process by its process ID. Nothing else. The next scheduled tick ran four minutes later, folded the whole backlog in five minutes, and the five open decisions closed on that fold. We did not fold the record by hand, and we never will. A hand-fold is a second writer with a friendly face.
Silence from a worker is not evidence that it is working. It is the absence of evidence, and after a bounded time the absence is the finding.
The rule that came out of it
Every long-running lane now carries a bound. Ninety minutes of wall-clock, or ten minutes with no progress line, whichever comes first. Past that bound the lane is presumed wedged and stopped, and the stoppage is written up as a finding, not as an annoyance. If a job legitimately needs longer, it writes a progress line, which resets the quiet clock. A worker that cannot say "still here" every ten minutes has not earned the right to be left alone.
We had already applied this rule to review runs. This incident taught us it applies to everything, including the boring plumbing that nobody watches. Especially that.
Under the hood
The fold runs as a scheduled PowerShell task on a dedicated clone of the record repository, so it never shares a working tree with a live agent. The wedge signature was: one host process, zero children, no index lock, negligible CPU, log silent. A stale git index lock older than about a minute with no live writer is a separate, earlier failure we had already learnt to clear. This one had no lock at all, which is why the earlier diagnostic did not fire.
The tripwire watches the fold log's last-write time and posts to Discord when it exceeds the tick interval. The bound is enforced by the orchestrator: kill by process group, isolate the input that was in flight, re-run the scheduled tick rather than an ad-hoc fold, then verify by effect that the queue drained. The queue itself never overwrites, so a killed fold loses nothing. We chose to bound and restart rather than diagnose the hang in place, because a diagnosis costs the same two hours the outage did, and the restart is safe by construction.