← FIELD NOTES
TECHNICALFeb 19, 2025·4 MIN READ

If you can't see it run, it's not in production.

Three things every system we hand over does on day one: emits structured events, redacts what shouldn't be logged, and signs every state transition. Here's why.

B
Brynn
FOUNDER, TRANSFORMATE

You'd never call a car roadworthy if you couldn't see the fuel gauge. No warning lights, no idea how much is left in the tank — just an engine that goes until, one day, it doesn't, with no warning you could have read. Nobody would drive that car. Yet teams put software live in exactly that state all the time: it runs, and the first sign of trouble is a customer phoning to say it stopped.

So here's a claim that sounds like wordplay and isn't: if you can't see a system run, it isn't really in production. It's just running somewhere you can't see.

"In production" gets used to mean "it's installed on the live server." That isn't the same thing. A system is in production when you can watch it work and prove what it did — not when it's merely switched on. Every system we hand over does three things from day one to earn that word.

The logbook

The first is a clear logbook. The system writes down what it did, as it does it, in a tidy and searchable form — not scattered notes, but proper entries: "at 14:32 it received order 5512, decided to hold it for review, flagged it for a human to check." When something looks wrong later, you read back exactly what happened. You don't squint at the code and guess. Most of the difference between a five-minute fix and a five-hour mystery is whether the system bothered to write down what it was doing.

The things it refuses to write down

The second is the opposite of the first: the logbook is deliberately blind to the sensitive stuff. Passwords, ID numbers, payment details — none of it lands in the log. This sounds like a detail and isn't. A logbook that quietly records secrets is a data leak with a timestamp on it: the moment a password gets written into a log, every person and tool that can read logs can now read passwords. So we decide, up front, exactly what's safe to record, and the system records only that.

The signature

The third is that the system signs its important steps. Every time it changes something that matters — money moved, a status flipped — it notes who or what made the change, in a form nobody can quietly edit afterwards. The point is proof. Months later, when someone asks "did this actually happen, and who did it?", the honest answer is a record you can trust, not a shrug.

A system you can't watch isn't in production. It's just running where you can't see it — and "it's running" is not the same as "we're operating it."

Why all three on day one

None of this is expensive to build in at the start. It's a logging habit, a short list of what's safe to record, and a tamper-proof note of the important moments. It's brutally expensive to add later, because after the fact you're trying to reconstruct a history you never kept. The night something breaks at 2am is the wrong night to discover the system never wrote anything down.

This is also why we won't hand over a black box that merely works in the demo. A thing that runs but can't show you what it's doing is asking for your trust on faith. Faith is a fine thing. It's a poor operations plan.

Under the hood

The three plain-English habits map to three concrete practices.

Structured events. The system emits machine-readable event records — think one tidy JSON line per event, sent to a log store or an events table — rather than free-text print statements. Each event carries a correlation ID so a single request can be followed across every service it touches, plus a standard shape: timestamp, actor, action, entity, outcome. Free text is for a human to read one line at a time; structured events are for you to query a thousand at a time when you're hunting the one that went wrong.

Redaction by allowlist, not denylist. We define the fields that are safe to log and emit only those, rather than maintaining a blocklist of secrets to strip. A denylist fails silently the first time someone adds a new sensitive field nobody remembered to ban; an allowlist fails closed — a field you didn't explicitly clear simply never gets logged. Secrets and personal data are scrubbed before anything reaches the sink, and full request or response payloads never get logged on reflex.

Signed state transitions. Every meaningful change of state emits an event whose identity is asserted by the part of the system that made the change — stamped at the source, not guessed by whatever reads it later — and written to an append-only audit trail. Append-only means entries can be added but not quietly rewritten, so the history carries integrity you can actually lean on in a dispute or an incident review.

The honest test for whether something is in production: can you answer "what did the system do at 14:32 last Tuesday, and can you prove it?" If you can't, it isn't being operated. It's just switched on.

IF YOU LIKED THIS

Want one of these in your inbox once a month?