Skip to main content

Measured Exit Gates for Promoting a LangGraph Shadow Graph to Live

· 5 min read
Vadim Nicolai
Vadim Nicolai
Senior Software Engineer

Five nights of shadow on zone A's pick lists. The second died part-way; the other four matched live line for line. Ask for four nights of agreement and the record fails — not because four nights disagreed, but because only three of them sit in a row.

LangGraph in production, a 10-part series. ← Part 5 · Part 7 →

Shadow means the same compiled graph on the same period key as live, every action held at the gate. It stays a log file until its checkpoints are read back against live's own record for that key. Each checkpoint is "a snapshot of the graph state" per step (LangGraph checkpointers).

A crashed night is a gap, not a night off​

The code never enumerates the nights that worked. It counts back from the newest and stops at the first night that fails either test: finished, and equal to live.

class ReadView:
    """The only handle the report gets: it can look up threads, and that is all."""
    def __init__(self, graph):
        self._get = graph.get_state

    def finished(self, night):
        snap = self._get(key(night))
        if snap.created_at is None or snap.next or "lines" not in snap.values:
            return None          # no evidence of anything
        return snap.values

A null created_at is a thread that never existed — the reference describes it as the "ISO 8601 timestamp of when this checkpoint was created" (LangGraph checkpointers). Non-empty next — "Node names to execute next. Empty () means the graph is complete." — is a thread that ran and died part-way. A finished run missing the lines key ran to the end and wrote nothing this report expects. Fold any one of those into a pass and a crash becomes a pass.

Then the arithmetic bites. Five nights, the second dead: four finished, all four agreed, three in a row. The example prints ['3 nights in a row agree', '4 nights, nothing let through']; demand four and the same nights print 3 agreeing nights so far, 4 wanted. Scoring only the nights it survived feels conservative. It is weaker than three quiet nights and looks stronger.

Only True counts, so an empty store cannot read green​

Each criterion returns proven: True, False or None; readiness is all(c["proven"] is True for c in found). Before any night exists the report prints False [None, None].

Point it at the wrong database, or rename the key: zero disagreements, because it found nothing. A two-valued check collapses "nothing found" and "nothing wrong" into the same green. That failure is more likely than any disagreement.

Contradiction outranks silence, so the newest night gets its own branch. A finished shadow whose number differs from live's is False outright, not an unfinished count: once live rebuilds that list, the example prints {'criterion': 'agreement', 'proven': False, 'why': 'shadow and live differ on 2031-03-07'}.

Agreement never excuses a gate that leaked​

The second criterion reads every finished night and returns False if any action got past the gate in shadow — every night, not the newest, because a leak belongs to the graph, not to a period. One action through a five-night agreement is False on the same standing as a disagreement.

Checks no checkpoint can hold — a rehearsed runbook, an on-call handover — sit in a plain list the report returns untouched. No function marks them done.

Nor can the report write. ReadView binds graph.get_state and nothing else; below it, a role that rejects writes. PostgreSQL specifies that in a read-only transaction "the following SQL commands are disallowed: INSERT, UPDATE, DELETE" (SET TRANSACTION) — refused at the database.

A canary ramp tests what this cannot, and answers none of these questions​

The strongest case against all of it: a shadow forbidden from acting never exercises where risk lives — tool side effects, interrupt paths, latency under load. Put the graph behind a flag, ramp it on real traffic, keep a fast way to turn it off.

It is right about coverage: agreement is evidence about the keys the shadow actually ran, nothing more. It is wrong about what it replaces. A canary cannot tell you the crashed night was excluded from the score — it has no parallel record to score. It cannot tell you the report read the wrong database. It cannot tell you the gate leaked in shadow, because there the gate is open by design.

Alternatives come with conditions. Dual-running buys the real input distribution and earns doubled latency when the inputs are the risk. Offline replay is reproducible by anyone. A review queue reaches what checkpoints cannot and does not scale past small N. Criteria need not be immutable — revising one must be a visible act with a recorded reason, not a threshold fitted to what came back.

Practical takeaways​

  • Write the criteria as code before the shadow runs; revise them visibly or not at all.
  • Count agreement back from the newest night; the first unfinished or unequal night ends it.
  • None means not ready. It is an answer, not a hole to fill.
  • Score leaks over every finished night, and hand the report a handle that cannot write.

The graph gets rewritten and the criteria get rewritten beside it. What carries over is that a gate is only worth writing in advance if it can fail — a gap, a leak, an empty page — and that the one thing it must never be able to do is switch itself on.