Imagined failure narrative
It is roughly a month after launch. The deployment failed — not loudly, but by erosion. The system kept producing output, so nobody pulled the cord early. But the oversight everyone was told was watching had been running in simulation the whole time; the verifier kept catching errors that shipped anyway; a concurrent process moved the server off the port the campaign was pinned to and stranded a run nobody noticed for hours; the diagrams the deliverable was supposed to carry had quietly degraded to prose; the Paused queue filled with so many escalations that the human stopped opening it by week two; and a model deprecation swapped a slot to a lower-quality fallback that no one saw. By the time anyone audited the deliverables against ground truth, trust was already gone — including for the work that was actually fine. The post-mortem found every one of these visible in the architecture before launch.
Before the inventory, the frame this post-mortem rests on — correct any load-bearing assumption and the inventory shifts:
- “Launch” = the first time the full stack ran live simultaneously on a real campaign at volume — oversight daemon, meta-layer, vision, RAG, multi-model adversarial routing — not a public software release. The premium campaign already running and halted at 117/196 (2026-06-15) was the closest live precedent. (The 117/196 figure is confirmed against the package’s own state note; it is a private system-state fact, not externally verifiable, and is kept as-is.)
- “Failure” = users lost confidence and disengaged, or hit show-stopping silent regressions — a credibility collapse by attrition, not a crash. Most pathways below were quiet-degradation.
- Decision horizon ≈ 30–60 days post-commit, written as a completed post-mortem.
- Reconstruction caveat: every mechanism is anchored to a documented design choice in the codebase / CLAUDE.md / memory index, but the projection that it caused the launch to fail is inference, not observation of live state —
human-queue.jsonl, the active models.json, the current ORA_OVERSIGHT_LIVE are not visible from here. Confidence is marked per finding.
Two competing claims about which failure was most load-bearing are both preserved, because the disagreement is itself information about which premise the team thinks is most fragile:
- Claim A — the coordination / self-inflicted class is the likeliest to recur: port-drift (EX-1), config-drift (EX-3), and single-operator-overload (MO-1) have already happened more than once in the memory trail, making them the highest-probability launch-scale recurrence — joined by two trust-axis preconditions (simulated oversight, confidentiality) that are go/no-go, not tuning knobs.
- Claim B — the human-in-the-loop inversion plus the silent-degradation throughline is the killer: queue flood (IN-1) is the classic and most probable killer because it inverts the entire oversight premise, and the silent-failure throughline (MO-4) is what collapsed confidence across the board, even on clean output.
Failure mode inventory
EX-1 — Port drift stranded the live campaign — class: execution. Plan-specific mechanism: parallel sessions pkill + nohup the shared :5000 server; a 5000→5001 drift stranded a campaign pinned to 5000.
EX-2 — Vault↔ora drift silently reverted live behavior mid-run — class: execution (class-disputed — see tension T1, also classifiable as assumption). Plan-specific mechanism: an ora .md edited during the launch push without vault propagation, then overwritten by the next vault→ora sync because the vault is canonical and had three concurrent writers under load.
EX-3 — Production ran a different config than the repo — class: execution. Plan-specific mechanism: config/models.json / routing-config.json are machine-specific/gitignored/auto-managed; a redeploy or teammate pull reset hand-tweaked routing to repo defaults, silently re-pointing slots.
EX-4 — The verifier caught errors that shipped anyway — class: execution. Plan-specific mechanism: the Step-6 verifier is a gate, not an editor — its ### Verified Final Output block is captured in the trace but never propagated; the consolidator reads the reviser’s draft directly, so a corrected date or citation never reached the user.
EX-5 — The oversight daemon wedged and re-armed silently — class: execution. Plan-specific mechanism: the 2026-06-12 wedge shape (maintenance scheduler re-arming because no stamp landed; fast-lane >300s / slow-lane >2h watchdog) recurred during the campaign, and the only signal was a warning prepended to the next chat response a headless run never surfaced.
EX-6 — A project-plugin tool invocation failed and stranded a campaign step — class: execution. Plan-specific mechanism: frameworks reference project tools by abstract name resolved at runtime via invoke_project_tool (subprocess, JSON-on-stdout); a tool emitted non-JSON on stdout or an abstract-name resolution missed, and the step ran without its data.
AS-1 — Oversight ran SIMULATED; users trusted governance that never happened — class: assumption (class-disputed — see tension T2, also classifiable as execution). Plan-specific mechanism: the oversight router defaults to simulated Process-Coherence invocations; ORA_OVERSIGHT_LIVE=1 is required for live calls. The daemon ran, the sidebar showed Paused/Operating panels and Decision Log entries — and none of the verdicts were real.
AS-2 — Embedding/collection mismatch made RAG return noise or nothing — class: assumption. Plan-specific mechanism: the bge-m3 (1024-dim) migration over nomic (768-dim) with machine-specific config/chromadb.json; a collection carried over a Mac↔server round-trip under a stale mapping returned low-relevance or empty neighbors.
AS-3 — The pre-routing pipeline misrouted prompts, invisibly — class: assumption. Plan-specific mechanism: Stage 1 does substring matching against the signal-vocabulary registry; messier real prompts bypassed the analytical pipeline or dispatched to a subtly-wrong sibling mode, announced only as an “educational parenthetical.”
AS-4 — “$1K consumer hardware” was true for cost, false for quality — class: assumption. Plan-specific mechanism: the qwen-9b-only config drives the reproducibility demonstration, but real quality lives on the 60–80 GB models plus adversarial Gear 3/4; the marketing config and the demo config diverged.
AS-5 — Content users assumed local-only crossed a confidentiality boundary at scale — class: assumption. Plan-specific mechanism: the campaign indexed content into ChromaDB conversations/project collections and vault_export.py wrote to ~/Documents/vault/Sessions/, a path with cloud-reaching writers (claude.ai sessions, DCP routine) plus side-channel sync.
AS-6 — Two large models exceeded RAM — class: assumption. Plan-specific mechanism: CLAUDE.md states two large models exceed 128 GB; the active config’s Gear-4 depth and breadth cells both bound to a 60–80 GB slot at once, driving swap thrash. (Downgraded pending confirmation of whether active_configuration.py / routing-config.json already forecloses that assignment.)
CS-1 — A model slot was knocked out or silently mispriced by provider churn — class: context-shift. Plan-specific mechanism: OpenRouter rate-limited the campaign and a slot’s model was deprecated/removed from the registry mid-run; fallback chains engaged at lower quality (the 117/196 shape). Pricing sub-case: the documented video-pricing $0.0/$0.0/M misread shows the registry can misread provider economics, so a pricing/endpoint-schema change silently dropped a capability.
CS-2 — A model deprecation silently changed which model filled a slot — class: context-shift. Plan-specific mechanism: the DEPRECATED flag is a UI signal; during the automated campaign no human watched the Models pane, so a primary→fallback substitution happened unobserved at quietly lower quality.
CS-3 — The paid-tier budget exhausted mid-campaign and forced a silent downgrade — class: context-shift. Plan-specific mechanism: the premium campaign already required an Opus refill; at-scale spend (fan-out × retries × throttling) outpaced projection, hit the ceiling mid-run, and with no pre-decided downgrade path the run halted or auto-fell to cheaper models.
IN-1 — The oversight queue flooded and the human stopped reading it — class: interaction. Plan-specific mechanism: every milestone fan-out, DRIFT_DETECTED, and ESCALATE landed in human-queue.jsonl and the Paused supergroup, plus cross-project parent-side Decision Log entries; at campaign scale this was dozens-to-hundreds of cards and the human hit alert fatigue.
IN-2 — Adversarial visual review degraded every diagram to prose; the vision system “didn’t work” — class: interaction. Plan-specific mechanism: the visual adversarial layer forces fallback to table/prose on any Critical finding (overlap / truncation / WCAG contrast) per Protocol §8.5; on heterogeneous campaign content the threshold tripped often enough that users mostly saw prose. Precedent: V3 historically rendered envelopes as raw JSON (“V3 visual-render gap still open”). Separately, vision-extraction failures dropped image inputs silently.
IN-3 — Health warnings contaminated responses — class: interaction. Plan-specific mechanism: oversight_health.py injects daemon_down/stale warnings into the next AI response; in interactive use this read as “the system is malfunctioning,” and in a campaign emitting artifacts the warning text baked into published output (or, if stripped, the human never saw it).
IN-4 — Conversation-as-state markers were lost, stranding mid-framework runs — class: interaction. Plan-specific mechanism: elicitation/resolution chains carry state only in HTML-comment markers in the transcript; a render pass, stealth-conversation purge, or compaction event dropped the marker and the next turn no longer detected continuation.
IN-5 — Retention/archival made live conversations vanish — class: interaction. Plan-specific mechanism: the retention sweeper’s session archival is off by default because archived sessions disappear from the sidebar; an operator set ORA_RETENTION_SESSIONS_DAYS nonzero during launch cleanup and active campaign conversations left the sidebar mid-run, and rotated JSONL sinks the stealth-purge depends on.
MO-1 — Single-operator complexity exceeded sustainable attention; parallel sessions stepped on each other — class: motivational. Plan-specific mechanism: ~14 oversight modules, a 358-model registry, a visual compiler, multi-writer vaults, a watchdog daemon — an operational surface exceeding one person under load; the tell was that capability kept being added while operational hardening stayed informal.
MO-2 — Unfinished scaffolding shipped as if complete — class: motivational. Plan-specific mechanism: the Tier 2 question-bank modules (6 files) are still under “Remaining Work”; Stage 3’s input_contract elicitation degraded to generic prompts for modes whose front-end was never finished — a known gap deprioritized under launch pressure.
MO-3 — Theatrical oversight → the user stopped trusting the meta-layer entirely — class: motivational. Plan-specific mechanism: AS-1 (simulated verdicts) + IN-1 (queue flood) + IN-3 (noisy warnings) compounded so the user concluded the apparatus was ceremony, disengaged, and therefore caught none of the real drift it was built to catch. (Citation correction: “Klein’s theatrical pre-mortem failure mode” is the analyst’s own coinage — Klein originated the premortem in 2007, but no named Klein concept by that term is verifiable; retained only as the analyst’s characterization, not a Klein attribution.)
MO-4 — Silent failures destroyed the ability to tell good output from bad (the throughline) — class: motivational. Plan-specific mechanism: EX-2, AS-2, EX-4, EX-6, CS-1/CS-2, IN-2 share one property — they fail without saying so; the user couldn’t distinguish a clean deliverable from a degraded one, so confidence collapsed across the board, including work that was fine.
Causal pathways to failure
EX-1 → breakage point: a second operator/automated task restarted the server on the wrong port mid-run. Immediate consequence: the pinned client’s requests black-holed. Cascade: throughput went to zero → the run read as “stuck.” Surfaced failure: operator misdiagnosed it as a model/quality problem — the exact misread that already occurred once.
EX-2 → breakage point: a fast ora edit during the launch push with vault propagation skipped. Immediate consequence: side-channel sync rewrote the file, the ora edit gone. Cascade: runtime loaded the stale mode → degraded output. Surfaced failure: user distrust with no visible cause; the system’s behavior changed under the campaign’s feet with nothing flagging it.
EX-3 → breakage point: live machine routing had hand-tweaks never reconciled; a redeploy or pull reset to repo defaults. Immediate consequence: slots silently re-pointed at different models mid-campaign. Cascade → surfaced failure: quality shifted with no deploy to explain it.
EX-4 → breakage point: the verifier corrected a fabricated date or broken citation. Immediate consequence: the correction lived in the trace; the uncorrected text reached the user. Cascade: the load-bearing trust damage rests entirely on this non-propagation. Surfaced failure: users received known-uncorrected text. (Conditional, unconfirmed: if a session export surfaces the internal “VERIFIED” verdict as a user-facing label, the damage compounds — users trust a stamp over uncorrected text — but the finding bites without this second claim; whether the verdict is ever rendered to a user cannot be confirmed from the package.)
EX-5 → breakage point: a sweep wedged during the campaign. Immediate consequence: watcher heartbeats went stale. Cascade: the only signal was a warning prepended to the next chat response. Surfaced failure: a headless campaign run never surfaced it to a human.
EX-6 → breakage point: a tool (MSI’s index_msi_news or fred_api) emitted non-JSON on stdout, or an abstract-name resolution missed. Immediate consequence: the dispatcher’s JSON parse failed, or the framework step ran without its data. Cascade: the subprocess boundary surfaced the failure as an opaque step error far from its cause. Surfaced failure: confidently-shaped output built on missing inputs.
AS-1 → breakage point: launch shipped with the daemon on and the simulated default. Immediate consequence: realistic-looking Decision Log / escalations with no model behind them. Cascade: operator acted on a PROCEED no model produced → a bad call shipped under false assurance. Surfaced failure: on discovery, all prior oversight output was retroactively suspect → trust collapse.
AS-2 → breakage point: stale/missing chromadb.json mapping. Immediate consequence: a collection bound to the wrong embedder/dim returned irrelevant or zero neighbors. Cascade: the analyst grounded on noise or ran context-starved → confident-but-wrong output. Surfaced failure: user caught one factual miss and distrusted all outputs — the confabulation class the system was built to avoid.
AS-3 → breakage point: real-world phrasing hit a substring miss or over-match. Immediate consequence: a wrong mode_id. Cascade: competent analysis of the wrong framing → user couldn’t articulate why it was off. Surfaced failure: quiet abandonment; no error fired because the pipeline succeeded at the wrong task.
AS-4 → breakage point: the launch marketed cheap-hardware reproducibility while impressive demos ran on premium models. Immediate consequence: early adopters on the 9B config got a materially weaker system than promised. Surfaced failure: “this doesn’t live up to it.”
AS-5 → breakage point: an at-scale campaign put third-party/sensitive content through the system. Immediate consequence: content indexed into ChromaDB + exported to vault Sessions. Cascade: vault paths reachable by cloud writers / side-channel sync → content left the local boundary. Surfaced failure: a user discovered their campaign material had left their machine → categorical trust loss (privacy breaches don’t average out).
AS-6 → breakage point: Gear-4 depth and breadth cells both bound to a 60–80 GB slot on the 128 GB M4 Max. Immediate consequence: swap thrash. Cascade: latency collapsed from seconds to minutes. Surfaced failure: throughput assumptions broke — an OOM-class stall discovered at turn 400, not turn 1.
CS-1 → breakage point: external rate-limit / deprecation / pricing change. Immediate consequence: a primary slot model became unavailable or mispriced. Cascade: fallback engaged (lower quality), stalled, or a capability silently dropped. Surfaced failure: throughput + quality degraded; the campaign read as “stuck and getting worse” (the premium run sat at 117/196).
CS-2 → breakage point: a slot’s primary model was deprecated and dropped from the registry. Immediate consequence: the DEPRECATED flag fired in a UI no one was watching. Cascade: a fallback substituted silently → quality dropped with no surfaced cause. Surfaced failure: degradation wrongly attributed to the system’s reasoning, not a model swap nobody saw.
CS-3 → breakage point: at-scale spend (fan-out × retries × throttling) outpaced projection. Immediate consequence: the budget ceiling hit mid-run. Cascade: halt or silent downgrade to cheaper models. Surfaced failure: an incomplete or quality-drifted deliverable — “it stalled / got worse near the end”; the premise “we provisioned enough” was wrong because per-run cost was never measured at scale before commit.
IN-1 → breakage point: every fan-out, DRIFT_DETECTED, and ESCALATE landed in the Paused queue at campaign scale (dozens-to-hundreds of cards). Immediate consequence: the human hit alert fatigue. Cascade: stopped opening Paused by week two → the human-in-the-loop premise inverted. Surfaced failure: the safety feature became noise the user routed around; oversight succeeded mechanically and failed at its purpose.
IN-2 → breakage point: a strict Critical gate met messy real content (or a systematic envelope defect in one mode). Immediate consequence: frequent fallback to prose/table. Cascade: the user rarely saw a rendered diagram. Surfaced failure: concluded the headline feature was broken → discounted the whole system. (Vision-extraction failures separately dropped image inputs so multimodal turns ran text-only without flagging the loss.)
IN-3 → breakage point: a watcher went stale (a real recurring state per the 2026-06-12 wedge). Immediate consequence: oversight_health.py injected a warning into the next AI response. Cascade: in interactive use, users read otherwise-fine analysis as malfunctioning; in a campaign, the warning baked into published output (or, if stripped, the human never saw it). Surfaced failure: a narrow win (honest health surfacing) defeated the broader goal (user trust) — a wedge that self-recovers in 30s scared a user.
IN-4 → breakage point: a path that stripped, edited, or compacted the transcript (a render pass, stealth-purge, or compaction event). Immediate consequence: the continuation marker dropped. Cascade: the next turn no longer detected continuation. Surfaced failure: the user was stranded mid-elicitation with no recovery and no error.
IN-5 → breakage point: an operator set ORA_RETENTION_SESSIONS_DAYS nonzero during launch cleanup. Immediate consequence: the sweeper archived active campaign sessions. Cascade: conversations left the sidebar mid-run, and rotated JSONL sinks left some stealth-purge operations without their target. Surfaced failure: the operator lost visibility into the live campaign and misdiagnosed the gap.
MO-1 → breakage point: the operational surface exceeded what one person could hold under load. Immediate consequence: coordination failures (not code) — server restarts, port drift, config clobbers, PR-mid-task collisions. Cascade: capability kept being added while operational hardening (single-owner locks, drift gates, port pinning) stayed informal. Surfaced failure: most outages were self-inflicted.
MO-2 → breakage point: users hit modes whose front-end (Tier 2 question banks) was never finished. Immediate consequence: input_contract elicitation degraded to generic prompts. Cascade: a hollow elicitation experience — shallow questions or none. Surfaced failure: the “guided” promise fell flat; the gap was known and deprioritized, not discovered.
MO-3 → breakage point: AS-1 + IN-1 + IN-3 compounded. Immediate consequence: the user concluded the oversight apparatus was ceremony. Cascade: disengaged from it entirely. Surfaced failure: caught none of the real drift it was built to catch — the premortem’s own cautionary case realized at runtime.
MO-4 → breakage point: a cluster of failures that fail without saying so (EX-2, AS-2, EX-4, EX-6, CS-1/CS-2, IN-2). Immediate consequence: the user couldn’t distinguish a clean deliverable from a degraded one. Cascade: confidence collapsed across the board. Surfaced failure: including work that was actually fine — how a system that “mostly works” still loses its users.
Leading indicators per failure mode
EX-1 — leading indicators: a liveness probe from the campaign client against its pinned port emitting a heartbeat (gap = stranded); >1 process bound/attempting on :5000. Signal-acquisition cost: trivial. Lead time before visible failure: minutes.
EX-2 — leading indicators: (a) scheduled byte-level diff of the ~58 mode files + framework files against vault pairs (DCP diff); (b) count of ora-side .md commits in the launch window with no corresponding vault commit within N hours, plus unexpected vault git-history commits mid-campaign. Signal-acquisition cost: (a) cron + git, near-zero; (b) low. Lead time: (a) within a day, before sync clobbers; (b) hours / continuous.
EX-3 — leading indicator: a config fingerprint (hash of resolved routing + active configuration) emitted into every pipeline trace; a mid-campaign change with no corresponding deploy = drift. Signal-acquisition cost: low. Lead time: immediate.
EX-4 — leading indicator: trace shows ### Verified Final Output differing from emitted text. Signal-acquisition cost: cheap (diff trace vs output). Lead time: per-turn.
EX-5 — leading indicator: watcher heartbeat staleness in oversight_health; daemon stack dumps in logs. Signal-acquisition cost: cheap. Lead time: hours.
EX-6 — leading indicator: project-tool subprocess non-zero exit or non-JSON on stdout; manifest abstract-name resolution miss in logs. Signal-acquisition cost: cheap. Lead time: per-invocation.
AS-1 — leading indicators: (a) value of ORA_OVERSIGHT_LIVE asserted at boot and printed in the health surface, with actions.jsonl showing no real model latency on verdicts; (b) every Decision Log / verdict record tagged simulated|live at the source. Signal-acquisition cost: (a) one line; (b) low. Lead time: (a) pre-launch; (b) immediate.
AS-2 — leading indicators: (a) a startup RAG self-test — embed a known query against each live collection, assert top-hit relevance above a floor; (b) retrieval-distance distributions logged per call (a collection with uniformly high distances or zero chunks is mis-embedded). Signal-acquisition cost: low. Lead time: (a) pre-launch + continuous; (b) first few calls.
AS-3 — leading indicators: log dispatched mode_id + triggering signals per turn, with a human spot-audit of the first ~100 live dispatches for routing precision; also the rate of users re-asking the same question rephrased as a misroute proxy. Signal-acquisition cost: moderate (human review) / low (rephrase rate). Lead time: first days.
AS-4 — leading indicator: a side-by-side quality delta (9B-only vs premium) on the launch’s own demo prompts, measured before any external claim. Signal-acquisition cost: moderate. Lead time: pre-launch.
AS-5 — leading indicators: (a) a pre-launch audit of exactly which collections and which vault/sync targets the campaign writes to — any write to a cloud-reachable path or shared collection is an exposure surface; (b) a per-turn tag of whether the turn’s content was indexed/exported and to which target. Signal-acquisition cost: low. Lead time: (a) pre-launch; (b) immediate.
AS-6 — leading indicator: RSS/swap during Gear-4 runs; two large slots co-bound to depth+breadth in the active configuration. Signal-acquisition cost: cheap. Lead time: seconds.
CS-1 — leading indicators: (a) per-slot 429/error-rate and fallback-engagement counters streamed live, plus the campaign progress counter flatlining (the 117/196 shape); (b) a daily diff of the model catalog (model-catalog-changes.jsonl already exists) flagging any in-use model newly marked deprecated, plus per-call cost or capability availability deviating from a pre-recorded baseline. Signal-acquisition cost: (a) low; (b) trivial. Lead time: (a) minutes / live; (b) ~1 day.
CS-2 — leading indicator: a reconciliation check between two existing signals — the deprecation flag present in Models-pane/registry state but absent from the campaign health surface = an unobserved substitution is live. Signal-acquisition cost: low (diff two existing signals). Lead time: ~1 day.
CS-3 — leading indicators: (a) cumulative spend tracked against a per-campaign budget ceiling with a burn-rate projection flagging the crossing before completion (per-call cost already lands in pipeline traces); (b) a pilot-batch cost estimate (run N representative turns, extrapolate). Signal-acquisition cost: low. Lead time: (a) days; (b) pre-launch.
IN-1 — leading indicator: human-queue.jsonl growth rate; count of engagement: unseen cards climbing; time-since-human-last-opened-Paused. Signal-acquisition cost: cheap. Lead time: live.
IN-2 — leading indicators: (a) fallback rate = (prose/table fallbacks ÷ total envelopes) on a realistic content sample, plus vision-extraction error rate; (b) raw-JSON-fence leakage rate in the V3 transcript (the old failure signature). Signal-acquisition cost: low / trivial. Lead time: (a) pre-launch if piloted / per-artifact; (b) immediate.
IN-3 — leading indicator: frequency of health-warning injection per 100 turns in a staging run; warning text appearing inside emitted artifacts. Signal-acquisition cost: low. Lead time: pre-launch / per-artifact.
IN-4 — leading indicator: continuation expected but marker absent in the last assistant turn. Signal-acquisition cost: cheap. Lead time: per-turn.
IN-5 — leading indicator: boot-time read of ORA_RETENTION_SESSIONS_DAYS ≠ 0, or a drop in sidebar conversation count with no corresponding user deletion. Signal-acquisition cost: trivial. Lead time: immediate (boot) / minutes (count drop).
MO-1 — leading indicator: count of self-inflicted incidents (server restarts, port drift, config clobbers, PR-mid-task collisions) per week in the run-up — already happening and already in memory. Signal-acquisition cost: zero (tally). Lead time: continuous.
MO-2 — leading indicator: a coverage check — which dispatched modes have complete input contracts + question banks vs stubs — run against the modes the campaign actually exercises. Signal-acquisition cost: low. Lead time: pre-launch.
MO-3 — leading indicator: time-since-human-last-opened-Paused; no Decision Log reads; rising user “is this right?” queries. Signal-acquisition cost: cheap. Lead time: daily.
MO-4 — leading indicator: there is no single signal — the class is caught by audit, not by one observable (see the class-level mitigation in the next section).
Pre-commitment mitigations
-
Single-owner server lock + port pinning (decision-gate / kill criterion) — what it locks in before commitment: pin the campaign to a dedicated port/process with a single-owner lock (refuse second bind, or a PID-file guard making parallel pkill+nohup fail loudly); a startup assertion that the bound port matches the campaign’s expected port or aborts; the convention that interactive CC sessions don’t touch the campaign’s server. Addresses: EX-1 (and the MO-1 self-inflicted class). Cost to implement before launch: low.
-
CI drift gate + vault writer freeze (decision-gate / precondition) — what it locks in: a drift gate that fails if any ora/vault pair diverges, on every commit and on a timer; freeze the multi-writer vault during the launch window (pause the DCP routine — which the memory note says runs “til ~2026-07-01,” overlapping launch — and quiesce side-channel sync) so there is a single writer under load. Addresses: EX-2 (and MO-1). Cost: low (cron + git + a freeze procedure).
-
Config snapshot + boot fingerprint assertion (assumption-check / decision-gate) — what it locks in: snapshot and pin the exact resolved config (active-configuration.json + routing) as a launch artifact; assert-on-boot that the live fingerprint matches, or refuse to start the campaign. Addresses: EX-3 (and MO-1). Cost: low.
-
Verifier-correction decision (decision-gate) — what it locks in: before launch, decide — either propagate the verifier’s fix, or surface a hard “verifier found N corrections that did NOT ship” banner on any artifact the verifier touched; separately confirm whether any export surfaces the verdict as a label and that the label means what the user thinks. Addresses: EX-4. Cost: low to moderate (the banner is cheap; propagation is a pipeline change).
-
Daemon load-validation + out-of-band health channel (test) — what it locks in: validate the two-lane split + watchdog survive sustained launch-like load; route staleness to an operator channel rather than the next chat response. Addresses: EX-5, IN-3 (and EX-2/EX-5 visibility). Cost: low to moderate.
-
Project-tool dry-run harness (test) — what it locks in: pre-launch, exercise every project tool the campaign invokes through invoke_project_tool on synthetic input, asserting clean JSON-on-stdout with no stray output, and that every abstract tool name in the campaign’s frameworks resolves against the registered manifest. Addresses: EX-6. Cost: low.
-
Oversight live-or-off decision + boot assertion (decision-gate / kill criterion) — what it locks in: decide explicitly before launch whether oversight is live or off, and make “silently simulated” impossible. If live: ORA_OVERSIGHT_LIVE=1 + a boot assertion that fails if the daemon runs without it during a campaign. If off: disable the sidebar panels so no one infers governance that isn’t there. Kill criterion: any verdict reaching a human queue without a live tag during the campaign → halt and reconcile. Addresses: AS-1, MO-3. Cost: low (one line + tagging).
-
Reindex-and-verify RAG gate (assumption-check / decision-gate) — what it locks in: a pre-launch reindex-and-verify pass on every collection the campaign touches; assert dimensionality + embedder identity per collection at boot; treat a clean RAG self-test as a launch gate. Addresses: AS-2. Cost: low.
-
Routing-precision pilot + “not what I meant” control (pilot / decision-gate + pre-built recovery affordance) — what it locks in: a routing-precision pilot on a held-out set of realistic prompts before launch with a go/no-go threshold (provisional default: misroute rate >10% on the held-out set → no-go, calibrated to the pilot’s baseline); and, distinct from the gate, build pre-commit a one-click “not what I meant” control into the dispatch parenthetical that re-routes, so a misroute is a 5-second correction not a silent loss (locked in during the pre-commitment window — explicitly not a post-hoc remediation). Addresses: AS-3. Cost: moderate (pilot + a small UI control).
-
Promise-config alignment (assumption-check / decision-gate) — what it locks in: decide before launch which config is the promise; if 9B, validate the demos on 9B; if premium, state the hardware honestly; don’t let the marketing config and the demo config diverge. Addresses: AS-4. Cost: moderate (a quality-delta measurement).
-
Sync/indexing exclusion audit + confidentiality blocker (assumption-check / decision-gate) — what it locks in: scope sync and indexing exclusions explicitly — assert campaign collections are local-only, exempt the campaign’s vault Sessions path from cloud writers and side-channel sync, and deliberately decide whether conversation RAG indexing is on for campaign content at all; make “campaign content reaches a cloud writer” a launch blocker, not a discovery. Addresses: AS-5. Cost: low.
-
Large-slot co-binding validator (assumption-check / test) — what it locks in: first confirm whether the routing/active-configuration layer already forbids co-loading two large slots on depth+breadth; if it does, verify and move on; if not, add a configuration validator that rejects any active config binding two 60–80 GB slots simultaneously before the campaign runs — not an OOM discovered at turn 400. Addresses: AS-6. Cost: low.
-
Fallback-degradation kill criterion + paid-tier pre-provisioning + model pinning (kill criterion / decision-gate) — what it locks in: decide the campaign’s tolerance for fallback degradation and set a kill criterion (provisional default: depth-slot fallback-engagement >20% over any rolling 50 calls → pause, calibrated to the quality floor); pre-provision paid-tier capacity (the escape hatch) before launch; pin the campaign’s models to a snapshot and alert on any deprecation of those specific IDs; record a per-call cost/capability baseline and assert it at start. Addresses: CS-1. Cost: low to moderate.
-
Deprecation-flag runtime channel (test) — what it locks in: route the deprecation flag into the campaign log / health surface, not just the settings UI — a config-time signal needs a runtime channel, locked in before launch. Addresses: CS-2. Cost: low.
-
Budget ceiling + pilot cost estimate + pre-decided downgrade path (kill criterion / decision-gate / pilot) — what it locks in: set a hard budget ceiling with a kill criterion (provisional default: pause at 80% of budget with completion <80%), estimate per-run cost on a pilot batch before committing the full run, and pre-decide the downgrade path (which config the campaign falls to, and whether a downgrade is acceptable at all). Addresses: CS-3. Cost: low.
-
Paused-queue budget + severity digest threshold (decision-gate) — what it locks in: a pre-committed cap on Paused cards per day and a severity threshold below which events go to a digest, not the queue; configure fan-out and escalation to respect an acceptable human triage load decided now. Addresses: IN-1, MO-3. Cost: low.
-
Visual-pipeline fallback-rate pilot + V3 render verification (pilot / test / decision-gate) — what it locks in: run the visual pipeline on a representative content sample before launch and measure fallback rate; if high, recalibrate the Critical threshold or fix the upstream layout issues as a pre-commit; verify V3 actually renders (the v3-visual-dispatch.js path) on the launch build; confirm the vision slot is filled by a vision_capable model; set a threshold above which a visual deliverable is “failed-as-designed.” Addresses: IN-2. Cost: low to moderate.
-
Marker-survival test (test) — what it locks in: before launch, test that elicitation/resolution markers survive your compaction and purge paths; if they don’t, add a recovery path or move state out of the transcript for campaign-critical flows. Addresses: IN-4. Cost: low.
-
Retention precondition assertion (decision-gate / kill criterion) — what it locks in: assert ORA_RETENTION_SESSIONS_DAYS=0 as a launch precondition; treat any nonzero value as a blocker; exempt the campaign’s conversation sinks from rotation during the run. Addresses: IN-5. Cost: trivial.
-
Feature-freeze + convert recurring incidents into enforced guards (decision-gate) — what it locks in: before launch freeze feature work and convert the recurring self-inflicted incidents into enforced guards — single-owner server lock, port pinning, drift gate, config-fingerprint assertion, vault writer freeze — making the known failures structurally impossible during the launch window rather than relying on operator vigilance under load. Addresses: MO-1 (and the coordination class broadly). Cost: low (mostly discipline + the guards already listed above).
-
Scope launch to complete modes (decision-gate) — what it locks in: scope the launch to only the modes whose front-end is complete; gate the incomplete ones behind a “preview” label or disable them. Addresses: MO-2. Cost: low.
-
Ground-truth deliverable audit at fixed checkpoints (test — the class-level defense) — what it locks in: pre-commit to auditing a random sample of campaign deliverables against ground truth at fixed checkpoints (e.g., every 25 outputs); this is the only defense that catches the silent-degradation class even when individual indicators are missed. Addresses: MO-4 (and every silent-failure member — EX-2, AS-2, EX-4, EX-6, CS-1/CS-2, IN-2). Cost: moderate (human review time).
-
Consolidated pre-flight gate (decision-gate — highest-leverage single mitigation) — what it locks in: a pre-flight script that refuses to start the campaign unless, fail-closed: ORA_OVERSIGHT_LIVE=1 (or an explicit, logged “oversight intentionally simulated” acknowledgment); ora↔vault parity is clean (DCP diff = 0) for every mode/framework the campaign touches; the embedder in chromadb.json matches the campaign’s collections and a test query returns non-empty; no campaign slot references a DEPRECATED model; a recorded per-call cost/capability baseline matches the registry at start; the config fingerprint matches the pinned launch snapshot; ORA_RETENTION_SESSIONS_DAYS=0; the active config does not co-bind two large slots on depth+breadth. This converts several silent failures (AS-1, EX-2, AS-2, CS-1, EX-3, IN-5, AS-6) into one loud refusal-to-start. Cost: moderate (it composes the assertions already specified above into one script). Kill criteria to set now, alongside it: (a) live ground-truth audit error rate above a pre-agreed threshold; (b) campaign throughput implying completion beyond an acceptable horizon; (c) Paused queue growing faster than the human clears it for two consecutive days; (d) depth-slot fallback-engagement >20% over a rolling 50 calls; (e) spend at 80% of budget with completion <80%. Any one → halt and reassess. (All thresholds are provisional defaults to calibrate.)
Residual unmitigated risks
Simulated-oversight acceptance (AS-1 residual) — which failure modes remain exposed: AS-1 and MO-3 remain partially exposed even with tagging and assertions. Conditions under which it materialises: if the team chooses simulated for cost during launch while the campaign markets “AI oversight,” the governance promise is hollow by design. Whether it should warrant rethinking the plan: yes — this is a go/no-go, not a tuning knob; if the launch’s value proposition includes meta-layer governance, shipping it simulated is a plan-level contradiction no leading indicator fixes. (Highest-stakes residual.)
Confidentiality after the sync freeze lifts (AS-5 residual) — which failure modes remain exposed: AS-5. Conditions: excluding campaign content from cloud writers during launch defers, doesn’t solve, the structural multi-writer/cloud-reach problem; it returns the moment normal vault sync resumes, or when a later campaign reuses the same vault paths without re-checking exclusions. Whether it should warrant rethinking: yes — a precondition, not a tuning knob; if a real-world deployment can’t guarantee where campaign content goes, that’s a launch blocker on the highest-trust axis.
Vault multi-writer correctness after the freeze lifts (EX-2 residual) — which failure modes remain exposed: EX-2. Conditions: a pre-flight parity gate checks at start, not continuously; cloud writers and side-channel sync can mutate the vault during a long campaign (the DCP routine resumes ~2026-07-01); materialises if a DCP run or claude.ai session edits a campaign-relevant file mid-run. Whether it should warrant rethinking: it warrants a continuous freeze window the pre-flight check can’t enforce alone.
External provider instability (CS-1 residual) — which failure modes remain exposed: CS-1. Conditions: paid-tier provisioning reduces but cannot eliminate OpenRouter throttling/deprecation; the upstream is outside your control; materialises if a provider deprecates a pinned flagship (e.g. an Opus version) mid-campaign with no equivalent fallback. Whether it should warrant rethinking: not on its own — manage with pinning + alerting + a fallback policy.
Budget projection error (CS-3 residual) — which failure modes remain exposed: CS-3. Conditions: a pilot estimate reduces but cannot eliminate per-run cost surprise; retries under throttling and Gear-4 fan-out are the volatile terms; materialises if real-world content triggers far more correction cycles than the pilot batch. Whether it should warrant rethinking: not on its own — the budget kill criterion bounds the damage.
Routing precision on truly novel phrasing (AS-3 residual) — which failure modes remain exposed: AS-3. Conditions: a pilot reduces misroute rate but substring matching has an irreducible tail on inputs unlike any test prompt; materialises for a user domain whose vocabulary isn’t in the signal registry. Whether it should warrant rethinking: not on its own — the “not what I meant” recovery affordance bounds the per-instance cost.
Normal-accident coupling (whole-system residual) — which failure modes remain exposed: the interaction class as a whole. Conditions: the stack is both complex (oversight router + watchers + daemon + RAG + adversarial pipeline + project plugins + visual compiler) and tightly coupled (vault→ora sync, conversation-as-state, shared :5000 server, shared RAM), so some interaction failure no single mitigation anticipates is structurally likely (Normal Accident Theory lens). Whether it should warrant rethinking: yes, at the scale dimension — the honest move is blast-radius limitation: run the first campaign at reduced scale with a fast manual kill, accepting that not every interaction can be pre-listed. The mitigations shrink holes; they don’t make the cheese solid.
Confidence per finding
- EX-1 (port drift): high — directly evidenced and recurring in the memory trail (shared :5000 server, 5000→5001 drift stranding a pinned campaign).
- EX-2 (vault drift): high that the mechanism exists; medium it fires at launch — CLAUDE.md is explicit on silent overwrite and three concurrent vault writers; whether it fires depends on launch-window edit discipline.
- EX-3 (config drift): medium — mechanism evidenced (gitignored/auto-managed config, chore-PR landing); occurrence depends on a redeploy/pull happening mid-run.
- EX-4 (ungated verifier): high on mechanism — the f-verify scope rewrite documents non-propagation explicitly; the user-facing-verdict-label compounding is conditional and unconfirmed.
- EX-5 (daemon wedge): medium — the 2026-06-12 wedge shows the lane can stall; recurrence under load is inference.
- EX-6 (plugin tool failure): medium — load-bearing only if the launch is the MSI campaign; the subprocess JSON-on-stdout contract is the documented brittleness.
- AS-1 (simulated oversight): high on mechanism; the highest-stakes finding — the simulated default and
ORA_OVERSIGHT_LIVE requirement are documented; decisiveness rated high per one reading, medium per another.
- AS-2 (RAG mismatch): medium — mechanism evidenced (bge-m3 migration, machine-specific
chromadb.json, named as a known machine-state failure); occurrence not.
- AS-3 (misrouting): medium — substring matching at Stage 1 is documented; misroute rate on real prompts is inference.
- AS-4 ($1K hardware): medium — the
qwen-9b-only reproducibility config vs premium-model quality gap is documented; the marketing/demo divergence is the inferred trigger.
- AS-5 (confidentiality): medium — mechanism evidenced (ChromaDB indexing +
vault_export to cloud-reachable paths); occurrence depends on deployment config.
- AS-6 (RAM): medium — downgraded pending a routing-guard check — the “two large models exceed RAM” statement is documented; whether a guard already forecloses depth+breadth co-binding cannot be confirmed from the package.
- CS-1 (provider churn): high on the throttling/deprecation precedent; medium on the pricing sub-case — the throttling modal and DEPRECATED machinery exist because these recur; the pricing-misread sub-case rests on the documented
$0.0/$0.0/M video-pricing bug.
- CS-2 (silent substitution): medium — the DEPRECATED flag is documented as a UI signal; the unobserved-substitution pathway is inference for an automated run.
- CS-3 (budget exhaustion): medium — precedented by the Opus refill — the premium campaign already required a refill; at-scale burn outpacing projection is inference.
- IN-1 (queue flood): high — the classic and most probable killer per one reading — every fan-out/DRIFT/ESCALATE landing in the queue plus cross-project fan-out is documented; scale makes flood likely.
- IN-2 (visual degradation): medium — the Critical-fallback gate (Protocol §8.5) and the V3 raw-JSON precedent are documented; fallback rate on real content is inference.
- IN-3 (health-warning contamination): medium — the injection-into-next-response mechanism and the recurring staleness state are documented; contamination of artifacts is inference.
- IN-4 (lost markers): medium — conversation-as-state markers and compaction/purge paths are documented; marker loss is inference.
- IN-5 (retention archival): low — guarded by defaults — session archival is off by default; materialises only if an operator sets
ORA_RETENTION_SESSIONS_DAYS nonzero.
- MO-1 (single-operator overload): high — repeatedly evidenced — the memory trail is full of self-inflicted coordination incidents.
- MO-2 (unfinished scaffolding): medium — the Tier 2 question banks are documented as still-open “Remaining Work”; the hollow-elicitation consequence is inference.
- MO-3 (theatrical oversight): medium — a compound of AS-1/IN-1/IN-3; the Klein-attribution phrase was corrected to the analyst’s own coinage.
- MO-4 (silent-failure throughline): high — the shared “fail without saying so” property across EX-2/AS-2/EX-4/EX-6/CS-1/CS-2/IN-2 is structural, not speculative.
- Overall grounding: confidence is split between mechanism (anchored to a named module, env var, config file, or documented incident — high where the design choice is explicit) and occurrence at launch (inference, generally medium-to-low). Every failure mode ties to a specific artifact —
ORA_OVERSIGHT_LIVE, chromadb.json, port 5000→5001, qwen-9b-only, the retention sweeper, oversight_health injection, vault_export cloud writers, the Opus refill, invoke_project_tool, Protocol §8.5 — so none would read identically for a generic project.
- No optimism-residue flag warranted — the inventory runs to ~24 distinct failure modes across all five Klein classes (no concentration in execution), matching the system’s documented complexity; both contributing analyses independently judged a shorter inventory as optimism leaking through, and the breadth here does not show truncation.
- Two open items would re-weight the inventory and remain genuinely unresolved: (1) whether this launch is the MSI news campaign specifically (load-bearing for EX-6 and IN-2) or another project; (2) whether oversight is intended to run live or simulated for it (load-bearing for AS-1/MO-3).
Additional considerations
Several pre-commitments are reusable operational infrastructure, not launch-only cost: the drift gate (EX-2), single-owner server lock + port pinning (EX-1), config-fingerprint assertion (EX-3), and the sync/indexing exclusion audit (AS-5) become permanent guards that harden every subsequent campaign and directly attack the self-inflicted-incident class (MO-1) the memory trail shows recurring. Prioritizing these first buys both launch safety and durable operational maturity. (This is the one forward-looking note; the failure stance above stands unchanged.)