ADR-034: Rehearsal persistence: the persistent entity is the decision, not the rehearsal

Status: Proposed. Supersedes nothing; extends the sealed decision record shipped in sv0-platform#1780.

Context

Ivan asked whether a rehearsal should become a persistent, browsable, comparable entity, visible when a user logs in. Verified today: it is not persisted at all. The rehearsal API is three router.gets (src/api/routes/rehearsal.ts:181, :340, :607); the engine runs on a Map behind a throwing Proxy (src/rehearsal/in-memory-adapter.ts:5-23, :168-181); grep -rn "rehearsal" src/storage/ returns zero matches and there is no rehearsal collection among the 44 in src/storage/mongo/collections.ts:38-94. One exception: a sealed decision record embeds the full revocation RehearsalVerdict in content.break_set.rehearsal_verdict, and only when the pre-rehearsal verdict is Constrain with a ScopedFix (src/services/verdict-compute.ts:78-120). The deployment/promotion shape RehearsalDeploymentVerdict (src/rehearsal/types.ts:381) is never sealed.

Two research streams reached opposite conclusions and both were read in full. research/persistence-analysis.md argues persistence is necessary now, on one load-bearing premise: between the check-run posting a cone and a human confirming "minutes to hours later", the graph can move, so the human approves a different computation than was reported. research/persistence-challenge.md overturns that premise on its own evidence — nothing refreshes the graph automatically (no cron, no CI --submit, src/workers/connector-driver.ts:172 hardcodes DEFAULT_BINARY = "sv0-aws", and that binary is absent from the platform image, Dockerfile:1-29), the "minutes to hours" figure is the one unlabelled number in the document, and GitHub's 30-day wait is for "a webhook event response" — the App's budget, not a human's. Both agreed the fix for substitution is a hash rather than a store, and both pointed at the shipped one: verdict-engine.ts:362-364 computes sha256(canonicalSerialize(rehearsal.verdict)) and refs it as rehearsal_verdict_hash.

The round-2 adversarial review showed that hash covers the wrong object, in both directions at once. This ADR now specifies three hashes with separated jobs, and the correction is recorded rather than silently applied.

• Too narrow. A human does not approve a rehearsal. They approve an outcome derived from the rehearsal plus the policy row and its thresholds, the named authority delta, the freshness class, the coverage factors, the subject and enforcement-point binding, the artifact digest, and the approval owner. None of that is inside canonicalSerialize(rehearsalVerdict). ADR-039 explicitly allows the same rehearsal to yield a different gate outcome under a different policy row — so every policy effect sits outside the shipped hash, and a policy edit between status report and confirmation passes the comparison silently. That is the exact substitution the hash was introduced to catch.
• Too broad. as_of is inside the hashed verdict (src/rehearsal/types.ts:342) and comes from deriveAsOf (src/rehearsal/service.ts:245-252) over the entity set loaded at src/rehearsal/service.ts:361 — the max updated_at across the entire tenant. Any connector scan touching any unrelated entity moves it. Comparing that hash at confirmation refuses a release for a reason with no connection to the decision. totals.entities_materialized and totals.baseline_paths have the same property.

One hash cannot be both. Too broad produces false refusals on unrelated ingest and trains the approval owner to click past a warning; too narrow misses the policy-derived substitution. Both failures land on the release path, so both are named here rather than resolved by choosing a side.

Ivan's actual ask — see the rehearsals we did before, when I log in — is separately and cheaply satisfiable: queryDecisionRecords(tenantId, {workload_id, verdict, limit}) exists in the interface and the adapter (src/storage/storage-adapter.ts:893, src/storage/mongo/adapters/decision-record-adapter.ts:87) with zero non-test callers. A browse surface is one route and one page.

Decision

Ivan's question has two halves. Both are real; only one of them is built now.

The correctness half — necessary, and solved without a rehearsal store. The defect the persistence research found is genuine: GitHub's hold can last up to 30 days, so a human can confirm long after the status report was posted and approve a materially different computation than the one that was reported, with nothing recording the substitution. The fix is a hash, not a store — but it must be a hash of the decision, and it has a prerequisite (item 5) that the first draft of this ADR assumed away.

1. The persistent entity of the gate is the gate decision, not the rehearsal. No rehearsal_runs collection, no rehearsal ReportDoc type, no synthetic-overlay collection, no temporal graph replay.

2. Three hashes, with different jobs. None is the other's fallback.

   Hash	Preimage	Role
   rehearsal_verdict_hash	sha256(canonicalSerialize(rehearsalVerdict)) over the complete, unmodified verdict — the shipped pattern at verdict-engine.ts:362-364	Evidence and drift labelling only. Recorded on the gate_decisions record and in the sealed row so a reader can identify which rehearsal backed the decision. It is not a field of the ApprovalPayload, and it is never a release gate on its own, because it moves on unrelated tenant ingest (Context).
   rehearsal_projection_hash	sha256(canonicalSerialize(projection)), where the projection is the same verdict with as_of, totals.entities_materialized and totals.baseline_paths removed	The rehearsal identity the payload carries. This is the one inside ApprovalPayload, and therefore inside approval_content_hash.
   approval_content_hash	sha256(canonicalSerialize(ApprovalPayload)) — the complete immutable payload rendered to the approver, with the volatile fields below excluded	The compared hash. This is the fingerprint of the thing the human actually approved.

   The ApprovalPayload field table is in ADR-036 Decision 2 — that table, not this hash, is the contract; the hash is only its fingerprint. Hash scope:

   Excluded from approval_content_hash	Why
   as_of	Tenant-wide max updated_at; moves on any unrelated scan (src/rehearsal/service.ts:245-252)
   totals.entities_materialized	Tenant-wide count; same property
   totals.baseline_paths	Tenant-wide count; same property
   any other tenant-scoped counter that moves on unrelated ingest	Same property — the rule is the class, not the enumeration

   The general rule, stated so it is not re-broken: no field inside approval_content_hash may be derived from an excluded field. A nested hash, digest or fingerprint placed in the payload re-imports every field of its own preimage, so any such value must be computed over the same projection the payload uses. rehearsal_verdict_hash fails that rule by construction — its preimage contains as_of and both totals — which is exactly why the payload carries rehearsal_projection_hash instead, and why an earlier draft that put the verdict hash inside the compared hash could not have passed its own acceptance test.

   as_of and both totals are stored on the gate_decisions record as evidence and for drift labelling. They are rendered to the approver. They are simply never inside the compared hash. Everything else the approver sees — policy row id and version, mapper version, threshold set, named authority delta, freshness class (ADR-038), coverage factors, subject and enforcement-point binding, artifact digest (ADR-036 Decision 6), computed outcome, approval owner, and rehearsal_projection_hash — is inside it.

3. At the moment the named human confirms, the full decision is recomputed and approval_content_hash compared. Any difference in policy, freshness class, mapper version, subject, evidence set, owner, computed outcome or backing rehearsal requires a new human confirmation — the hold is re-presented with the current computation and a named diff. It is never a silent pass, and it is never an auto-fail on an unrelated counter. This fails toward fresh state, not toward stale bytes, and detects substitution without storing a cone.

4. rehearsal_verdict_hash mismatch alone is a labelled drift disclosure on the re-presented hold, not a refusal. It tells the approver the backing rehearsal is no longer bit-identical — including for reasons that are only tenant-wide volatility; whether that changes the decision is answered by approval_content_hash. A change in rehearsal_projection_hash, by contrast, is a change to the thing approved: it moves approval_content_hash and therefore requires a new confirmation under item 3.

5. Byte-determinism of the recomputation is a PREREQUISITE of items 3 and 4, not an assumption. It does not hold today. canonicalSerialize sorts object keys but deliberately preserves array order (src/rehearsal/engine.ts:1552-1557). The live exposure is tie order, not subset selection: queryCorrelations sorts { last_confirmed_at: -1 } with no secondary key (src/storage/mongo/adapters/correlation-adapter.ts:60), Mongo does not guarantee a stable sort, and the in-memory adapter deliberately mirrors the omission (src/rehearsal/in-memory-adapter.ts:133-139, "No _id tie-break is added"). Two identical queries can therefore return equal-timestamp correlations in different orders and produce different bytes for the same state. (The 100-record cutoff does not bite on this path: all four rehearsal call sites pass { limit: 0 } — src/rehearsal/service.ts:362, 486, 568, 661 — and both adapters gate on if (limit > 0) (correlation-adapter.ts:60-62, in-memory-adapter.ts:146-147). Any other paginated or limited read later fed into the verdict reintroduces the subset risk.) Required before the first compared hash:

   • add { last_confirmed_at: -1, _id: 1 } to both adapters, kept mirrored;
   • give every array serialized into the verdict a total comparator ending in a guaranteed-unique key — sortReachConeByRisk (src/rehearsal/engine.ts:867-888) is the shipped pattern to follow;
   • the acceptance test is a permutation test: shuffle the loaded correlation array, recompute, assert byte-identical output. Not a 101-tie cutoff test — that exercises a truncation this path does not reach.

6. Every gate verdict is additionally stamped, at seal time, with: a new rehearsal_engine_version constant; the materializer flag set already computed per run (src/rehearsal/service.ts:676-680); as_of; the absolute wall-clock last_synced_at per contributing source_system (EntityDoc.last_synced_at, src/domain/entities/types.ts:296); and the coverage factors. An engine-version mismatch at confirmation is an explicit, named outcome — not a hash mismatch dressed up as drift.

The visibility half — what Ivan actually asked to see. "All the rehearsals created before, visible when a user logs in" is delivered by the sealed gate decision plus a list surface, not by a rehearsal store.

7. The browse surface is the sealed gate decision plus a list over decision_records — every deployment we gated, with its verdict, its evidence, and who approved it. One route plus one page over the already-written queryDecisionRecords.
8. State the difference in the surface itself: what reopens is the sealed verdict and its evidence, not a live graph. Nothing in this slice can re-render a past reach cone, and no screen, label or link may imply that it can. It also does not reopen a tamper-evident record — see Consequences.

The boundary.

9. Hard rule, adopted verbatim: persist the result, never the inputs-as-entities. Nothing representing a hypothetical may exist in a collection an entity query can reach. Overlays are built client-side from verdict bytes, as ui/src/lib/proposed-overlay.ts already does.
10. Explicitly out of scope now, recorded so the deferral is a decision and not an omission: the overlay entity collection, a re-renderable pinned reach cone, rehearsal-to-rehearsal comparison, and temporal graph replay. Revisit at the pilot's first genuinely disputed hold, or when the selected use case demands a reopenable graph — whichever comes first. If that trigger fires, the shape is a widening of the sealed record to carry RehearsalDeploymentVerdict (extending #1780), not a new store. Correction to the earlier draft: that widening is not free. DecisionRecordContent is a single closed interface with a fixed field set and no discriminator (src/domain/decision-records/types.ts:229-284), and sealDecisionRecord accepts only an existing VerdictResult (src/services/verdict-engine.ts:1267). The deferral avoids a store, not a schema change — ADR-036 Decision 7 decides the widening shape (a versioned discriminated record_kind) and prices it.

Alternatives considered

• (a) Status quo, persist nothing extra. Rejected in part: it leaves no way to prove the cone the human saw is the cone that exists, and no engine version on the record.
• (b) Snapshot the cone verbatim inside the sealed record now. Deferred, not rejected — it is item 10 above. It also sits behind a storage-shape question: decision records chain per (tenant_id, workload_id) with unique indexes (src/storage/mongo/schema.ts:216-221), while a gate fires per deployment. ADR-036 keeps the per-deployment hold lifecycle out of that chain entirely and seals exactly one summary row at the terminal decision, so a snapshot embedded in that terminal row becomes possible later without touching shipped index semantics. It is not built now.
• (b′) Persist as a new ReportDoc report type. Rejected for the first slice. ReportDoc has exactly one subject field, subject_workload_id (src/domain/reports/types.ts:19) — no deployment, SHA, or environment key — and ReportQuery sort is documented as not implemented (src/storage/mongo/adapters/report-adapter.ts:27-34). Worse for honesty: ReportsPage.tsx and ReportDetailPage.tsx contain zero occurrences of stale / as_of / freshness, so a persisted cone would render "generated 2 hours ago" over a graph of unknown absolute age. Today the platform is structurally incapable of showing a stale cone; that safety property is not traded away for free.
• (c) rehearsal_runs + an overlay entity collection. Overlay half rejected outright. A persisted synthetic identity would be read unfiltered by src/evaluator/index.ts:52, producing a real FindingDoc, a SHA256-sealed evidence pack about an agent that does not exist, a wrong headline posture count (src/services/posture-service.ts:103-105), and — via verdict condition A3 (verdict-engine.ts:88-92) — could flip a real workload from Approve to Unknown silently.
• (d) Full temporal replay as-of a past moment. Rejected. EntityQuery (src/storage/storage-adapter.ts:99-127) has no temporal field, and correlations, tenant_correlation_settings, system_classifications, promotion_counterpart_maps are unversioned current-state — replaying entities as-of T against four inputs as-of now produces a different answer and labels it the old one.

Consequences

• The gate can detect whether the decision the human saw still holds, which is what a dispute turns on, without a storage subsystem before the first hold — once item 5 ships. Before the ordering fix and the permutation test, a recompute-and-compare can differ for a state that did not change, so item 3 is not implementable and must not be described as working.
• The audit artifact is not tamper-evident against a write-capable actor, and this ADR does not make it so. decision_records is insert-only, pointer-linked, with an unkeyed integrity checksum: the preimage includes previous_record_id, so corruption, cross-tenant or cross-workload transplant and re-parenting are verify-loud, but the code states in its own words that "a write-capable actor who recomputes hashes after editing defeats it; a keyed MAC or signature plus predecessor-hash chaining is the documented future step before decision records back a confirmed production approval" (src/evidence/integrity.ts:25-36). A T2 hold on a production deployment is precisely that use case. The keyed-MAC/signature plus predecessor-hash-chaining work is therefore a named prerequisite before T2 holds back a production approval, not a later hardening item — ADR-036 Decision 3 budgets it. No document in this family may describe the gate's audit artifact as hash-chained, cryptographically authenticated, or tamper-proof.
• Absolute per-source last_synced_at is emitted regardless — necessary because computeStaleSources anchors on asOf = max(entity.updated_at), never wall clock (src/rehearsal/service.ts:276-310), so a uniformly 60-day-old graph raises zero freshness warnings today. This is the single highest-value honesty fix in the research and it is independent of the persistence fork.
• What a user can reopen is the sealed verdict and its evidence — text, numbers and hashes — not a live graph. That is the honest description of the visibility half, and every artifact in this family says it the same way.
• Accepted, named risk: retrofitting the past is impossible. If a hundred verdicts ship and number 40 is disputed, it cannot be reconstructed — the four unversioned collections above, plus CounterpartChangedError → 422 (src/rehearsal/promotion.ts:86-95) firing on exactly the deployments most worth re-examining. Mitigation is scale, not storage: cap the pilot and treat the first dispute as the trigger in item 10.
• The determinism fix is on the critical path and was not in the first estimate. Item 5 is two adapter changes kept mirrored, an audit of every array serialized into the verdict, and a permutation test — work that the earlier "the hash is already shipped" framing priced at zero. Treat the published gate estimate as an undercount until it is re-run with item 5, the ADR-036 reliability substrate and the sealed-record widening included; do not silently adjust the number.
• Rehearsal-to-rehearsal comparison is out of scope (item 10); it stays possible later only because item 6 emits the engine version and flag set now.
• Reset-list regression (test/scripts/seed-reset-collections.test.ts) is untouched, because no new tenant-scoped collection is added.

Falsifier (what would overturn this)

Evidence that the graph actually moves inside a hold window — i.e. connector refresh becomes automated (ADR-027 slice work) and holds routinely span a refresh. That, the first genuinely disputed hold, or a selected use case that demands a reopenable graph are the three named revisit triggers; any one of them fires item 10 and the pre-committed shape is built.

---