ADR-037: Failure posture: behaviour split by which component failed, three tiers, and a deadline that escalates rather than decides

Status: Proposed. Supersedes the "fail open when SecurityV0 is down" posture in the first draft of this ADR — that promise cannot be honoured and has been deleted, not softened. The component that releases a hold obtains its authorization from the platform; if that component, its credential, the network or the GitHub API is unavailable, nothing can post approved and the run waits on GitHub's timeout. Fail-open is only meaningful for the failures where the release path is still alive.

Context

GitHub's documented timings are exactly two numbers: the App must return 2XX within 10 seconds or the delivery counts as failed, and a triggered rule waits up to 30 days for a webhook event response before timing out and failing the job. GitHub does not automatically redeliver failed deliveries. There is no published latency SLO for protection-rule event delivery. (All: research/github-provider.md, docs.github.com.)

Ours: no synchronous scan-now path exists — POST /api/v1/scan-runs returns 202 and enqueues (src/api/routes/scan-runs.ts:166), then ingestion, then a 60s stitch debounce (src/services/stitching/stitch-debounce.ts:54). A promotion rehearsal materializes the whole tenant twice and hard-rejects at scale (#1570 review). So the rehearsal cannot run inside the webhook request.

And the product constraint: promotion-control-subtraction and promotion-correlation-asymmetry are unconditional engine invariants pushed on every promotion verdict, with control-subtraction forced to render first, precisely because "a promotion verdict without the control-subtraction caveat reads as a clean CI gate — exactly the false confidence the issue forbids" (src/rehearsal/coverage.ts:59-64, :458-493).

Decision

1. ACK fast, decide asynchronously. The response adapter receives the delivery (ADR-035 Decision 3), validates it, writes it to its own gate_inbox, and returns 2XX well inside 10s, then enqueues the compute. Nothing about the verdict happens in the request.

2. A redelivery poller is a required component, not optional. Without it a downed adapter leaves a deployment silently held for up to 30 days.

3. Behaviour is determined by which component failed. There is no general fail-open posture, because there cannot be one. "SecurityV0 is down" is not one failure — it is at least two, and only one of them leaves a working release path.

   Failure	Is the release path alive?	Behaviour
   Platform evaluation errors or is unreachable from the adapter; the adapter, its credential, the network and the GitHub API are all healthy	Yes	MAY fail open, per tenant and per environment, after an explicit bounded timeout — the adapter releases and posts a status report on the run stating plainly that no SecurityV0 verdict was applied
   Adapter process down · private key unavailable or expired · network partition to GitHub · GitHub API unavailable · installation removed	No	Necessarily fails closed. Nothing can post approved. The deployment stays held until the component recovers, or a customer admin bypasses at GitHub. Say this plainly; do not promise otherwise
   The named human has not decided	Yes, but no decision exists to deliver	Stays held. See (b) below

   On the fail-open row — what it actually costs. It is not a flag. It requires a cached, signed, per-environment failure policy already present in the adapter before the outage (an adapter that cannot reach the platform cannot ask the platform what to do), plus the durable inbox from ADR-036 Decision 4 — gate_inbox, in the adapter's own separately credentialed store. And even then, the adapter cannot synchronously write the audit record while the audit platform is down — the release happens, and the record of it is written later from the adapter's own durable store, with a gap that must be reconciled. That consequence is stated here rather than discovered in an incident review. The commercial argument for fail-open remains real — an early-stage vendor sitting fail-closed inside a customer's production deploy path is a procurement blocker on its own — but it buys coverage of one failure class, not of "SecurityV0 is down".

   (b) The named human has not decided. The deployment stays held. Constrain, Reject and Unknown hold on the grounds in coverage.ts above, and Approve releases only after a named human confirms, for the whole pilot. There is no auto-approve at any holding tier, ever — it would void the human-in-the-loop property the pilot exists to prove. This ADR settles that the gate does not decide for the human; which dispositions a human may take from each computed verdict is a separate, open decision — the matrix in ADR-039 goes to the working session with empty cells, and nothing here pre-fills it.

4. Our decision deadline reports and escalates; it does not decide. On expiry at a holding tier the gate posts a status report, moves the hold to overdue, escalates the notification to the approval owner, and keeps holding. It does not release and it does not reject. GitHub's own 30-day timeout is the terminal fail-closed backstop, and it is GitHub's behaviour, not ours — the job fails, and we record timed_out_by_github. Label it that way on every surface. GitHub documents nothing between 0 seconds and 30 days, so any shorter number we publish is ours, and it is an escalation trigger rather than a decision.

5. Enforcement tiers, per tenant and per environment, as configuration. This is the one tier vocabulary; the monitor/enforce/off and 0-advisory/1-soft/2-hard vocabularies are retired.

   Tier	Name	Behaviour
   T1	Advisory	Rule enabled. The run is held — briefly — while the verdict is computed; the adapter then approves regardless of the verdict and posts it as a status report
   T2	Hold-with-human	The pilot tier. Hold; release only on a named human's confirmation
   T3	Automatic	Auto-release on a clean verdict. Not in the pilot

   T1 holds. Describe it as a brief hold, never as "never holds". GitHub sends deployment_protection_rule only once the rule is enabled on the environment, and from that moment the run waits until something answers. T1's honest claim is "the gate answers immediately and does not ask a human" — the wait is short, and how short is a measured number the dogfood produces, not a published one.

   T3 is not reachable until enforcement and verification have both been proven in the bounded scenario.

   T0 "Observe" has been deleted from the tier vocabulary. It was defined as "rule not enabled, compute anyway" — which has no trigger, because with the rule disabled GitHub sends no event to compute from. Enabling the rule to get the event and then posting nothing back leaves the run waiting on GitHub's 30-day timeout, which is not an observation posture but an outage. A shadow mode is still wanted and is a separate piece of work with its own trigger — an explicit workflow step or an API call from the customer's pipeline. File it; do not imply it already exists.

6. Kill switch: default hold_all. The default position of the switch keeps deployments held. release_all is an explicit, human-initiated, per-tenant action — never a default, never a side effect of a flag flip — and every decision it drains records the actor who invoked it and posts a status report on that run saying no verdict was applied. The kill switch loses to an already-committed human terminal decision: release_all and hold_all drain only holds that have not reached release_authorized or refusal_authorized, and never overwrite a disposition a named human already committed (ADR-036 Decision 4(c) — the one-way terminal lock is what enforces this). The operational fallback is uninstalling the gate App, which also removes the rule's owner from the environment. Both paths must be exercised before the pilot, not documented only.

7. The gate never claims a hold it does not have. Onboarding verifies, per gated environment: admin bypass deselected, deployment: false not set, OIDC sub bound with StringEquals to the environment, and the protection rule present. Loss of any of these degrades the tenant to T1 and raises an operational alert.

Alternatives considered

• Rely on GitHub's 30-day timeout as our own timer. Rejected as our behaviour — a 30-day silent hold is an outage for the customer's pipeline. The timeout itself is not rejected: it remains the terminal fail-closed backstop, GitHub's, and labelled as GitHub's.

• Auto-approve on our deadline at a holding tier. Rejected outright: it releases exactly the deployments that were held because no human had decided, which is the property the pilot exists to prove. Escalation, not decision (Decision 4).

• Auto-reject on our deadline at a holding tier. Also rejected: a rejection the customer's team never chose fails their pipeline on our clock and teaches them to remove the rule.

• A general "fail open when SecurityV0 is down" default. Not an alternative — impossible, and claimed in the first draft of this ADR. The component that can release the hold obtains its authorization from the platform; when the adapter, its credential, the network or the GitHub API is the thing that failed, there is nothing left to post approved with. Replaced by the split-by-component table in Decision 3.

• Fail-open on platform evaluation failure. Considered, and not the recommended default — DECIDED 2026-07-25, resolving a contradiction between this ADR and #1908. An earlier revision of this paragraph recommended against fail-closed on the grounds that it makes a customer's deploy path conditional on an early-stage vendor's uptime. That argument is real and is why the option remains configurable per tenant — but it is not the pilot default.

  The pilot default is fail-closed: hold, and require manual re-approval. Reasons, in order. A gate that releases when it cannot evaluate is not a gate — it is a gate-shaped thing whose failure mode is exactly the outcome an attacker wants, and the first person to notice will be the one who wanted the release. It is also a quiet failure, which this design rejects everywhere else. And the vendor-uptime objection has a better answer than fail-open: the cached signed policy in Decision 3, a bounded timeout, and the break-glass path (#1915) — which is why #1915 must exist before the gate is enabled in production at all.

  This remains a per-(tenant, repository, environment) configuration decision (#1908). A customer may choose fail-open knowingly. The default they get without choosing is hold.

• Synchronous verdict inside the webhook. Not viable — see the 202/enqueue/debounce chain above.

• Refresh the graph, then decide. Architecturally available (GitHub holds for up to 30 days) but not in slice 1: there is no automated ingestion of any kind today (research/connector-dataflow.md — the only cron in sv0-connectors is a ServiceNow keepalive; entra-servicenow-scan.yml:87-88,100-101 passes only --json/--md, never --submit).

Consequences

• Latency is an unknown that must be measured, not cited: GitHub publishes no SLO. Sergey's latency gate can only be answered with measured numbers from the T1 advisory dogfood on SecurityV0's own repos — which is now the first environment that carries real hold time, because T0 is gone.
• T1 advisory dogfood is the measurement path, and it is not risk-free. Deleting T0 removes the "compute with no exposure" option: the first honest measurement now comes from a tier that genuinely holds our own deploys for the length of a verdict computation. Dogfood on our own repos first, size the bounded timeout from what we measure, and only then offer T1 to a design partner.
• The failure posture changes what we may claim, and it is narrower than the first draft implied. The pilot's guarantee is "when we answer, the answer is authoritative, recorded, and human-confirmed" — not "nothing deploys without us", and equally not "your pipeline is never blocked by us". When the adapter or its path to GitHub is down, deployments stay held. Say both halves to the design partner in the first conversation rather than discovering either in an incident review; it is also why the removal-detection check in Decision 7 is substrate rather than polish.
• The adapter's own availability is now a named availability dependency of the customer's deploy path. It needs the operational treatment that implies — redundancy, credential-expiry alerting, and a documented customer-side admin bypass that the design partner is told about before the first hold, not during one.
• Holding past our deadline is a live operational load, not a resolved state. Because the deadline escalates rather than deciding, the pilot needs a named fallback approver and a runbook for a hold that is still open when the first escalation lapses. Sizing the pilot without one converts a held deployment into a support incident.
• The kill switch has a documented dependency: per-tenant feature-flag writes on deployed environments are super-admin-only today (#1762 and the #1804/#1803/#1815/#1818 family). Attach that as a blocking dependency for any deployed pilot.

---