Skip to main content

Secrets & Credential Management

Bottom line. SecurityV0 has two runtime-secret domains, and the target design resolves both from the same substrate: Azure Key Vault read by a VM managed identity. (1) Our own platform runtime secrets are boot-fetched into a tmpfs file that never touches disk. (2) Customer/connector credentials are resolved per scan from a per-tenant Key Vault by a broker that maps tenant → vault server-side. The direction of travel is federation, where no long-lived secret is exchanged at all.

Read §0 before quoting anything here. Much of this document describes a target state. The connector Key Vault broker is merged; the production boot-fetch and the per-tenant vaults are not yet applied. And one boundary is weaker than it looks: IMDS is a VM-level boundary, not a process boundary (§3c) — today any code on the VM, including a connector subprocess, can request a token for the VM's managed identity.

This doc complements two neighbours:

0. Implementation status — what is actually true today​

Nothing below should be read as "operational" unless this table says so. Statuses as of 2026-07-22.

CapabilityDecisionCode mergedInfra authoredInfra appliedProvisionedE2E validated
Connector identity model (ADR-029)✅ accepted—————
Connector Key Vault broker (azure_keyvault)✅✅ platform#1874——❌❌
Connector paste store (paste, encrypted-in-Mongo)✅✅ in use today (#1277)n/an/a✅✅
Platform boot-fetch of app secrets (deploy side)✅❌ open draft platform#1873———❌
Prod Key Vault + managed identity + cloud-init✅—✅ open draft infra#134❌ not applied❌❌
kv-sv0-tenant-* vault, RBAC, tenant → vault map✅broker reads it❌ not authored❌❌❌
TPX credential deposit into a tenant vault✅———❌❌
Connector IMDS isolation (see §3c)⚠️ open❌❌❌❌❌
Key Vault diagnostics / alerting / access reviews (§7)⚠️ open—❌❌❌❌

What that means in plain terms: the broker code is merged and unit-tested, but no production Key Vault, managed identity, or tenant vault exists yet, the platform's own secrets are not yet boot-fetched (they still ship via the current deploy path), and the connector master key still comes from the deploy .env, not Key Vault. Treat §2 and most of §3 as design pending deployment.

1. The two runtime-secret domains​

Two domains share a substrate but differ in scope, lifetime, and who reads them.

Scope note. "Two domains" covers runtime secrets. A third category — control-plane / provisioning secrets — is out of scope here and is not delivered by this design: the Tailscale enrolment key, the Terraform-generated SSH private key, Cloudflare tunnel material, Terraform Cloud provider credentials, MongoDB Atlas keys, and the GitHub Actions Cloudflare-Access health-check secret. Some of those legitimately live in Terraform state or CI. See infra#134 for the residual Tailscale state exposure it documents.

Domain A — platform runtime secretsDomain B — connector / customer credentials
WhatMongo URI, WorkOS keys, session-cookie password, metrics bearer, the credential-encryption master key, GHCR/Grafana/tunnel tokensA customer's AWS key, Entra app secret, ServiceNow OAuth client
Vault (target)kv-sv0-prod (one, platform-wide)kv-sv0-tenant-<slug> — SV0-managed, one per tenant, in SV0's own Azure subscription
Who reads itsv0-kv-bootstrap at bootThe credential broker, per scan
Where it landstmpfs /run/sv0/app-secrets.envRAM only — the run's process env, for one scan
In Mongo?NeverDepends on provider. azure_keyvault → only a reference. paste → AES-256-GCM ciphertext in Mongo, decrypted per scan. Never plaintext, either way.
Statusdesign pending deployment (§0)broker merged; vaults not provisioned (§0)

Vault ownership. kv-sv0-tenant-* is a SecurityV0-managed vault dedicated to a customer, inside our Azure subscription — not a customer-owned "bring your own vault." Customer-owned vaults and cross-tenant federation are future options, not the near-term design.

2. Domain A — platform runtime secrets (boot-fetch) — target, not yet applied​

Status: design pending deployment. The deploy side is open draft platform#1873; the infrastructure is open draft and unapplied infra#134. Today's production still uses the pre-existing deploy path.

2a. The intended chain​

On every boot, a systemd unit (sv0-kv-bootstrap) authenticates as the VM's managed identity, reads platform secrets from kv-sv0-prod over the private endpoint, and writes them to tmpfs. Non-secret configuration is deploy-managed and lives on the OS disk — kept in separate files so the sensitive half never touches persistent storage.

2b. The secret / config split​

FileBacking storeWritten byContents
/run/sv0/app-secrets.envtmpfs (RAM), 0640, sole writer sv0-kv-bootstrapboot-fetch, every bootEvery value from the table below
/etc/sv0/app-config.envOS diskthe deploy (render_and_apply)Image ids, endpoints, feature flags — no secrets

Required platform secrets in kv-sv0-prod (names contract-shared between the infra Terraform and the deploy workflow):

Key Vault secretApp env var
sv0-prod-mongodb-uriMONGODB_URI
sv0-prod-credential-encryption-keySV0_CREDENTIAL_ENCRYPTION_KEY (the connector paste-store master key — §5)
sv0-prod-workos-api-keyWORKOS_API_KEY
sv0-prod-workos-authkit-domainWORKOS_AUTHKIT_DOMAIN
sv0-prod-workos-webhook-secretWORKOS_WEBHOOK_SECRET
sv0-prod-session-cookie-passwordSESSION_COOKIE_PASSWORD
sv0-prod-workos-app-claudecode-client-idPROD_WORKOS_APP_CLAUDECODE_CLIENT_ID
sv0-prod-workos-app-claudecode-client-secretPROD_WORKOS_APP_CLAUDECODE_CLIENT_SECRET
sv0-prod-metrics-bearer-token/metrics bearer — single source for BOTH the Alloy scrape and the app, so scrape/app drift is eliminated by construction
sv0-prod-ghcr-token, sv0-prod-grafana-cloud-token, sv0-prod-tunnel-tokeninfra tokens → other /run/sv0/*.env files

Optional (a clean 404 is a valid "not configured" state): sv0-prod-rescue-webhook-url, sv0-prod-data-health-prober-client-id.

2c. Why tmpfs — the backup-exposure boundary​

/run is tmpfs: RAM-backed, re-created empty every boot. That is the control, not a convenience. Azure VM Backup cannot exclude the OS disk, so anything under /etc is captured in every restore point. Keeping platform secrets under /etc would mean plaintext secrets in every backup, retained per policy and restorable by anyone with backup-operator rights.

Two further properties of boot-fetch, scoped precisely:

  • Platform application secrets stop travelling through the deploy pipeline. The prior shape shipped them via az vm run-command --parameters, which lands values in the Azure Activity Log (90-day retention) — the #936 exposure. This does not mean "no secret anywhere touches CI or Terraform": control-plane and provisioning secrets still do (§1 scope note).
  • Platform application secrets are not rendered into custom_data or Terraform state. Other infrastructure material (e.g. Tailscale enrolment) remains in state; see infra#134.

3. Domain B — connector / customer credentials​

Two providers exist today, and they differ in where the value rests. Both resolve through the same broker interface (ADR-027), and in both cases the connector subprocess receives a decrypted env map for one scan only.

ProviderWhere the value restsStatus
azure_keyvaultPer-tenant Key Vault; Mongo holds only a referenceBroker merged (#1874); no vault provisioned yet
pasteAES-256-GCM ciphertext in Mongo (connector_credentials), decrypted per scanIn use today (#1277)

3a. Per-scan resolution (the azure_keyvault provider)​

3b. What the broker's guards do — and do not — buy​

  1. Server maps tenant → vault; the request never does. The map is deploy config (SV0_AZURE_KEYVAULT_TENANT_VAULTS). A credentials_ref supplies only a name suffix, never a vault URL — accepting one would be an SSRF / credential-confusion surface. This is a real, enforced guard.
  2. Secret names are collision-resistant across tenants and refs. The name is the server-controlled tenant slug plus a truncated sha256("<tenant>:<ref>") (48 bits). This is collision-resistant at any realistic scale — it is not mathematically injective, so "can never collide" would be wrong. The server-controlled slug prefix is what actually carries the tenant separation.
  3. A bundle cannot set process-control env vars. parseBundle rejects loader/interpreter keys (PATH, LD_*, DYLD_*, NODE_OPTIONS, PYTHONPATH, …), so a malformed or hostile vault bundle cannot hijack the subprocess it is layered onto. Real, enforced.
  4. The broker holds no client secret of its own — it authenticates as the VM's managed identity. True, but see §3c: that identity is reachable by anything on the VM.

Separate vaults do not, today, create an on-host tenant boundary. What per-tenant vaults do buy is real but narrower: cleaner audit trails, operator RBAC separation, independent deletion/retention at offboarding, and protection against accidental mis-routing. What they do not buy — while one VM identity is granted across all mapped vaults — is containment of a VM compromise.

3c. ⚠️ Residual risk — IMDS is a VM boundary, not a process boundary​

This is the most important caveat in this document, and it is not yet mitigated.

Azure's instance metadata service (IMDS, 169.254.169.254) issues managed-identity tokens to any code running on the VM. Microsoft states this explicitly: all code on the VM can request tokens for the identities attached to it. The connector driver currently spawns an ordinary subprocess with an environment allowlist and no separate UID, no network namespace, and no IMDS egress restriction (src/workers/connector-driver.ts).

So the accurate statement is: the environment allowlist stops a connector from inheriting platform secrets; it does not stop a connector from minting a token and fetching them. Claims like "only the broker can read credential values" hold against application code paths and accidental disclosure — not against a compromised connector subprocess.

Before TPX production, one of the following must be true, and validated by a negative test (§7):

  • IMDS is proven unreachable from connector subprocesses (egress firewall / netns), or
  • connectors run under a separate UID + network namespace with IMDS blocked, or
  • the broker/worker runs under a separately isolated identity from the one connectors could reach.

Until then this residual risk is disclosed, not solved. Tracked as an open item in §0.

3d. Reference vs value​

For the azure_keyvault provider, Mongo stores a reference (provider + name), never a value. For the paste provider Mongo stores ciphertext. The precise, defensible claim is therefore: no plaintext customer secret is stored in Mongo — not "no customer secret in the database."

4. Credential intake — the two real paths​

There is no customer-facing credential UI today. Both intake paths terminate with a SecurityV0 operator, and they are genuinely different mechanisms.

Rules that matter:

  • Classify before receiving. Every inbound credential is one of: trust-id (public, not a secret), federation-config, machine-secret, private-key, or human-cred. The class dictates handling.
  • Guest-vault membership, not item share-links. A 1Password share-link hands over a static copy that never receives rotations; guest membership does. (Teams Starter Pack currently allows a small number of guests, each scoped to one vault — see 1Password guest docs.)
  • Removing a guest does not un-copy a secret. Guest removal ends future access only; anyone who already copied a value still has it. Offboarding must therefore pair removal with rotation/revocation at the source system wherever the credential is still live (§7).
  • The bridge sunsets. One guest vault, two named admins, short-lived. It exists to move the credential to our tenant vault, then it is removed.

5. The credential-encryption master key​

The paste store encrypts every value at rest with AES-256-GCM under a master key, SV0_CREDENTIAL_ENCRYPTION_KEY (base64, openssl rand -base64 32 — not hex).

  • Today: that key is supplied from the deploy .env.
  • Target: it becomes a Domain-A secret (sv0-prod-credential-encryption-key) boot-fetched from kv-sv0-prod — pending #1873/#134 (§0). At that point the key protecting Domain-B material is itself delivered by the Domain-A chain, and both terminate in Key Vault.

If the key is unset the paste store is inert (mirroring the broker being inert without its map): a missing master key degrades to "no paste store," never to "plaintext."

6. Where this is heading — the federation endgame​

Long-lived shared secrets are the worst-case shape. For first-class clouds the industry exchanges a trust relationship, not a secret. The broker interface is deliberately unchanged by this: for a federated cloud the broker mints a token instead of fetching a stored secret.

ConnectorTarget patternPer-tenant secret?
AWSCross-account assume-role + external-IDNo
Entra / GraphMulti-tenant app + admin consent + WIF (cert transitional)No
GCPWorkload Identity FederationNo
GitHubSV0 GitHub App (one private key)No
ServiceNow / other SaaSOAuth client_credentials → tenant KVYes

The blocker is code, not design: connectors hardwire ClientSecretCredential today. The injectable credential provider (connectors#274) is what lets the broker mint instead of fetch.

Note on control framing: the flat env-broker shape does not satisfy SecurityV0's selected tenant-isolation and least-privilege control design supporting SOC 2 CC6.1/CC6.6 and ISO 27001:2022 A.8.15 / A.5.15. An architecture does not by itself "pass" or "fail" an audit — formal statements wait on the control matrix and auditor/counsel review.

7. Operations — what must exist before this is production​

These are open items (§0), listed so they are not discovered at cutover.

  • Staged rotation and rollback. Not "set the value and restart." Key Vault versions the secret; the runbook needs: deposit new version → restart one replica → verify health → roll the second → keep the prior version until verified → documented rollback to the prior version. For paste-provider material, rotation means re-encrypting under the new master key, so master-key rotation needs its own re-wrap procedure.
  • Diagnostics and alerting. Key Vault AuditEvent diagnostic settings shipped to the log workspace, with alerts on: denied SecretGet, access-policy/RBAC change, secret delete/purge, and any SecretGet from an unexpected principal.
  • PIM / break-glass. Named standing roles kept minimal; elevation through PIM with approval and expiry; a documented break-glass identity with its own alerting.
  • Access-review cadence. Periodic review of who holds Key Vault Secrets User / Secrets Officer on each vault, and of the tenant → vault map.
  • Tenant offboarding. Vault deletion (with purge-protection/retention decided in advance), map entry removal, and source-system credential revocation — plus the 1Password guest-removal + rotation pairing from §4.
  • Backup / restore and reboot tests. Because secrets are tmpfs, a reboot must be proven to re-fetch cleanly and fail closed when a required secret is missing.
  • Negative tests. Prove a connector subprocess cannot reach IMDS (§3c) and cannot read a vault outside its tenant mapping. This is the test that converts §3c from disclosed risk to enforced control.

8. Environment and subscription separation​

Demo, dev, staging and production identities must be separate principals with disjoint vault access, even where they temporarily share an Azure subscription. Specifically: separate resource groups, separate Terraform state/workspaces, separate managed identities, separate vaults, and RBAC granted per-environment. No demo or staging identity may hold any role on kv-sv0-prod or on any kv-sv0-tenant-* vault. A shared subscription is an accepted interim, a shared identity is not.

9. Portability — what survives a move off Azure​

The security model is portable; the implementation is Azure-specific. The invariants to preserve in any future AWS (or other) topology:

Invariant (portable)Azure adapter (replaceable)AWS equivalent
Workload identity, no standing secret in the appUser-assigned managed identity + IMDSIAM role for the instance/task
Per-client secret storekv-sv0-tenant-* Key VaultSecrets Manager per client, or per-path KMS-scoped
Server-side tenant → store mappingSV0_AZURE_KEYVAULT_TENANT_VAULTSsame map, different resolver
Per-run resolution, value only in RAMKey Vault brokerSecrets Manager broker
Secrets never on the persisted disktmpfs /runtmpfs, or task-level secret injection

The IMDS caveat (§3c) has a direct AWS analogue (IMDSv2 hop limits / credential scoping), so the isolation work is not wasted on a migration.

10. What we can say externally — and what must wait​

Defensible today:

  • Customer credentials are never stored as plaintext in our database; azure_keyvault stores only a reference, paste stores AES-256-GCM ciphertext.
  • Connector credentials are resolved per scan and held in memory for the duration of one scan.
  • The connector interface is read-only toward customer systems, and access is outbound-only from a single egress IP.
  • The credential-intake path is operator-mediated, with classification before receipt and a short-lived 1Password guest bridge.

Must NOT be claimed until §0 shows applied + validated and §7's negative test passes:

  • "Per-tenant vault isolation" as a containment claim — until IMDS isolation (§3c) exists, a VM compromise spans every mapped tenant.
  • "Platform secrets never touch disk / never appear in backups" — true only once boot-fetch is deployed (#1873 / #134).
  • Any statement that Key Vault is the platform's runtime secret source — not yet true in production.
  • Any SOC 2 / ISO pass/fail phrasing, pending the control matrix and auditor review.

References​

Architecture​

Decisions​

  • ADR-029 — connector identity model (phased) — decision of record for Domain B
  • ADR-027 — automated connector pipeline (broker slices)
  • ADR-022 — Azure compute landing zone (Key Vault, MI)
  • ADR-024 — Azure deploy lifecycle

Research​

External​