Agent runbook: create a tenant + connector on a deployed env

Goal: stand up a new platform tenant on a deployed environment, mint a connector API key that env's API will accept, co-ingest one or more connectors, and verify — without the host/DB confusion that cost a real session hours.

Read the TRAPS box first. The #1 failure is minting a key against the wrong host's Mongo and getting a 401 on ingest — there is no auth bug.

🚧 Migration in progress (Hetzner → Azure VMs + Atlas). The Mongo access pattern depends on the env's backend, and that is changing:

• Local-Mongo-container envs — Hetzner dev (dev.securityv0.com, retiring) and dev-azure — you docker exec <container> mongosh <db>.
• Atlas-backed envs — staging, prod, and post-migration dev — there is NO container. You SSH to the env's VM and run mongosh "$MONGODB_URI" (from /etc/sv0/app.env) on the VM, because Atlas only accepts connections from the VM's allowlisted egress IP — a laptop/mini cannot connect directly. See cross-env-tenant-reconciliation.

Direction of travel: Hetzner goes away; dev.securityv0.com moves to an Azure VM + Atlas like staging. When it does, use the Atlas path below for dev, and the docker exec steps become dev-azure-only history.

⚠ TRAPS (read before doing anything)

1. dev.securityv0.com is NOT vm-sv0-dev-1. vm-sv0-dev-1 is dev-azure (dev-azure.securityv0.com, Cloudflare tunnel sv0-dev-azure, container sv0-mongo, DB sv0_dev). The real dev.securityv0.com is the Hetzner host 178.156.217.150 (ash-1, containers sv0-main-*, DB sv0_platform). They are separate Mongo databases. A key minted on one returns 401 INVALID_BEARER_TOKEN on the other.
2. Every env has its OWN Mongo. Keys and tenants do not propagate. Mint / create the tenant in the DB the target API actually reads.
3. Prod's real data is Atlas, NOT a local container. Prod (app.securityv0.com) reads Atlas M10 sv0_prod. The Hetzner prod host ash-2 (178.156.245.75) also runs a stale local sv0-platform-mongo container left over from before the Atlas cutover — that is not prod data; never write to it, and never treat deploy@178.156.245.75 as a target. On the Hetzner hosts the local DB name is sv0_platform on both ash-1 (dev) and ash-2 (prod host's leftover), so the container prefix is the only distinguisher: sv0-main-* = dev, sv0-platform-* = prod host → stop. (This whole footgun disappears once Hetzner is retired.)
4. The dev Mongo container name is a decoy. It's sv0-main-mongo and its init DB is sv0_main (empty), but the API reads sv0_platform. Always mongosh sv0_platform, never sv0_main.
5. --env staging in the staff CLI validates dev.securityv0.com. There is no --env dev (sv0-platform/scripts/cli/auth.ts). --env prod → app.securityv0.com.
6. The tenant DOC must exist before ingest. Connector-key ingest resolves the tenant via findTenantBySlug and 401s ("API key is not bound to a known tenant") if it's missing — a different 401 than a bad key. The connector instance need not exist; the tenant doc must.
7. Auth is plain sha256(plaintext). No salt, no HMAC. key_hash = createHash("sha256").update("sv0_<env>_<hex>").digest("hex") (workos-provider.ts verifyApiKey). Store the hash, keep the plaintext for the connector .env.

Env → host → Mongo map

Env	Domain	Host	Mongo backend	How to reach the Mongo	DB	Touch?
dev-azure (the future dev)	dev-azure.securityv0.com	Azure vm-sv0-dev-1	local container sv0-mongo	ssh sv0admin@vm-sv0-dev-1 (Tailscale) → sudo docker exec sv0-mongo mongosh sv0_dev	sv0_dev	✅ default target
dev (Hetzner) (🚧 retiring)	dev.securityv0.com + pr-N-dev	Hetzner ash-1 (178.156.217.150)	local container sv0-main-mongo	ssh mini1@mcmini4p → ssh deploy@178.156.217.150 → docker exec sv0-main-mongo mongosh sv0_platform (no sudo)	sv0_platform (NOT sv0_main)	✅ (until cutover; separate DB from dev-azure)
staging	staging.securityv0.com	Azure vm-sv0-staging-1	Atlas Flex sv0-staging	ssh sv0admin@vm-sv0-staging-1 → mongosh "$MONGODB_URI" (from /etc/sv0/app.env; VM egress is allowlisted, laptop is NOT)	sv0_staging	⚠ careful
PROD	app.securityv0.com	Hetzner ash-2 (178.156.245.75)	Atlas M10 sv0_prod (user sv0_prod_app) — the local sv0-platform-mongo container on ash-2 is a stale leftover, not prod data	(prod is Atlas via the VM; do not touch)	sv0_prod	🚫 NEVER

Two Mongo-access patterns — pick by the "Mongo backend" column:

• Local container (dev-azure, Hetzner dev): docker exec <container> mongosh <db> — Hetzner uses docker (no sudo); Azure VMs use sudo docker.
• Atlas (staging, prod, and dev after cutover): SSH to the env's VM and run mongosh "$MONGODB_URI" on the VM — the connection string is in /etc/sv0/app.env (and the env's GitHub MONGODB_URI secret), and Atlas only accepts the VM's allowlisted egress IP. Do the tenant/key inserts from there.

The API's 3000 port is not host-exposed on Hetzner; the localhost entry point is nginx on 127.0.0.1:8080, which proxies /api/* and bypasses Cloudflare Access.

Steps

The worked example below targets Hetzner dev (dev.securityv0.com, local-container) because that's what the reference run used. The Mongo commands use docker exec sv0-main-mongo mongosh sv0_platform. On an Atlas-backed env (staging, prod, dev after cutover) swap that prefix for mongosh "$MONGODB_URI" run on the env's VM (connection string from /etc/sv0/app.env) — everything inside the mongosh heredoc is identical. On dev-azure use sudo docker exec sv0-mongo mongosh sv0_dev.

0. Confirm which host you're on

ssh mini1@mcmini4p
ssh -i ~/.ssh/id_ed25519_mini1_agentic deploy@178.156.217.150
docker ps --format '{{.Names}}'   # MUST show sv0-main-* (dev). sv0-platform-* = PROD → abort.

1. Create the tenant doc (direct Mongo insert — idempotent)

There is no create-tenant API or tool (admin/tenants.ts is GET+PATCH only; apply-canonical-tenants.ts is update-only by design). Insert directly:

docker exec -i sv0-main-mongo mongosh sv0_platform --quiet <<'EOF'
db.tenants.updateOne(
  { slug: "contoso-v2" },
  { $setOnInsert: {
      slug: "contoso-v2",
      display_name: "Contoso v2 (rebuild compare)",
      description: "Fresh 3-connector co-ingest vs contoso",
      provider_org_id: "seed-contoso-v2",   // placeholder: fine for connector ingest.
                                             // A REAL WorkOS org id is needed only for human UI login.
      status: "internal",
      tenant_class: "demo_real",
      sso_enforced: false,
      verified_domains: [],
      created_at: new Date(),
      archived_at: null
  }},
  { upsert: true }
)
EOF

Fields per sv0-platform/src/domain/tenants/types.ts (TenantDoc). _id is auto-generated. $setOnInsert+upsert won't clobber an existing tenant.

2. Mint a connector API key

Path A — direct Mongo insert (non-interactive; the agent path). Generate the plaintext + hash locally, insert only the hash:

# On your workstation (or the mini):
PLAINTEXT="sv0_dev_$(openssl rand -hex 32)"
HASH=$(printf '%s' "$PLAINTEXT" | shasum -a 256 | cut -d' ' -f1)
echo "PLATFORM_API_KEY=$PLAINTEXT"        # ← put this in the connector .env; shown ONCE
echo "prefix=${PLAINTEXT:0:12} hash=$HASH"

# On the dev host, insert the HASH only (row shape = ConnectorInstanceApiKeyDoc):
docker exec -i sv0-main-mongo mongosh sv0_platform --quiet <<EOF
db.connector_instance_api_keys.insertOne({
  connector_instance_id: "contoso-v2-entra-servicenow",  // any stable id
  tenant_id: "contoso-v2",                                // ← binds the tenant
  key_hash: "$HASH",
  key_prefix: "${PLAINTEXT:0:12}",
  created_at: new Date(),
  created_by_user_id: "agent-manual",
  revoked_at: null,
  last_used_at: null
})
EOF

The sv0_dev_ prefix is cosmetic — verification matches on the hash, not the prefix. Multiple non-revoked keys per tenant are fine; to rotate, insert a new one, swap .env, then set revoked_at on the old row.

Path B — interactive admin API (humans only). Two super-admin steps, both blocked for agents (403 ADMIN_REQUIRES_INTERACTIVE_SESSION):

POST /api/v1/admin/connector-instances                       # create the instance (super-admin)
POST /api/v1/admin/connector-instances/:instanceId/api-keys  # mint (404s if instance absent); plaintext shown once

From a logged-in super-admin browser, or sv0-platform/scripts/cli/register-tenant-connector.ts from an interactive staff session (it prints handoff guidance and exits when run as an agent).

3. Co-ingest the connectors

A demo_real tenant is populated by multiple connectors that must co-ingest for cross-system bridges to form (see connector-tenant-mapping). For a contoso-shaped tenant run all three: entra-servicenow + azure-foundry + azure-sentinel-soc, each pointed at the new slug.

# from sv0-connectors, per connector; PLATFORM_API_KEY = the plaintext from step 2
CF_ACCESS_CLIENT_ID=... CF_ACCESS_CLIENT_SECRET=... PLATFORM_API_KEY="$PLAINTEXT" \
  entra-servicenow --all --submit \
    --platform-url https://dev.securityv0.com --tenant-id contoso-v2

The key goes in header X-Api-Key; CF Access service-token headers (CF-Access-Client-Id/Secret) gate the origin. CF-free alternative — submit on-host through localhost (skips CF, still needs the key):

curl -s -w '\n%{http_code}\n' -X POST \
  http://127.0.0.1:8080/api/v1/ingest/normalized-graph \
  -H "X-Api-Key: $PLAINTEXT" -H "Content-Type: application/json" \
  --data-binary @/tmp/graph.json          # expect 202 {data:{status:"accepted"}}

⚠ Deletion detection defaults ON. A payload without scanScope is treated as mode:"full" — deletion detection enabled — and releases every existing entity (per payload connectorId × source_system) absent from the new graph; only an explicit scanScope.mode:"incremental" is additive. The circuit breaker computes one aggregate ratio per connector per sync across its source systems combined — >50% overall (identity 30%, workload/role/permission 40% per type) trips it; amputations below every threshold go through silently, and a full wipe of one source system can hide under the combined ratio. Into an empty tenant (this runbook's case) that's harmless; into a populated tenant it's the Contoso-2026-07 failure mode — see Recipe 5 of agent-auth-deployed-envs and the ADR-033 deletion_policy protections in servicenow-pdi-rebuild-and-validate.

4. Verify

docker exec sv0-main-mongo mongosh sv0_platform --quiet --eval '
  print(db.tenants.countDocuments({slug:"contoso-v2"}));
  print(db.entities.countDocuments({tenant_id:"contoso-v2","properties.status":{$ne:"deleted"}}));'
TENANT=contoso-v2 npx tsx sv0-platform/scripts/check-tenant-bridge-coverage.ts   # diagnostic, read the table

Then eyeball https://dev.securityv0.com/t/contoso-v2/exposures.

When ingest 401s

• INVALID_BEARER_TOKEN → key hash not in this env's DB. You minted against the wrong host (the classic dev-azure↔Hetzner mix-up) or a typo. Re-mint in the target DB. There is no auth bug.
• UNAUTHORIZED "API key is not bound to a known tenant" → key is fine but the tenant doc is missing in this DB. Do step 1.
• 403 (no HTML body) → Cloudflare Access gate: CF headers missing/wrong.