Skip to main content

Multi-Region Database Deployment

How app.securityv0.com serves tenants in different regions while keeping each tenant's data physically pinned to its contracted region.

This document explains the topology, request flow, and operational model for a per-region MongoDB Atlas deployment. The decision rationale (why per-region clusters and not a Global Cluster, including the control-plane / tenant-data-plane split) lives in ADR-020.

Status note (2026-05-03): ADR-020 reverted to Proposed after an adversarial review. This reference doc reflects the post-review architecture. The Phase 2 topology adds a regional worker plane that the original draft left to Phase 3, and the operational model now includes the control-plane / data-plane split and the full residency control set.

The mental model​

The platform is four layers — a global UI, a region-aware API, a regional worker plane, and the storage layer where residency actually lives. The storage layer itself splits into a single control plane and per-region tenant data planes:

LayerRegion storyWhat lives there
UI bundleGlobal (CDN edge)Static React/CSS/JS. Identical for every user.
API serverRegion-bound, but movableRoutes requests; holds session state, control-plane cache, and per-region connection pools.
Worker planeRegion-bound from Phase 2Sync, evaluator, evidence-pack assembly. Each worker pulls jobs only from its region's data plane.
Control plane (storage)Single region (EU)tenants, users, memberships — operational metadata under SCCs/DPF.
Tenant data plane (storage)Per region, residency enforcedFindings, entities, evidence, audit logs — customer security data, never leaves the contracted region.

app.securityv0.com is a single global domain because the UI is global. The data is split — but a user never sees that, because the UI is the same and the API hides the routing.

Why the storage layer splits in two: the API needs to know which region a tenant lives in before it can route the data query — and that lookup has to happen somewhere. Putting it in the same data plane as customer data would either force every request through one region (collapsing the residency story) or require duplicating the lookup table across all regions (creating a sync problem). The clean answer is to recognize that user identities and tenant configuration are operational metadata with a different residency classification than customer security data, and to give them their own home. See ADR-020 §1.

Phased deployment topology​

The architecture evolves in four phases tied to commercial milestones, not to engineering convenience. Each phase only happens when a paying customer or commercial trigger fires. Phase 0 is the pre-revenue baseline; Phase 1 onward describes the post-trigger steady state.

Phase 0 — Today (single cluster, no replica set, pre-revenue)​

What actually runs today and through the unpaid demo period. One M10 single-region Atlas cluster in aws:eu-west-1 (Ireland). No multi-region replica set. No control-plane / data-plane physical split. The two planes are separated logically by database name on the same cluster: control_plane (tenants, users, memberships) and tenant_data_eu (everything keyed by tenant_id).

Cost: ~$60/mo prod + ~$30/mo paused-off-hours staging = ~$90/mo. The $500 startup credit covers ~5 months at this rate.

SLA: 99.5% single-region. No automatic failover within EU. Acceptable while clients are unpaid demos and no contract names a 99.9% target.

What is provisioned in Phase 0: Cedar Resource Policy at the org level, BYOK with AWS KMS, PrivateLink, Database Auditing, PITR. These are M10-tier features with no cluster-count multiplier on cost — keep them on from day one.

What is NOT provisioned in Phase 0: multi-region replica set, US Atlas project, regional API/workers, Cloudflare edge router. All deferred to Phase 1+ on commercial trigger.

Promotion to Phase 1 fires when any of: first paying EU customer signs, total tenant_data_eu exceeds 20 GB or P95 latency exceeds 300 ms, procurement explicitly requires control-plane / data-plane physical separation, first US-resident client signs (paid or demo), customer contracts 99.9% SLA. See ADR-020 §0.

Phase 1 — First commercial trigger (multi-region replica set within EU + cluster split)​

When a Phase 0 → Phase 1 trigger fires, two things happen in one rollout: the control plane is promoted to a multi-region replica set within EU jurisdiction, and the EU tenant data plane is split onto its own cluster. This is the architecture described in the rest of this document.

All current and near-term tenants (MediaPro, Pelayo) live in one EU cluster. The cluster holds both databases — _control (control plane) and tenant_data (EU data plane) — because there's only one region. One API server in the EU region. One worker pool. This is what runs through MediaPro and Pelayo onboarding.

Cost: $120–$150/month all-in (M10 multi-region replica set: 3 nodes Ireland + 2 nodes Frankfurt + BYOK surcharge + PrivateLink endpoint). M30 fallback ($700–$900/mo for the multi-region setup) only if procurement specifically requires dedicated CPU; M10/M20 are both burstable. See ADR-020 §3.

Residency story: every tenant's data is in the EU (Ireland primary, Frankfurt secondary). GDPR/LOPDGDD compliant for Spanish enterprises (neither MediaPro nor Pelayo is public sector — Spain only mandates in-country storage for public-sector data; both Ireland and Frankfurt are EU jurisdictions).

HA / DR: if aws:eu-west-1 has a regional outage, Atlas automatically promotes the Frankfurt secondary to primary within ~5 minutes. Application connection string is unchanged. Single sign-on, tenant resolution, and customer dashboards remain available throughout an Ireland regional incident. See ADR-020 §8 for the SLA commitment.

Phase 2 — First US client signs (regional everything: data plane + API + workers, in one rollout)​

A US enterprise contracts and their security questionnaire requires US-resident storage. We provision the complete US plane in a single rollout: US Atlas cluster (data plane only — control plane stays in EU), US API server, US worker pool, and a Cloudflare Worker at the edge that reads the user's session and routes to the correct regional API. Earlier drafts of this doc deferred the regional API to Phase 3 to "save ops complexity"; the second-pass review (Gemini 2026-05-03) flagged that as fatal — a 5-query US dashboard load against an EU API injects ~800ms of pure network latency per page, and the residency claim depends on an SCC disclaimer that strict customers will challenge. Adding a Hetzner box per region for the API removes both problems for ~€5/month.

Cost: ~$200–$270/month at M10 baseline (EU multi-region cluster ~$120–$150 + US single-region cluster ~$80–$120) + ~€10/mo for two regional Hetzner boxes (API + worker per region) + variable cross-region egress on cache-miss control-plane lookups. M30 fallback worst case: ~$1,500–$1,700/mo for both regions at M30.

Residency story: strongest possible. Each cluster is constrained by an Atlas Resource Policy. Customer security data is fetched, processed, and serialized inside its contracted region for the entire request lifecycle. The only cross-region traffic is the periodic control-plane lookup (cached 15min) for tenant→region resolution — operational metadata under SCCs/DPF, not customer data.

HA / DR:

  • Control plane (auth, tenant resolution): automatic failover within EU jurisdiction (Ireland → Frankfurt) — RTO < 5 min. An eu-west-1 outage no longer takes down US auth.
  • EU tenant data plane: rides the same multi-region replica set — same automatic failover.
  • US tenant data plane: single-region per residency model. RTO bounded by Atlas Cloud Backup PITR (~1 hour) for a us-east-1 region-wide outage. Affects only US tenants; EU tenants unaffected.

Performance: US users get sub-50ms RTT to their data plane; EU users get sub-30ms. No 800ms N+1 trans-Atlantic latency.

Phase 3 — Future expansions (APAC, control-plane replicas, JWT region claim)​

Cost: ~$120–$280/mo for the two clusters + ~$40-100/mo for two API hosting environments (Hetzner / Cloud Run / Container Apps). Residency story: strongest possible. Tenant security data never enters a server in a different region. Control-plane lookups (operational metadata) still cross to EU but are cached aggressively. Performance: US users get sub-50ms RTT to their data; EU users get sub-30ms.

Cost progression vs Global Cluster alternative​

The per-region cluster path stays under ~$400/month through Phase 2 at the M10 baseline, climbing to ~$1,000/month worst case if a customer's procurement explicitly requires dedicated CPU (M30 fallback). The Global Cluster path requires committing to ~$3K/month the moment a second region is added — unjustifiable at pilot scale, and even at the M30 fallback the gap remains ~3×. M10/M20 are both burstable per Atlas Cluster Tiers; see ADR-020 §3 for the verified cost band.

Request flow walkthrough​

These are the concrete request paths a request takes through Phase 2 (the most representative phase).

Flow A — MediaPro user opens the dashboard​

Result: the entire request stayed inside the EU. The US cluster was never opened, never queried, and would have refused the connection if attempted (Resource Policy + auth scoping).

Flow B — Acme US user opens the same dashboard​

Result: Acme's customer security data physically lived in us-east-1 for the entire request lifecycle. Both the API server and the database are in us-east-1, so the request stays in-region from edge to disk. The only cross-region traffic is the periodic control-plane lookup for tenant→region resolution, which is operational metadata under SCCs/DPF and is cached for 15 minutes per (API instance × tenant) — amortized cost is near-zero.

Per-page latency: ~30ms (CDN edge) + ~10ms (intra-region API) + ~10ms (intra-region DB) per query. Multi-query dashboard loads stay sub-100ms total. No 800ms N+1 trans-Atlantic latency.

Earlier draft trade-off (now removed): the original Phase 2 design kept the API in EU and accepted ~240ms RTT per query for US users. That design was changed after the second-pass review (Gemini 2026-05-03) flagged it as fatal for UX. The architecture above is the corrected design.

Implementation contract​

The application changes required to support this topology are small and live almost entirely in the storage adapter and middleware layers. This section describes the contract; the actual implementation lives in sv0-platform.

Control-plane vs data-plane storage​

Two namespaces, two MongoClients, two residency classifications:

Control plane (_control database in the EU cluster — operational metadata under SCCs/DPF):

CollectionPurpose
tenantsTenant registry, includes region field. Indexed by tenant_id and provider_org_id.
usersUser identity mirror, indexed by provider_user_id.
membershipsUser × tenant × role join.

Tenant data plane (tenant_data database in each region's cluster — customer security data, residency-pinned):

entities, entity_versions, events, findings, evidence_packs, posture_snapshots, correlations, stitched_paths, connector_instances, connector_syncs, scan_scopes, connector_instance_api_keys, audit_log, ...

Tenant configuration row:

FieldTypeNotes
tenant_idstringAlready used by X-Tenant-Id middleware
regionenum (EU, US, APAC, ...)Immutable under normal operation
provider_org_idstringWorkOS org id, indexed
namestringDisplay name
contracted_residencystringe.g., "EU (Ireland)" — for audit response

Connection management​

The storage adapter owns three things:

controlPlaneClient = MongoClient(MONGO_URI_CONTROL, { maxPoolSize: 20 })   // always EU
dataPlaneClients = {
  EU: MongoClient(MONGO_URI_EU, { maxPoolSize: 20 }),
  US: MongoClient(MONGO_URI_US, { maxPoolSize: 20 }),
}
regionCache = LRU(maxAge: 15min)                                            // tenant_id → region

Initialized once at startup. The adapter exposes:

  • controlPlane() — returns the control-plane client. For tenant lookup, user lookup, membership resolution.
  • forTenant(tenantId) — resolves region from cache or control plane, returns the right data-plane client.
  • forRegion(region) — explicit region selection. Used by region-scoped workers and the migration runner.

All existing storage-adapter call sites that take tenantId automatically route correctly. Control-plane operations (auth, tenant resolution) explicitly call controlPlane().

maxPoolSize=20 per client (down from the Node driver default of 100) to keep aggregate connection count bounded as region count grows. At Phase 2 with 4 API instances + 2 regions × 2 clients each, that's 4 × 3 × 20 = 240 max connections — safely inside the M10 ceiling of ~1,500.

Middleware​

The existing X-Tenant-Id middleware extends to:

  1. Resolve provider_org_id → tenant_id via control plane (cached).
  2. Resolve tenant_id → region via control plane (cached, 15min TTL).
  3. Attach both tenantId and region to the request context.

Every downstream handler receives both for free.

Region-tagged connector API keys​

Connector API keys carry their region in the prefix:

  • EU tenant key: sv0_prod_eu_<32-byte-random>
  • US tenant key: sv0_prod_us_<32-byte-random>

The bearer middleware reads the prefix → routes the hash lookup to the correct region's data plane → no fan-out across clusters on the connector ingest hot path. See ADR-020 §4.

Worker plane (regional from Phase 2)​

Each region runs its own worker pool with its own scheduler. A worker is configured with one region's MONGO_URI_<REGION> and only claims jobs from that cluster's scan_scopes. No cross-region claim primitive. Operational footprint per region: a Hetzner CX22 (~€5/mo) or equivalent.

Connector secrets live in regional 1Password vaults (sv0-bots-eu, sv0-bots-us); connector_instances.credentials_ref carries the region-aware URL.

Migration runner​

A scripts/migrate.ts runner accepts a --region flag (defaults to all configured regions) and applies pending migrations to each cluster in turn. Two migration namespaces: _control (applied to the EU cluster only) and tenant_data (applied to every region's data plane). The runner reports per-region success/failure independently and does not abort other regions on a single-region failure.

Expand-and-contract is mandatory for breaking changes​

Every breaking schema change MUST follow expand-and-contract across at least three deploys. The API code is deployed globally — there is no way to atomically synchronize a schema change across N independently-migrated clusters. A single-step "drop a column" migration that succeeds in EU and fails in US would instantly produce 500s for US tenants on the next API deploy.

The pattern:

  1. Migration N — expand. Add the new field, populate via dual-write, do not remove the old field. Deploy the API to write to both. All clusters now hold both shapes.
  2. Migration N+1 — switch reads. Update the API to read from the new field. Old field still exists. Deploy.
  3. Migration N+2 — contract. Drop the old field. Deploy.

Enforcement:

  • The runner refuses to apply a migration tagged breaking unless the previous deploy is at least N-1 in the sequence (tracked in _migrations_state in the control plane).
  • Pre-PR review checklist requires every schema change to be tagged either non-breaking (additive) or breaking (with the three-step plan documented).
  • Rollback procedure for a single-region failure: pause deploy, fix the failed cluster, resume. Never fast-forward past a partial migration.

See ADR-020 §9 for the full rationale.

Residency control set (the procurement-grade evidence)​

ADR-020 §6 enumerates the seven Atlas configuration knobs that together enforce residency. The Resource Policy alone is not sufficient — it covers cluster region only. Every cluster ships with the full set locked down:

KnobRequired settingWhere it's configured
Cluster regionSingle allowed regionResource Policy (Cedar) at project level
Backup snapshot regionSame as cluster regionCloud Backup policy → Region
Backup copy regionNone enabledCloud Backup policy → Copies disabled
KMS keyAWS KMS in same regionBYOK config → key ARN
Replica set membersAll in cluster's regionCluster topology
Database audit logCluster-local destinationDatabase Auditing config
Application logsPer-region observability projectApp-side logger config (see below)

Procurement evidence pack per region: Resource Policy export + 6 screenshots, bundled in docs/compliance/residency-evidence-<region>.md. Refresh quarterly.

Observability and audit log routing​

Application logs already carry tenant_id by design — see PR #763 which deliberately moved per-tenant verdict attribution from Prometheus metrics to structured logs (the /metrics surface was a tenant-enumeration oracle).

Do not strip tenant_id at the logger. Per-region log destinations are the right answer:

  • EU API + EU workers → Grafana Cloud EU project
  • US workers (Phase 2) and US API (Phase 3) → Grafana Cloud US project

Routing is by environment variable at process startup (GRAFANA_LOKI_URL per deployment). No app-side change. Two Grafana dashboards instead of one.

Database audit logs stay cluster-local (Atlas Database Auditing writes to the cluster). Residency-safe by construction. Do not export them to a centralized observability stack.

Operational model​

What is configured per-cluster​

ConcernPer-cluster?Notes
Backup policyYesAtlas Cloud Backup, daily snapshots, 7-day PITR
Resource Policy (Cedar)YesOne per Atlas project, restricts to the project's region
Network accessYesIP allowlist or VPC peering, region-local
Encryption (BYOK)YesAWS KMS key per region
Monitoring / alertsYesAtlas Project alerts → ops Slack channel
User accessCentralizedAtlas org users span all projects; project-level roles assigned
SchemaIdenticalAll clusters run the same schema; migrations applied per-cluster

What "adding a new region" actually involves​

When a new commercial trigger fires (first APAC client, first single-tenant deployment), the work to add a region is:

  1. Provision the Atlas project + cluster via Terraform (per ADR-019).
  2. Apply the region-specific Resource Policy.
  3. Add a MONGO_URI_* secret for the new region (e.g., MONGO_URI_APAC) to deployment secrets.
  4. Restart the API to pick up the new connection.
  5. Run the migration runner against the new region: npm run migrate -- --region APAC.
  6. Insert tenant config records pointing new tenants at the region.

End-to-end: half a day of work the first time, an hour from then on once Terraformed.

Backups, DR, and RPO/RTO​

Each cluster has its own backup and recovery posture, scoped to itself. RPO/RTO targets apply per-region: an EU cluster outage does not affect US tenants and vice versa. This is a feature, not a bug — a US tenant procurement question "what is your RTO for our data?" is answered by the US cluster's RTO, independently of any other region.

Tenant region change procedure​

Rare but possible (e.g., a customer reorganizes their data residency commitments). Procedure:

  1. Schedule a maintenance window with the customer.
  2. Pause connector sync for the tenant.
  3. Export the tenant's documents from the source cluster (mongoexport filtered by tenant_id).
  4. Import to the destination cluster (mongoimport).
  5. Update the tenant config record's region field.
  6. Resume connector sync; reissue any region-bound credentials.
  7. Verify queries hit the new cluster, then drop the source-cluster data after a cooling-off period.

Treated as a manual operation, not an automated capability.

What this design deliberately does NOT do​

  • No cross-region queries. The application never asks "give me findings across all regions." If a product feature ever needs that, it goes through an aggregation layer fed by per-region rollups, not by removing the residency boundary at the operational store.
  • No automated tenant region change. Region is set once at tenant onboarding and only changed by explicit operator action.
  • No global clusters or sharding. See ADR-020 for the cost rationale.
  • No multi-region replica sets pretending to be residency. A multi-region replica set has its primary in one region — secondaries are HA-only. That topology does not satisfy residency and is not used.