Skip to main content

Runbook: ServiceNow PDI change → rebuild + validate a demo tenant

Use when a ServiceNow Personal Developer Instance (PDI) that feeds a demo tenant is reclaimed, re-provisioned, or otherwise swapped, and you need to rebuild that tenant's data from the connectors. Distilled from the 2026-07 recovery of Contoso after dev217540 was reclaimed and replaced with dev284323.

This is the strategy + validation playbook. The mechanics of creating a tenant and minting a connector key on a deployed env (incl. the Hetzner-vs-Azure/Atlas Mongo-access split) live in create-tenant-and-connector-on-deployed-env.

Core principle: rebuild into a FRESH tenant, never in-place​

A single-window co-ingest of all the tenant's connectors into an empty tenant reconstructs a working graph — including cross-system reach and the demo use-cases — deterministically. Do NOT co-ingest into the existing populated tenant: ingest's default deletion detection releases every entity absent from the new scan, and the circuit breaker only trips on mass deletion — so a partial rebuild silently amputates nodes with no error. Once ADR-033 (sv0-platform#1864) is merged and the tenant is configured, deletion_policy: "disabled" removes the amputation risk — no ingest can delete entities without an explicit, audited server-side maintenance override, and the protected_entities floor is never deleted at any ratio or mode. It does not make an in-place co-ingest safe on a v1 tenant after a PDI swap: every rotated-sys_id node re-ingests as a duplicate while the stale originals linger (deletions are off). The fresh-tenant rebuild remains the recovery path until the tenant is on v2 identity keys (see the last section).

Playbook​

  1. Provision the new PDI + update creds. Update the connector .env (SERVICENOW_INSTANCE + username/password). Verify the instance is awake — an authenticated API call (/api/now/table/sys_user?sysparm_limit=1) returns JSON, not the "Instance Hibernating" HTML page.
  2. Prevent hibernation. PDIs sleep after ~2h idle. Run an authenticated hourly keepalive — anonymous pings do NOT reliably count as activity (and can't wake a sleeping instance; only the developer portal does). Reference impl: a launchd job on the mini (~/.sv0-sn-keepalive.sh doing an authed API GET, reading creds from the connector .env; com.sv0.sn-keepalive, hourly). Do not rely on an anonymous GitHub-Actions ping.
  3. Co-ingest ALL the tenant's connectors into a fresh tenant, in one window. Tenant/key mechanics: create-tenant-and-connector-on-deployed-env. Cross-system bridges form only when all partner connectors co-ingest into the same tenant in the same window (the "#1669 trap"); run every connector that feeds the tenant (Contoso = entra-servicenow + azure-foundry + azure-sentinel-soc). See connector-tenant-mapping for each tenant's connector set.
  4. Build the sys_id remap (new → old). A new PDI rotates every sys_id. Match new→old by natural key (name / api_name / endpoint_url). Most ServiceNow nodes are keyed on stable natural keys and match without a remap; only the provisioned records rotate. (2026-07: 8 of 11 rotated ServiceNow nodes mapped uniquely by name; remap table captured in the recovery notes.)
  5. Validate against the original — deterministic, no heuristics. Compare the new tenant vs the original in natural-key space (apply the remap first, since the slug is baked into every _id):
    • Entity parity by (source_system, source_id) per source_system — report exact / losses / additions; allowlist the enumerated known deltas.
    • Relationship-type histogram — assert the edge-type set matches and AUTHENTICATES_AS count > 0 (cross-system reach reproduced).
    • Named demo use-cases as hard pass/fail (e.g. ticket-router → Azure chain; agent → Defender isolateMachine; EDR stub).
    • Findings by type — structural findings must match; drift/temporal findings (scope_drift, reachability_drift, privilege_justification_gap, …) legitimately differ on a fresh tenant (no accumulated baseline) — informational, never a parity FAIL.

Gotchas (all bit us in 2026-07)​

  • New PDI rotates sys_ids AND OAuth client_ids. Named cross-system bridges (e.g. sn-ticket-router → Azure) key off client_id and may not reproduce — match bridges by endpoint, not client_id. Expect a small AUTHENTICATES_AS delta.
  • Each env has its own Mongo. dev.securityv0.com (Hetzner today; Atlas after the migration) is NOT vm-sv0-dev-1 (dev-azure). Create the tenant + mint the key in the DB the target API reads. On Atlas envs there's no docker exec — SSH to the env's VM and mongosh "$MONGODB_URI" (VM egress is allowlisted). See create-tenant-and-connector-on-deployed-env.
  • Some PDI objects are hand-created, not emitted by the provisioning script, and won't reconstruct. Enumerate + track them rather than assuming a lossless rebuild (2026-07: the gpt-nano AI-triage subgraph + automation_disabled, sv0-connectors#267).
  • --tenant-id on the connector sets the graph tenant; the ingest key's bound tenant_id is what actually scopes the write. They must agree or ingest 403s.

Permanent fix: PDI-immune identity (ADR-033) — supersedes the in-place remap MVP​

An earlier revision of this runbook recommended a "less-destructive in-place refresh": post-process the connector's --graph-json output, rewrite the new sys_ids / client_id back to the old values, inject scanScope: {mode: "incremental"}, and submit the post-processed graph into the existing tenant (researched as sv0-connectors#268). That approach is superseded — do not use it. Do not post-process connector output, do not rewrite IDs (in JSON or in Mongo), and do not rely on injecting incremental to suppress deletions. The durable answer is ADR-033 — PDI-immune ServiceNow identity (lands with sv0-documentation#430; code in sv0-connectors#273 + sv0-platform#1864):

  • Axis A — versioned typed natural keys (connector). With --identity-key-version v2 (env: IDENTITY_KEY_VERSION=v2; default stays v1), entra-servicenow emits typed natural keys for the structural ServiceNow nodes (oauth-provider:<name>, entra-cred:<client_id>:<oauth-provider-name>, rest-message:<name>, business-rule:<table>:<name>, script-include:<api_name or name>, scheduled-job:<name>, flow:<scope>:<internal_name>, flow-endpoint:<owning-flow key>) instead of the rotating sys_id. A reprovisioned PDI then lands on the same _ids — the remap step in this playbook disappears for v2 tenants. Under v1 (the default) the source IDs and identity behavior are unchanged from today; nothing changes for existing tenants until the platform requires v2 for that tenant × connector instance.
  • Axis B — server-enforced deletion policy (platform). The tenant doc carries deletion_policy: "disabled" plus a protected_entities floor. The platform forces deletion detection off regardless of what the payload declares, never deletes protected entities at any ratio or mode, and rejects ingest at admission (409) when the declared identityKeyVersion mismatches what the tenant requires for the authenticated connector instance. A deliberate cleanup goes through a server-side, time-bounded, single-use maintenance override — never a connector flag.
  • Migration to v2 is manifest-driven, not ad-hoc. The authoritative manifest comes from generate_migration_manifest.py (fail-closed: v2-input-only, 1:1 natural-key join, final-_id collision check across emitted + retained entities, deletion denominators per connector-instance × source_system) and is executed only after a rehearsal on a clone tenant and explicit go-approval. Until that approval, the affected tenant stays untouched.

Until ADR-033's code is merged and the tenant is flipped to v2, the fresh-tenant rebuild above remains the only safe recovery from a PDI swap.