Execution Evidence Linkage Plan

Date: 2026-02-15 Status: Draft v2 — review findings addressed Scope: Connector + Platform + UI

---

1. Problem Statement

Automations and execution evidence are completely disconnected in the product. A CISO sees "Execution count: 42 in the last 30 days" on an automation but has no way to drill into the proof. The number is an assertion without backing.

Two parallel data systems give contradictory impressions:

1. Azure sign-in evidence arrives as proper execution_evidence nodes from the Entra connector, gets stored in MongoDB via graph-transformer.ts → sync-ingestion.ts → insertExecutionEvidence(). The evaluator rules query it internally. But there is no API endpoint and no UI to display it.

2. ServiceNow execution data (flow runs from sys_flow_context, job runs from sys_trigger) is collected by the connector but flattened into 3 properties on automation nodes: execution_count_30d, last_observed_execution_timestamp, execution_evidence_refs. The refs are dangling ServiceNow sys_id values that point nowhere in our platform.

The result: the evaluator sees one reality (via getExecutionEvidence()), the UI shows another (via flattened properties), and ServiceNow execution data exists in neither system properly.

Why This Matters

• Breaks the evidence-grade value proposition. Two evaluator rules (dormant_authority, privilege_justification_gap) depend on execution evidence. Customers cannot see what the evaluator saw. A finding that says "no execution evidence on record" is unverifiable.
• Architecture diagram mismatch. 01-data-model.md shows EXECUTION_EVIDENCE EVIDENCES AUTOMATION as a relationship. The EVIDENCES relationship type does not exist in code. The diagram is aspirational, not actual.
• ServiceNow BR/SI limitation. ServiceNow has no execution logs for Business Rules or Script Includes. This is a fundamental platform limitation. The current approach (execution_count_30d = 0) is correct for these types. Proxy evidence via Azure sign-in correlation (RUNS_AS chain) is the best alternative.

---

2. Current State

What works (Azure/Entra sign-in evidence)

Step	Component	Status
Emit evidence nodes	Connector transformer.py:_add_sign_in_node()	Working
Transform to ExecutionEvidenceDoc	Platform graph-transformer.ts:nodeToExecutionEvidence()	Working
Store in MongoDB	sync-ingestion.ts → insertExecutionEvidence()	Working
Evaluator queries evidence	dormant-authority.ts, privilege-justification-gap.ts	Working
API endpoint to expose evidence	None	Missing
UI to display evidence	None	Missing

What is flattened (ServiceNow execution data)

Step	Component	Status
Fetch flow runs from sys_flow_context	servicenow_client.py:discover_flow_executions()	Working
Fetch job runs from sys_trigger	servicenow_client.py:discover_job_executions()	Working
Emit as execution_evidence nodes	Not implemented	Missing
Store as flat properties on automation	transformer.py:_enrich_automation_properties()	Working (wrong approach)
execution_evidence_refs (dangling sys_ids)	transformer.py	Present but resolve to nothing

ServiceNow tables providing deterministic evidence

Table	Automation Type	Join Key	Fields for Evidence
sys_flow_context	Flow Designer flows	flow → sys_hub_flow.sys_id	sys_id, state, started, ended, run_as
sys_trigger	Scheduled jobs	document → sysauto_script.sys_id	sys_id, last_action, next_action, state
None (no table exists)	Business Rules	N/A	N/A — fundamental SN limitation
None (no table exists)	Script Includes	N/A	N/A — fundamental SN limitation

---

3. Proposed Architecture

3.1 New Relationship Type: EVIDENCES

All surfaces that must be updated (adding to one is not enough — the Zod schema on the ingest endpoint will 400-reject unknown edge types):

#	File	What to add
1	src/domain/graph/relationship-types.ts	Add "EVIDENCES" to RELATIONSHIP_TYPES array
2	src/ingestion/types.ts	Add "EVIDENCES" to NormalizedEdgeType union
3	src/api/routes/ingest.ts	Add "EVIDENCES" to EDGE_TYPES array (feeds Zod z.enum())
4	ui/src/components/graph/GraphFilterSidebar.tsx	Add "EVIDENCES" to RELATIONSHIP_TYPES

Not needed in: path-materializer.ts (EVIDENCES is not a forwarding edge), chain-builder.ts EDGE_ROLE_MAP (EVIDENCES is not a chain traversal edge), paths.ts FORWARDING_EDGE_TYPES, evaluator rules, evidence-pack builder. These systems filter for specific edge types they care about and ignore the rest.

Direction: evidence → entity (the evidence "is about" the entity). This follows the same pattern as findings (CONCERNS finding → entity).

Staging (E1 vs E2): E1 adds EVIDENCES to all four type surfaces so the platform accepts it. E2 has the connector start emitting it. This is "accept first, emit later" — the type exists but no edges use it until E2. This is intentional to avoid a coordinated deploy.

3.2 Entity ID Resolution

Update nodeToExecutionEvidence() in graph-transformer.ts:

1. Look for EVIDENCES edges where source = evidence nodeId → resolve target
2. Fallback: look for EXECUTES_ON edges where target = evidence nodeId → resolve source (legacy)
3. Fallback: check node.properties.entity_source_id (qualified by sourceSystem)

Fix for entity_source_id collision: The current sourceIdToEntityId map is keyed by bare sourceId without sourceSystem qualification. In multi-system tenants, IDs from different systems can collide (e.g., both Entra and ServiceNow have records with ID "abc-123"). The map key must be {sourceSystem}:{sourceId} to prevent mis-linking.

// Current (collision-prone):
sourceIdToEntityId.set(node.sourceId, entityId);

// Fixed:
sourceIdToEntityId.set(`${node.sourceSystem}:${node.sourceId}`, entityId);

// Lookup:
const key = `${node.sourceSystem}:${sourceId}`;
const resolved = sourceIdToEntityId.get(key) ?? nodeIdToEntityId.get(sourceId);

Result:

• Azure sign-in evidence → entity_id = Service Principal entity
• ServiceNow flow execution → entity_id = Flow Designer flow entity
• ServiceNow job execution → entity_id = Scheduled Job entity

3.3 Evidence Node Schema (ServiceNow)

Following the _add_sign_in_node() pattern:

Flow execution evidence:

{
  "nodeId": "evidence-flow-exec-{sys_flow_context.sys_id}",
  "nodeType": "execution_evidence",
  "sourceSystem": "servicenow",
  "sourceId": "{sys_flow_context.sys_id}",
  "displayName": "Flow Execution: {flow_name} ({state})",
  "status": "active",
  "createdAt": "{started}",
  "properties": {
    "source_table": "sys_flow_context",
    "source_timestamp": "{started}",
    "evidence_type": "flow_execution",
    "action": "flow_run:{flow_name}",
    "target_resource": "{flow_name}",
    "outcome": "success | failure | unknown",
    "ended": "{ended}",
    "state": "{state}",
    "run_as": "{run_as}"
  }
}

Scheduled job execution evidence:

{
  "nodeId": "evidence-job-exec-{sys_trigger.sys_id}",
  "nodeType": "execution_evidence",
  "sourceSystem": "servicenow",
  "sourceId": "{sys_trigger.sys_id}",
  "displayName": "Job Execution: {job_name}",
  "status": "active",
  "createdAt": "{last_action}",
  "properties": {
    "source_table": "sys_trigger",
    "source_timestamp": "{last_action}",
    "evidence_type": "scheduled_job",
    "action": "scheduled_execution:{job_name}",
    "target_resource": "{job_name}",
    "outcome": "success | failure | unknown",
    "state": "{state}",
    "next_action": "{next_action}"
  }
}

The evidence_type values "flow_execution" and "scheduled_job" already exist in the platform's EVIDENCE_TYPES enum.

Mutable SN records: ServiceNow sys_flow_context records can evolve (e.g., state transitions from "In Progress" → "Completed" → "Error"). The platform must handle re-ingestion of previously-seen evidence records. See §3.5 for the upsert contract.

3.4 API Endpoint

GET /api/v1/entities/:id/execution-evidence

Follows existing entity sub-resource pattern (like timeline, versions, blast-radius).

Query parameters:

• evidence_type — filter by sign_in | flow_execution | scheduled_job (optional)
• outcome — filter by success | failure | unknown (optional)
• since / until — ISO timestamp bounds on source_timestamp (optional)
• limit — max records, default 50, max 200 (optional)
• cursor — cursor-based pagination token (optional)

Cursor contract: Evidence is sorted by source_timestamp DESC, _id DESC. The cursor encodes a compound key { ts: "<ISO timestamp>", id: "<ObjectId hex string>" } (base64url JSON, following the existing encodeCursor/decodeCursor pattern in pagination.ts).

_id type contract: ExecutionEvidenceDoc._id is currently optional string (types.ts:13). Evidence docs do NOT set _id explicitly — MongoDB auto-assigns ObjectId. The cursor must decode id back to ObjectId before querying:

import { ObjectId } from "mongodb";

// Decode cursor and cast _id to ObjectId for query:
const cursorId = new ObjectId(decoded.id);

// Compound cursor query for source_timestamp DESC, _id DESC:
filter.$or = [
  { source_timestamp: { $lt: cursorTs } },
  { source_timestamp: cursorTs, _id: { $lt: cursorId } }
];

This prevents duplicate/missing rows when multiple evidence records share the same source_timestamp (common for batch-ingested SN records). The ObjectId sort order is monotonically increasing by insertion time, so _id DESC is a stable tiebreaker.

total_count semantics: meta.total_count reflects the filtered count — the number of records matching ALL applied filters (evidence_type, outcome, since, until) for this entity. This is consistent: if you filter by evidence_type=flow_execution, the count tells you how many flow execution records exist, not the total across all types. When no filters are applied, it equals the unfiltered entity total.

Response:

{
  "data": [
    {
      "source_system": "servicenow",
      "source_table": "sys_flow_context",
      "source_record_id": "abc-123",
      "source_timestamp": "2026-02-14T14:00:00Z",
      "evidence_type": "flow_execution",
      "action": "flow_run:HR Onboarding Sync",
      "target_resource": "HR Onboarding Sync",
      "outcome": "success",
      "payload_hash": "sha256:...",
      "sync_id": "sync-42"
    }
  ],
  "meta": {
    "total_count": 147,
    "entity_id": "abc123",
    "entity_name": "HR Onboarding Sync",
    "filters_applied": { "evidence_type": "flow_execution" }
  },
  "cursor": { "next": "...", "has_more": true }
}

3.5 Storage Changes

Keep existing getExecutionEvidence(tenantId, entityId, limit) for evaluator use. Add:

interface ExecutionEvidenceQuery {
  entityId: string;
  evidenceType?: string;
  outcome?: string;
  since?: Date;
  until?: Date;
  limit?: number;
  afterTs?: Date;    // compound cursor: timestamp component
  afterId?: string;  // compound cursor: _id component
}

queryExecutionEvidence(tenantId: string, query: ExecutionEvidenceQuery): Promise<ExecutionEvidenceDoc[]>;
countExecutionEvidence(tenantId: string, query: Omit<ExecutionEvidenceQuery, 'limit' | 'afterTs' | 'afterId'>): Promise<number>;

Count accepts the same filters as query (minus pagination params). This ensures total_count is always consistent with the filtered result set.

Evidence persistence: upsert, not insert-only. The current insertExecutionEvidence() uses insertMany with duplicate-key error swallowing. This locks in stale state when SN records transition (e.g., "In Progress" → "Completed"). Change to bulkWrite with updateOne + upsert: true per record, keyed on the existing unique index { tenant_id, source_system, source_record_id }.

Uniqueness assumption: source_record_id must be unique within a (tenant_id, source_system) tuple. This holds today because Entra uses sign-in record IDs (UUIDs) and ServiceNow uses sys_id values (UUIDs), both globally unique within their respective systems. If a future connector reuses IDs across tables within the same source_system, the unique index would need source_table added. Hard assumption for now; document in connector interface contract.

async upsertExecutionEvidence(evidence: ExecutionEvidenceDoc[]): Promise<{ upserted: number; updated: number }> {
  const ops = evidence.map(doc => ({
    updateOne: {
      filter: {
        tenant_id: doc.tenant_id,
        source_system: doc.source_system,
        source_record_id: doc.source_record_id,
      },
      update: { $set: doc },
      upsert: true,
    }
  }));
  const result = await this.c.executionEvidence.bulkWrite(ops, { ordered: false });
  return { upserted: result.upsertedCount, updated: result.modifiedCount };
}

The existing insertExecutionEvidence method is kept as a deprecated alias during migration. The sync-ingestion.ts handler calls the new upsertExecutionEvidence instead.

3.6 UI Integration

New "Evidence" tab on AutomationDetailPage (6th tab alongside Overview, Graph, Effects, Findings, Timeline):

1. Summary strip: total count, breakdown by evidence_type, latest timestamp
2. Evidence table: Timestamp, Type (badge), Action, Target Resource, Outcome (green/red/gray), Source
3. Sorted by source_timestamp descending, paginated at 50/page

E1 automation evidence: RUNS_AS cross-lookup. In E1, ServiceNow evidence nodes don't exist yet (that's E2). Existing Entra sign-in evidence links to identities, not automations. To avoid an empty Evidence tab on AutomationDetailPage in E1, the UI should:

1. Query evidence for the automation's own entity_id (will populate once E2 lands)
2. Look up the automation's RUNS_AS relationship targets (identity entities)
3. Query evidence for those identity IDs
4. Display both sections: "Direct evidence" (empty until E2) and "Related identity evidence" (sign-ins for the SP this automation runs as)

This provides immediate value in E1: a BR that RUNS_AS an Azure SP shows the SP's sign-in evidence on the automation's Evidence tab. In E3, the evaluator gets the same cross-lookup server-side.

New hook: ui/src/hooks/use-execution-evidence.ts

---

4. Implementation Phases

Phase E1: Expose Existing Entra Evidence (Platform + UI)

Scope: No connector changes. Uses data already in MongoDB. Adds EVIDENCES type for acceptance (not emission).

#	Task	File
1	Add EVIDENCES to all 4 edge type surfaces	relationship-types.ts, types.ts, ingest.ts, GraphFilterSidebar.tsx
2	Add queryExecutionEvidence + countExecutionEvidence to StorageAdapter	src/storage/storage-adapter.ts, src/storage/mongo/adapter.ts
3	Change evidence persistence from insert-only to upsert	src/storage/mongo/adapter.ts
4	Create evidence API route with compound cursor pagination	src/api/routes/evidence.ts (new)
5	Register route in app	src/api/app.ts
6	Update nodeToExecutionEvidence to accept EVIDENCES edges	src/ingestion/graph-transformer.ts
7	Fix sourceIdToEntityId map to qualify keys by sourceSystem	src/ingestion/graph-transformer.ts
7b	Skip evidence nodes in nodeIdToEntityId map to prevent dangling relationships	src/ingestion/graph-transformer.ts
8	(Conditional) Add evidence retention TTL index if approved by product/legal	src/storage/mongo/schema.ts — only if retention policy is decided before E1 ships
9	Create use-execution-evidence.ts hook	ui/src/hooks/use-execution-evidence.ts (new)
10	Add Evidence tab to AutomationDetailPage with RUNS_AS cross-lookup	ui/src/pages/AutomationDetailPage.tsx
11	Write tests (unit for route, integration for storage, upsert behavior)	test/api/evidence.test.ts (new)

Phase E2: ServiceNow Evidence as First-Class Nodes (Connector)

Scope: Connector changes only. Platform already accepts evidence nodes and EVIDENCES edges from E1.

#	Task	File
1	Return full records from discover_flow_executions()	servicenow_client.py
2	Return full records from discover_job_executions()	servicenow_client.py
3	Update DiscoveredEntities.execution_data type	correlator.py
4	Add _add_flow_execution_node() method	transformer.py
5	Add _add_job_execution_node() method	transformer.py
6	Call new methods from _enrich_automation_properties()	transformer.py
7	Remove redundant execution_evidence_refs property	transformer.py
8	Update _build_evidence_completeness()	transformer.py
9	Emit EVIDENCES edges (evidence → automation) for SN evidence nodes	transformer.py
10	Migrate sign-in evidence edges from EXECUTES_ON to EVIDENCES	transformer.py
11	Keep summary properties (execution_count_30d, etc.) as cache	No change needed

Phase E3: Evaluator RUNS_AS Evidence Lookup (Platform)

Scope: Evaluator enhancement. When evaluating an automation for dormancy, also check evidence linked to the identity it RUNS_AS. This makes the server-side evaluator consistent with the UI cross-lookup added in E1.

#	Task	File
1	Query evidence for RUNS_AS identity targets	src/evaluator/rules/dormant-authority.ts
2	Same for privilege justification gap	src/evaluator/rules/privilege-justification-gap.ts

Phase E4: Evidence Completeness Indicators

Scope: Update evidence packs to reflect which sources have evidence.

#	Task	File
1	Link execution_evidence completeness to actual evidence records	src/evidence/sections.ts

---

5. Volume and Performance

Connector-side capping

• Emit at most 50 evidence records per automation per sync (most recent within 30-day window)
• Current connector fetches max 10 records; expand to 50 for temporal analysis
• API rate: ~140 calls for typical deployment, well within SN's 10 req/sec limit

Platform-side storage

• No artificial storage cap. Evidence data is governance/compliance-relevant.
• Retention policy: requires product/legal review before implementing TTL. Evidence retention may have regulatory requirements (SOC 2, ISO 27001). Default: no TTL. If a TTL is approved, implement in src/storage/mongo/schema.ts alongside the existing TTL pattern used for events (schema.ts:225, TWO_YEARS_SECONDS).
• Cardinality estimate: 50 records/automation * 100 automations * 2 syncs/day = 10,000 records/day (mostly deduped by upsert). Steady-state growth is bounded by actual new executions, not sync frequency.
• Existing indexes are sufficient:
  • (tenant_id, entity_id, source_timestamp) — primary query pattern
  • (tenant_id, source_system, source_record_id) — duplicate prevention / upsert key
  • (tenant_id, evidence_type, source_timestamp) — filtered queries

API pagination

• Default page size: 50, max: 200
• Compound cursor: (source_timestamp, _id) for stable pagination
• total_count in response meta reflects filtered count

---

6. Backward Compatibility

Summary properties retained

Keep execution_count_30d, last_observed_execution_timestamp on automation entities as a cache. The evidence collection is the source of truth.

Rationale:

• Properties are consumed by the connector's security_relevance classification
• Properties drive internal_inventory filtering
• Properties are displayed in the Overview tab
• Computing from evidence collection at query time would add unnecessary overhead

Edge migration

Accept both EXECUTES_ON (legacy) and EVIDENCES (canonical) during one release cycle. Deprecate EXECUTES_ON for evidence linkage after migration.

Dangling relationship prevention

Evidence nodes are not persisted as entities (graph-transformer.ts:116-121 skips entity creation for execution_evidence nodes). However, the nodeIdToEntityId map (line 106-108) includes evidence nodes, so buildRelationships() can create relationships whose target_id points to an evidence "entity" that does not exist in the entities collection.

Fix in transformGraph(): When building the nodeIdToEntityId map, either:

• Skip execution_evidence nodes (they don't need entity IDs), or
• Filter buildRelationships() output to exclude relationships targeting evidence nodes

Recommendation: skip evidence nodes in the map. Evidence entity ID resolution happens separately in nodeToExecutionEvidence() via edge traversal. This prevents any entity from storing a relationship pointing to a non-existent entity.

Evidence persistence migration

Change from insert-only (insertMany + swallow duplicates) to upsert (bulkWrite with updateOne + upsert: true). Keyed on the existing unique index { tenant_id, source_system, source_record_id }. Existing insertExecutionEvidence kept as deprecated alias.

---

7. What Not To Do

• Do not build a separate "execution evidence explorer" page. Evidence should be contextual (visible from the automation/identity it belongs to).
• Do not retroactively create evidence records from flattened properties. The next sync produces real data.
• Do not try to create execution evidence for Business Rules/Script Includes. ServiceNow has no logs for these. The execution_count_30d = 0 approach is correct.
• Do not implement circumstantial evidence via sys_audit (Tier 3). It adds complexity and the evidence is not deterministic, violating the platform's design constraint.
• Do not implement TTL on evidence data without product/legal sign-off on retention requirements.

---

8. Risks and Mitigations

Risk	Mitigation
Volume explosion from high-frequency automations	Connector-side cap (50/automation/sync) + upsert dedup on re-sync
Breaking existing evaluator rules	getExecutionEvidence interface unchanged; queryExecutionEvidence is additive
Connector migration complexity	Accept both EXECUTES_ON and EVIDENCES during migration window
Entity_id ambiguity (evidence relates to both identity and automation)	Each evidence record has exactly one entity_id via EVIDENCES edge. Cross-lookup via RUNS_AS edges handled in UI (E1) and evaluator (E3)
Business Rules have no execution evidence	Documented as fundamental SN limitation. Proxy via sign-in correlation (E3)
Stale evidence state from SN record transitions	Upsert persistence (not insert-only) ensures latest state wins
sourceIdToEntityId collisions in multi-system tenants	Qualify map keys by {sourceSystem}:{sourceId}
Evidence data retention requirements	No TTL applied without product/legal review
Cursor instability with equal timestamps	Compound cursor (source_timestamp, _id) ensures stable pagination
E1 Evidence tab empty for automations (no SN evidence yet)	RUNS_AS cross-lookup shows related identity sign-in evidence immediately
EVIDENCES edge type rejected at ingest	All 4 edge type surfaces updated in E1 before connector emits in E2
Dangling entity relationships to evidence nodes	Skip evidence nodes in nodeIdToEntityId map; evidence resolves via separate path
Cursor _id type mismatch (string vs ObjectId)	Evidence docs use auto-assigned ObjectId; cursor decodes hex string back to ObjectId
source_record_id collision across tables in same source_system	Documented as hard assumption; both Entra and SN use globally-unique sys_ids/UUIDs

---

9. Open Decisions

These require product input before implementation:

1. Evidence retention policy. Should evidence records have a TTL? If so, what duration? This has compliance implications (SOC 2, ISO 27001). Default recommendation: no TTL until reviewed.

2. EVIDENCES in graph UX. Should EVIDENCES edges be visible and filterable in the Graph Explorer, or purely an ingestion-time linkage? Recommendation: visible (already in GraphFilterSidebar edge type list) but not rendered by default (add to a "show evidence edges" toggle to avoid visual clutter).

3. BR/SI proxy evidence in UI. For automations with no native execution logs (Business Rules, Script Includes), should the Evidence tab show proxied RUNS_AS identity evidence with a clear "via [SP name]" label? Recommendation: yes, implemented in E1 via the RUNS_AS cross-lookup.

---

10. Verification Checklist

Phase E1

• EVIDENCES added to all 4 type surfaces; connector can POST a graph with EVIDENCES edges
• GET /api/v1/entities/:id/execution-evidence returns Azure sign-in evidence for identity entities
• Compound cursor pagination stable across equal timestamps (ObjectId decoded correctly)
• total_count reflects filtered count (not unfiltered total)
• Evidence upsert updates stale records (test: insert then re-insert with changed outcome)
• Evidence tab on AutomationDetailPage renders evidence table
• Evidence tab shows RUNS_AS cross-lookup for automations with linked identities
• sourceIdToEntityId qualified by sourceSystem (no cross-system collisions)
• Evidence nodes excluded from nodeIdToEntityId map (no dangling entity relationships)
• Evaluator rules continue to pass (no regression)

Phase E2

• Fresh connector scan produces execution_evidence nodes for flows and jobs
• EVIDENCES edges link evidence to automation entities
• execution_evidence_refs property removed
• Evidence tab shows ServiceNow flow execution records
• dormant_authority rule correctly evaluates automations with evidence
• graph.json node count increases (evidence nodes added)
• Summary properties (execution_count_30d, etc.) still populated
• Re-synced SN evidence records update (not duplicate) via upsert

Phase E3

• dormant_authority evaluating a BR also checks sign-in evidence for its RUNS_AS SP
• privilege_justification_gap same behavior
• Finding count may change (some dormant automations now have evidence via RUNS_AS)

Phase E4

• Evidence completeness section accurately reflects available sources
• Evidence pack links to actual evidence endpoint