Skip to main content

Temporal Diff Capability and Native Audit Feasibility Analysis

Date: 2026-02-09
Audience: Platform architecture, ingestion, and connector teams

Executive Summary​

SecurityV0 should not use one temporal strategy for every connector.

Current code in sv0-platform still computes most events via internal snapshot diff (platform_diff), while architecture docs already target an audit-log-first model. The right direction is a hybrid connector-level strategy:

  1. native_audit for systems with strong, queryable change history.
  2. audit_plus_reconcile where audit is good but incomplete or retention-limited.
  3. snapshot_diff_only where reliable audit trails are unavailable.

This minimizes duplicate historical state storage while preserving deterministic, evidence-grade timelines.

1) Current Platform Capability (Reality Check)​

Implemented in code now​

  • Sync ingestion sets sync_mode: "full" and runs transform -> diff -> upsert (sv0-platform/src/workers/handlers/sync-ingestion.ts).
  • Event generation is currently diff-derived with audit_source.source_api: "platform_diff" (sv0-platform/src/ingestion/diff-engine.ts).
  • Entity versions are inserted for changed entities on each sync run (sv0-platform/src/workers/handlers/sync-ingestion.ts).

Documented target state​

  • Architecture docs already define sync_mode: "audit_log", sync cursors, and source-audit provenance (sv0-documentation/docs/architecture/03-database.md).
  • Connector contract already includes supportsAuditLog and extractAuditLogs() (sv0-documentation/docs/architecture/05-connectors.md).

Gap​

The model is designed for audit-log-driven temporal tracking, but runtime implementation is still mostly snapshot diff + generated events.

2) When Native Audit Is Feasible​

A source system is a strong candidate to reduce internal historical replication when it satisfies most of these:

  1. Stable event identity: immutable event ID for dedupe.
  2. Incremental retrieval: cursor/time-window API (since, nextLink, token, or ordered key).
  3. Attribution completeness: actor + operation + target entity references.
  4. Temporal trust: source event timestamp (not only fetch time).
  5. Coverage fit: emits events for the entity/relationship changes SecurityV0 must reason about.
  6. Retention safety: retention window longer than worst-case sync gap, or export/streaming path exists.
  7. Operational access: available with practical licensing and permissions.

If any of 3-6 fail, use audit_plus_reconcile (not pure native audit).

3) Source System Feasibility Matrix (Checked Feb 9, 2026)​

SystemNative Audit/History CapabilityFeasibility for Replacing Internal HistoryKey Caveats
Microsoft Entra IDMicrosoft Graph directoryAudits provides actor, activity, timestamp, target resources.audit_plus_reconcileCoverage and retention constraints vary by log type/licensing; keep periodic state reconciliation for missing event classes.
ServiceNowsys_audit stores field-level changes and is intended for long-term audit; sys_audit_role available for role history when enabled.native_audit for audited tables, otherwise audit_plus_reconcilesys_history_set/sys_history_line are not durable audit sources (table cleaner purges quickly). Ensure auditing is enabled for required tables/fields.
OktaSystem Log API supports polling, filtering, and actor/target event model.audit_plus_reconcileOkta event retention is limited (commonly 90 days) unless exported; treat export pipeline as required for long investigations.
GitHub Enterprise CloudEnterprise audit log supports API retrieval and streaming with at-least-once delivery guarantees.audit_plus_reconcileRetention and event scope are plan-dependent; stream to durable storage for longer historical windows.
AWS (IAM + CloudTrail)CloudTrail captures management API events with principal + resource context.audit_plus_reconcileEvent History is short-window unless trails/lake are configured; some semantic diffs require interpreting request/response payloads.
Google Cloud (IAM + Cloud Audit Logs)Cloud Audit Logs provide Admin Activity logs with principal, method, and resource names.audit_plus_reconcileStrong for admin changes, but normalized event mapping is still needed; cross-product consistency varies.
GitLabAudit Events API for user/project/group changes plus streaming destinations.audit_plus_reconcileCoverage depends on tier and scope; validate required event families before claiming full historical parity.

4.1 Add per-connector temporal strategy​

Add to connector config/capabilities:

type TemporalStrategy =
  | "native_audit"
  | "audit_plus_reconcile"
  | "snapshot_diff_only";

4.2 Keep one canonical events model, but mark origin​

For each event, keep provenance and add explicit origin semantics:

  • event_origin: "source_audit" | "platform_diff" | "reconcile_backfill"
  • existing audit_source.source_api and source_record_id

This keeps query APIs stable while allowing mixed strategies per source.

4.3 Replace per-sync historical replication with baselines + targeted replay​

  • Keep entities as current state.
  • Keep events as canonical change log.
  • Keep entity_versions as cache/sparse snapshots (materialized on demand, significant changes, or baseline boundaries).
  • Use scheduled baselines primarily for retention-gap protection and replay performance.

4.4 Require evidence-completeness gating​

Before enabling native_audit for a connector, require a machine-readable completeness report:

  • covered event families
  • unsupported entities/relationships
  • retention window
  • license/access dependencies

If incomplete, force audit_plus_reconcile.

5) Practical Feasibility Rules​

Use these rules at connector onboarding:

  1. If source has durable audit with target IDs + actor + cursor -> start with audit_plus_reconcile.
  2. Promote to native_audit only after 2-4 weeks of parity checks against periodic full snapshots.
  3. If retention < baseline cadence safety window -> require export pipeline or increase baseline frequency.
  4. If event coverage for key findings (ownership, role/scope drift, credential lifecycle) is partial -> stay in audit_plus_reconcile.
  5. If no reliable audit endpoint exists -> snapshot_diff_only.

6) Implementation Plan (Incremental)​

  1. Schema/contract: add TemporalStrategy and event_origin; keep backward compatibility.
  2. Runtime routing: branch ingestion path by strategy (audit path vs snapshot diff path).
  3. Pilot: Entra + ServiceNow in audit_plus_reconcile mode first.
  4. Parity checks: run periodic full snapshots and compare with audit-derived state deltas.
  5. Promotion policy: allow native_audit only after parity SLO is met.
  6. UI/API transparency: expose strategy + evidence completeness on sync status endpoints.

7) Main Decisions Still Needed​

  1. What parity threshold promotes a connector from audit_plus_reconcile to native_audit?
  2. What baseline cadence is mandatory per strategy and connector retention profile?
  3. Which finding types are blocked when audit completeness is below threshold?
  4. What is the default event retention policy in-platform versus external archive?

Sources​

SecurityV0 internal references​

  • sv0-platform/src/workers/handlers/sync-ingestion.ts
  • sv0-platform/src/ingestion/diff-engine.ts
  • sv0-documentation/docs/architecture/03-database.md
  • sv0-documentation/docs/architecture/05-connectors.md
  • sv0-documentation/docs/architecture/08-reference-impl-entra-servicenow.md

Vendor documentation​

Next Action​

Status: adopted — shipped Decision: Hybrid diff approach adopted. last_seen_at / first_seen_at fields implemented on authority paths and execution chains in sv0-platform. last_activity_at on entities (from execution logs).

Gap: 01-data-model.md documents last_activity_at on entities but does not explicitly document first_seen_at/last_seen_at on authority paths. These fields are covered in 10-access-paths.md. Minor — no blocking issue.

No further action required. Research complete, findings reflected in platform code.