Skip to main content

SecOps Analyst Review: W1 Demo (demo-w1 tenant)

Verdict: NEEDS WORK​

The product is close to daily-usable. The authority path model, evidence chain, and remediation service are genuinely strong -- better than what I get from Wiz for non-cloud-native findings. But several operational gaps would prevent me from making this my primary triage tool today. The biggest: I cannot write a ticket from the UI (button is disabled), I cannot tell what changed since yesterday, and the execution evidence records lack the target resource and action detail I need to validate findings independently.

First Impression (60-second test)​

What I understood immediately:

  • The Overview page gives me a clear picture: 29 active authority paths, 769 executions in 30 days, 19 with invalid ownership. The 838% execution delta screams "something new started running." Good.
  • 6 risk clusters, ranked by severity. "Unowned Sensitive Access" (critical, 13 paths, 681 executions) is obviously where I start.
  • The cluster labels are functional, not attribute-based -- "Unowned Sensitive Access with LLM" tells me what happened without decoding.

What confused me:

  • The posture summary has delta.executions_delta_pct: 838 and paths_delta_pct: 81 but no absolute prior values. I can calculate backwards, but I shouldn't have to.
  • No timestamp on the delta -- 838% compared to what period? Prior 30 days? Prior scan? This matters for my CISO report.
  • The findings endpoint pagination is inconsistent: ?severity=critical returns 51 findings, but the default endpoint (no severity filter) returns only 2 dormant findings. The meta.total_count with severity=critical says 51, but the default meta.total_count says 2. This is confusing -- the severity filter appears to widen the result set rather than narrow it.

What I'd click first:

  • "Unowned Sensitive Access" cluster (critical, 13 paths, 681 executions, no owners). This is the obvious priority.

Actionability Assessment​

  • Finding clarity: CLEAR

    • The deterministic_explanation field on findings is excellent. Example: "Authority path from workload 'Foundry Agent701' to 'c5c0298db90d18c6f607cd1d' gained 'foundry_ai_executor' (0 -> 1 roles) since baseline. Path reaches confidential data domain 'identity'. Exercised 7 times in last 30 days." This tells me the what, the why, and the scale in one sentence.
    • Severity is always backed by a reason (sensitivity + governance condition + execution state).

  • Remediation specificity: MIXED -- prescriptive for scope_drift, generic for others

    • Strong: For scope_drift, the remediation says Remove role 'foundry_ai_executor' -- this is a concrete action I can put in a ticket. The evidence_refs include new_roles, baseline_role_count, and current_role_count. I know exactly what changed.
    • Strong: For orphaned_ownership, it says Assign a valid owner to this workload with signal 1 owner(s) inactive: James Torres (deleted). Tells me who left and what to do.
    • Weak: For llm_egress, the remediation is Restrict LLM endpoint access -- but restrict how? In Entra ID? In the Foundry agent configuration? By revoking the foundry_ai_executor role, or by adding a Conditional Access policy? The applies_to field says Egress: Foundry Agent701 which names the workload but not the specific control plane.

  • Weak: For reachable_sensitive_domain, Reduce execution paths to least-privilege scope is advice, not a remediation step. I need "Remove role X from identity Y in system Z."

  • Missing: No remediation action tells me which system to go to. The source_system is "entra_id" on the path, but the remediation doesn't say "Go to Entra ID > Enterprise Applications > svc-foundry-agent701 > Permissions and remove foundry_ai_executor."

  • Ticket-ready: PARTIAL -- missing system-of-record links and step-by-step

    • I can copy the deterministic_explanation into a ticket body -- it reads well.
    • I have entity names, role names, data domains, execution counts.
    • Missing: No deep links to Entra ID, ServiceNow, or Foundry admin consoles. No source_id for the service principal in the finding itself (have to go to entity detail). No Entra object ID for the role assignment.
    • Missing: "Create Ticket" button exists in the UI but is disabled (cursor-not-allowed, title="Coming soon"). This is a critical workflow gap.

  • Engineer handoff quality: NEEDS CONTEXT

    • An engineer would get "remove role 'foundry_ai_executor'" but wouldn't know: which Entra app registration? What's the object ID? Is this a custom role or built-in? What will break if I remove it?
    • The blast radius endpoint helps (/entities/{id}/blast-radius), but it shows what the workload can reach, not what removing this specific role would affect.

Validation Workflow​

  • Evidence chain: PARTIAL

    • Identity -> Permissions: YES. The authority path shows workload -> identity (via RUNS_AS) -> destination (via role). Roles are named. I can trace Foundry Agent701 -> svc-foundry-agent701 -> Azure OpenAI Endpoint via foundry_ai_executor.
    • Permissions -> Execution: PARTIAL. The path has execution_30d: 7 and last_execution_at, but the execution evidence records (/entities/{id}/execution-evidence) show action: "read" with target_resource: "" (empty). I can't validate that the 7 executions actually hit Azure OpenAI -- the target resource is not populated. This is a significant gap.
    • Execution -> Data: NOT AVAILABLE. I know the path reaches "identity" domain with "confidential" sensitivity, but there's no data access log. This is acceptable for W1 (read-only connector, no DLP integration) but should be called out.
    • Ownership timeline: YES. Evidence pack includes James Torres (deleted) as former owner. Clear and verifiable.

  • Manual verification possible: PARTIAL

    • I could go to Entra ID and look up the service principal. But the entity's source_id isn't surfaced in the findings -- I'd need to call /entities/{id} separately to get it. The evidence pack does include entity_id and source_system.
    • The evidence_completeness object is a nice touch: it tells me exactly which evidence dimensions are available vs. unavailable. Example: role_history: "available", execution_evidence: "available", approval_records: "unavailable_not_applicable".

  • False positive distinguishable: YES

    • The scope_drift finding includes baseline vs. current role counts and the specific new roles. If I verify in Entra ID that the role was always there, I know the baseline was wrong, not the current state. The temporal markers and baseline_version_date make this traceable.

Remediation Safety​

  • Fix specificity: MIXED

    • Scope drift: CONCRETE. "Remove role 'foundry_ai_executor'" with evidence of 0->1 roles since baseline.
    • Orphaned ownership: CONCRETE. "Assign a valid owner" with "James Torres (deleted)" context.
    • LLM egress: VAGUE. "Restrict LLM endpoint access" -- how? Where?
    • Sensitive access: VAGUE. "Reduce execution paths to least-privilege scope" -- which paths? Which roles to remove?

  • Blast radius of fix explained: PARTIAL

    • The blast radius API shows Foundry Agent701 reaches 5 resources across identity, it_operations, and engineering domains. This tells me the current blast radius.
    • Missing: No "blast radius of the fix" -- if I remove foundry_ai_executor, will the Foundry Agent stop working? Will incident creation fail? No dependency analysis for the remediation action itself.
    • The remediation service's impact_score is actually a category priority rank (0-40), not a business impact score. This is confusing in the UI where it's displayed as a bar chart labeled "impact."

  • Would execute in production: WITH CAVEATS

    • For orphaned_ownership: yes, I'd assign an owner with confidence.
    • For scope_drift with named roles: yes, I'd remove the role after verifying in the source system.
    • For LLM egress: no, the remediation is too vague to act on without additional investigation.

Closure & Drift​

  • Post-remediation verification: AVAILABLE (structurally)

    • Finding intervals track effective_from and resolved_at with resolution_reason. If I fix the scope drift and re-scan, the finding should transition to resolved with a clear timestamp.
    • The entity timeline shows lifecycle events (created, circuit-breaker blocks). This would capture remediation-driven changes.

  • Drift detection: USEFUL

    • The product detects scope_drift, reachability_drift, and ownership_drift as first-class finding types. Each has evidence_refs with baseline vs. current comparisons.
    • Ownership drift shows specific removed/disabled owner names -- not just "ownership changed."
    • Reachability drift shows new destination names and sensitive domains affected.
    • Not yet testable: I can't verify drift detection in action because the demo data only has one temporal version. But the architecture (baseline comparison, interval tracking) is sound.

  • Drift noise potential: LOW-MEDIUM

    • Findings are deduplicated per path, and intervals collapse overlapping periods. This should prevent alert fatigue.
    • Concern: In the current demo, every finding has the same detected_at (2026-03-10T23:28:51.883Z) because they were all generated in the same evaluation pass. In production with continuous scanning, I'd want to verify that re-evaluation doesn't re-fire existing findings.

  • Closure demonstration: CAN PROVE

    • Finding status transitions (active -> remediated) with resolved_at timestamps would give me auditable closure proof. The evidence pack with integrity_hash (SHA256) makes this defensible.

Triage Workflow​

  • Steps to find critical findings: 1 API call (or 1 click to Overview page)

    • The Overview page surfaces risk clusters ranked by severity and execution count. One click into the "Unowned Sensitive Access" cluster gives me 13 paths.

  • Clicks from dashboard to actionable context: 3

    • Overview -> Cluster Detail (Section A-D gives narrative + governance + remediation) -> Authority Path Detail (full evidence). This is good.

  • Filter/sort to my area: PARTIAL

    • I can filter findings by severity, finding_type, status, entity_id.
    • Missing: No filter by team, business unit, or responsibility area. If I own "engineering" domain findings, I can't filter to just those. I'd have to scan all findings manually.
    • Missing: No saved filters or views. Every morning I start from scratch.

  • "What's new since yesterday": MISSING

    • The delta in posture summary shows period-over-period changes (new_paths: 0, removed_paths: 0), but all findings have the same detected_at from the initial evaluation.
    • There is no ?since=2026-03-14 query parameter on findings.
    • The authority paths endpoint accepts sort=-updated_at but there's no "changed since" filter.
    • This is a daily workflow blocker. I need to know: what's new, what changed, what was resolved since my last session.

Investigation Quality​

  • Authority path trace: COMPLETE

    • Foundry Agent701 (workload, microsoft_foundry) -> svc-foundry-agent701 (identity, entra_id) -> Azure OpenAI Endpoint (destination, microsoft_foundry) via foundry_ai_executor role.
    • The graph shows workload -> identity -> destination -> data_domain. Roles are displayed as pills with links to entity detail. Cross-system boundaries are clear (source_system on each node).

  • Blast radius clarity: CLEAR

    • Blast radius API returns 5 resources with sensitivity and domain breakdown. The UI renders this as a full list with sensitivity badges.
    • Breakdown by domain (identity: 1, it_operations: 2, engineering: 2) and sensitivity (confidential: 4, internal: 1) is immediately useful.

  • Temporal comparison: PARTIAL

    • Scope drift evidence includes baseline_role_count vs current_role_count and baseline_version_date. I can see what changed.
    • Entity timeline shows created/deleted events. But there are only 2 timeline events for Foundry Agent701 (created + deletion_blocked). No intermediate state changes.
    • Missing: No side-by-side "before/after" view. I have the data (baseline_role_count vs current_role_count) but the UI doesn't render it as a comparison.

  • Cross-reference navigation: LINKED

    • Authority path detail page: clicking a node opens an entity details drawer. Via roles are clickable pills that link to entity detail pages. Finding tiles link to finding detail.
    • Cluster detail page: each path row links to authority path detail. Remediation targets link to individual paths.
    • Gap: Cross-system paths for Foundry Agent701 returned auth_chains: [] (empty). Either there's no cross-system auth to show, or the connector didn't resolve it. The UI handles this gracefully (shows "no cross-system auth chains detected") but this leaves a gap in the identity->execution trace.

Reporting Capability​

  • CISO summary producible: PARTIAL

    • The Overview page gives me: 29 active paths, 769 executions, 19 with invalid ownership, 838% execution growth.
    • Risk clusters provide natural "top 5 risks" grouping with narrative sentences.
    • Missing: No export button works (disabled in UI). I'd have to screenshot or copy-paste.
    • Missing: No "changes this week" aggregate. I can't say "we had X critical, closed Y, Z remain" because there's no closure tracking in the demo data.

  • Trend views: MISSING

    • The delta fields (executions_delta_pct, paths_delta_pct) provide one comparison point.
    • No time-series data. No "findings over time" chart. No "paths trending up/down." This is important for CISO weekly reporting.

  • Export: MISSING

    • "Export Report" button on Overview: disabled.
    • "Export" button on Cluster Detail: disabled.
    • Evidence packs are exportable via API and have a markdown renderer (evidence/markdown.ts), which is good for audit. But the UI doesn't surface this.

Top Workflow Gaps​

1. "What changed since yesterday" -- MISSING (blocks daily triage)​

There is no way to see new/changed/resolved findings since my last session. Every morning I'd have to mentally diff the entire findings list. This is the #1 gap for daily operational use. Need: a ?changed_since= filter on findings, and a "New since last visit" section on the Overview page.

2. Remediation lacks system-specific instructions (Sergey's exact concern)​

Remediations like "Restrict LLM endpoint access" and "Reduce execution paths to least-privilege scope" fail the "what do I do next?" test. The analyst sees the finding, understands it, but hits a wall at the remediation step. Need: system-specific remediation templates that reference the actual admin console (e.g., "In Entra ID > Enterprise Applications > {service_principal_name} > Permissions, remove the '{role_name}' app role assignment").

3. Execution evidence is hollow -- target_resource and action detail are empty​

The execution evidence records have action: "read" and target_resource: "" for all entries. This means I can't validate which resource was actually accessed during execution. The path model says Foundry Agent701 can reach Azure OpenAI Endpoint, but the execution evidence doesn't confirm it hit that specific endpoint vs. another resource. This undermines the "observed authority" promise.

Additional Gaps​

4. "Create Ticket" button is disabled​

This is on both the Authority Path Detail and Cluster Detail pages. For a tool meant for daily analyst use, the inability to create a ticket from a finding is a significant workflow gap.

5. No team/domain filtering​

I can filter findings by severity and type, but not by business domain, team, or responsibility area. In a 2,000-person company, I'd own specific systems -- I need to filter to my scope.

6. Findings pagination seems broken for severity filter​

GET /findings?severity=critical returns 51 findings, but GET /findings (no filter) returns only 2. The default endpoint appears to filter to a different severity level rather than showing all findings.

7. impact_score is misleading​

The remediation impact_score is actually a category priority rank (0=egress resolving, 1=egress mitigating, ... 31=governance mitigating), not a business impact score. The UI renders this as a bar chart with 0-10 scale, which makes "Restrict LLM endpoint access" (score 0) look like it has zero impact, when it's actually the highest priority. Either rename the field or invert the visual.

What Works Well​

  1. Deterministic explanations are genuinely useful. Every finding has a plain-English sentence that I could paste into a ticket body. Example: "Authority path from workload 'ServiceNow HR Sync' to '2895d998776242fd5118124f' gained 'hr_data_manager' (0 -> 1 roles) since baseline. Path reaches confidential data domain 'hr'. Exercised 5 times in last 30 days." This is better than what Wiz gives me for identity findings.

  2. Cluster -> Path -> Finding -> Evidence navigation is coherent. Three clicks from "Unowned Sensitive Access" to the evidence pack for a specific scope drift finding. No dead ends, no 404s (once I used the correct route).

  3. Evidence packs are audit-ready. SHA256 integrity hash, sealed timestamp, identity summary, authority snapshot, ownership timeline, blast radius, and temporal context -- all in one payload. The markdown renderer means I could attach this to a change request.

  4. Governance condition checklist on cluster detail is exactly right. Section C shows "Orphaned identities" and "Scope drift" as failing conditions, with "Long-lived tokens" and "Lack of runtime telemetry" as not-assessed. This is honest and useful -- I know what's covered and what's a gap.

  5. Scope drift evidence is best-in-class. Baseline role count, current role count, specific new roles, grants added -- all in evidence_refs. The inline drift breakdowns in the UI (amber tiles showing "Roles: 0 baseline -> 1 current") are immediately comprehensible.

  6. Risk cluster narratives pass the 5-second test. "3 autonomous identities accessed sensitive systems 681 times in the last 30 days. None have an assigned owner." This is exactly what my CISO wants to hear.

  7. Standing Authority panel on path detail. Collapsed by default, shows other access granted to the same workload with zero observed executions. This is the observed-vs-potential separation the spec calls for, and it works well.

Bottom Line​

This is 70% of a daily-usable SecOps tool. The authority path model, evidence chain, and cluster narratives are strong enough to justify adoption. But three things block me from making it primary: (1) I can't tell what changed since yesterday, (2) remediation guidance doesn't tell me where to go and what to click in the source system, and (3) the execution evidence doesn't confirm which resource was actually accessed. Fix those three and a SOC team would adopt this. A partner could use it today for assessments (point-in-time snapshots) but not for managed services (ongoing operations).