Skip to main content

Visual Review Pipeline — reg-cli + Cloudflare Pages

Date: 2026-03-04 Status: Implemented — see As-Built Notes below Scope: sv0-platform (CI, scripts, Cloudflare infrastructure) Tracking: GitHub Issue #24

Problem​

When reviewing PRs that change the UI, there is no automated way to see what changed visually. The current process is manual: check out the branch, run the app, compare in a browser. Attempts to embed screenshots directly in PR comments failed — GitHub has no API for uploading images to comments on private repos. All URL formats tested (raw.githubusercontent.com, blob URLs with ?raw=true, release asset downloads, tokenized content API URLs) either return 404 or render as broken images because GitHub's Camo image proxy cannot authenticate against private repo resources.

What was tried (and failed)​

ApproachResult
raw.githubusercontent.com URLs404 for private repos
/blob/...?raw=true URLsGitHub renders as <a> link, not <img>
Release asset download URLsNot rendered as images by GitHub markdown
Tokenized content API URLsExpire within minutes
Orphan branch with screenshotsSame auth proxy limitation
GitHub Check Run output.imagesRequires publicly accessible image URLs

Root cause: GitHub's markdown renderer proxies all images through camo.githubusercontent.com, which cannot authenticate against private resources. The only working mechanism is the user-attachments CDN (web UI drag-and-drop), which has no public API (open discussion).

Solution​

Generate interactive HTML visual diff reports and host them on Cloudflare Pages with GitHub-authenticated access.

Architecture​

PR opened / updated
    │
    ▼
GitHub Actions workflow (ubuntu-latest)
    │
    ├─ 1. Build PR branch (docker compose)
    ├─ 2. Capture "after" screenshots (Playwright)
    ├─ 3. Build main branch (docker compose)
    ├─ 4. Capture "before" screenshots (Playwright)
    ├─ 5. reg-cli compares → interactive HTML report
    ├─ 6. wrangler pages deploy → Cloudflare Pages
    │     Preview URL: pr-<N>.sv0-visual-reviews.pages.dev
    └─ 7. Post sticky PR comment with report link
         (updates on re-runs, not duplicated)

PR closed
    │
    ▼
Cleanup workflow deletes Cloudflare Pages deployments

Why reg-cli​

reg-cli (145K weekly npm downloads, MIT, actively maintained) generates a single index.html report with 5 comparison modes:

  1. Diff — pixel-level diff overlay (changed pixels highlighted in magenta)
  2. Slide — horizontal slider revealing before/after
  3. 2up — side-by-side
  4. Blend — opacity crossfade
  5. Toggle — flip between before/after

The report embeds all JS/CSS inline. Images are referenced via relative paths, so the entire report/ directory is deployed as a static site.

Why Cloudflare Pages​

  • Already in use for sv0-documentation (sv0-docs.pages.dev)
  • Same secrets already configured (CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN)
  • --branch=pr-<N> flag produces stable preview URLs
  • Free tier: unlimited bandwidth, unlimited sites
  • Cloudflare Access (Zero Trust) protects with GitHub org authentication — same pattern as sv0-docs

Implementation Plan​

Phase 1: Screenshot capture script​

New file: sv0-platform/scripts/visual-screenshot.ts

Lightweight Playwright script that captures full-page screenshots. Reuses patterns from existing scripts/visual-qa.ts:

  • Tenant injection via localStorage init script
  • 1440x900 viewport, light color scheme
  • networkidle + 1.5s extra wait for async React rendering

Differences from visual-qa.ts:

  • No assertions, checks, or report generation — pure screenshot capture
  • Configurable output directory via QA_OUTPUT_DIR
  • Accepts QA_DETAIL_PATHS for entity detail pages (e.g., /authority-paths/abc123)
  • Outputs a manifest.json listing captured files
# Capture all pages
QA_BASE_URL=http://localhost:8080 npx tsx scripts/visual-screenshot.ts

# Capture specific pages only
QA_PAGES=dashboard,findings QA_OUTPUT_DIR=./screenshots/after \
  npx tsx scripts/visual-screenshot.ts

Phase 2: Diff report generator​

New file: sv0-platform/scripts/visual-diff-report.ts

Wrapper around reg-cli:

  1. Validates before/ and after/ directories exist with matching filenames
  2. Runs reg-cli ./after ./before ./diff -R ./report/index.html --matchingThreshold 0.01
  3. Copies image directories alongside the HTML (report/after/, report/before/, report/diff/)
  4. Outputs ready-to-deploy report/ directory
npx tsx scripts/visual-diff-report.ts \
  --before ./screenshots/before \
  --after ./screenshots/after \
  --output ./visual-report

New dependency: reg-cli@^0.18.14 (devDependency)

Phase 3: GitHub Actions workflow​

New file: sv0-platform/.github/workflows/visual-review.yml

name: Visual Review

on:
  pull_request:
    paths: ['ui/**', 'src/api/**']

jobs:
  visual-review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      deployments: write
      pull-requests: write

    services:
      mongodb:
        image: mongo:7
        ports: ['27017:27017']

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: npm

      - run: npm ci

      # Install Playwright browsers
      - run: npx playwright install --with-deps chromium

      # ── "After" screenshots (PR branch) ─────────────────
      - name: Build and start PR branch
        run: |
          npm run build
          npm start &
          npx wait-on http://localhost:3000/api/v1/health
          cd ui && npm ci && npm run build && npx serve -l 8080 dist &
          npx wait-on http://localhost:8080

      - name: Seed demo data
        run: npx tsx scripts/seed-demo.ts

      - name: Capture "after" screenshots
        run: |
          QA_BASE_URL=http://localhost:8080 \
          QA_OUTPUT_DIR=./screenshots/after \
          npx tsx scripts/visual-screenshot.ts

      - name: Stop PR branch
        run: kill $(lsof -ti:3000) $(lsof -ti:8080) || true

      # ── "Before" screenshots (main branch) ──────────────
      - uses: actions/checkout@v4
        with:
          ref: main
          path: main-branch

      - name: Build and start main branch
        working-directory: main-branch
        run: |
          npm ci && npm run build
          npm start &
          npx wait-on http://localhost:3000/api/v1/health
          cd ui && npm ci && npm run build && npx serve -l 8080 dist &
          npx wait-on http://localhost:8080

      - name: Capture "before" screenshots
        run: |
          QA_BASE_URL=http://localhost:8080 \
          QA_OUTPUT_DIR=./screenshots/before \
          npx tsx scripts/visual-screenshot.ts

      - name: Stop main branch
        run: kill $(lsof -ti:3000) $(lsof -ti:8080) || true

      # ── Generate diff report ────────────────────────────
      - name: Generate visual diff report
        run: |
          npx tsx scripts/visual-diff-report.ts \
            --before ./screenshots/before \
            --after ./screenshots/after \
            --output ./visual-report

      # ── Deploy to Cloudflare Pages ──────────────────────
      - name: Deploy report
        id: deploy
        uses: cloudflare/wrangler-action@v3
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          command: >-
            pages deploy ./visual-report
            --project-name=sv0-visual-reviews
            --branch=pr-${{ github.event.pull_request.number }}
            --commit-hash=${{ github.sha }}

      # ── Post PR comment ─────────────────────────────────
      - name: Comment PR with report link
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          header: visual-review
          message: |
            ## Visual Review Report

            Interactive comparison (slider, overlay, side-by-side):
            **${{ steps.deploy.outputs.pages-deployment-alias-url }}**

            Deployment: `${{ steps.deploy.outputs.deployment-url }}`
            Commit: `${{ github.sha }}`

Phase 4: Cleanup workflow​

New file: sv0-platform/.github/workflows/visual-review-cleanup.yml

name: Visual Review Cleanup

on:
  pull_request:
    types: [closed]

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - name: Delete preview deployments
        env:
          CF_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          CF_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
        run: |
          BRANCH="pr-${{ github.event.pull_request.number }}"
          DEPLOYMENTS=$(curl -s \
            -H "Authorization: Bearer $CF_API_TOKEN" \
            "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/pages/projects/sv0-visual-reviews/deployments" \
            | jq -r --arg branch "$BRANCH" \
              '.result[] | select(.deployment_trigger.metadata.branch == $branch) | .id')
          for id in $DEPLOYMENTS; do
            curl -s -X DELETE \
              -H "Authorization: Bearer $CF_API_TOKEN" \
              "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/pages/projects/sv0-visual-reviews/deployments/$id"
          done

Phase 5: Cloudflare Access setup (manual)​

Same pattern already in use for sv0-docs.pages.dev:

  1. Zero Trust dashboard → Identity providers → GitHub (already configured)
  2. Access → Applications → Add self-hosted application
    • Domain: *.sv0-visual-reviews.pages.dev
    • Session duration: 24h
  3. Policy: Allow → Include → GitHub Organization = SecurityV0

This ensures only SecurityV0 org members can view visual review reports.

Files Summary​

ActionFileDescription
Createscripts/visual-screenshot.tsPlaywright screenshot capture (reuses visual-qa.ts patterns)
Createscripts/visual-diff-report.tsreg-cli wrapper, assembles deployable report directory
Create.github/workflows/visual-review.ymlCI workflow: screenshot, compare, deploy, comment
Create.github/workflows/visual-review-cleanup.ymlCleanup Cloudflare Pages deployments on PR close
Modifypackage.jsonAdd reg-cli devDependency, visual:screenshot and visual:diff scripts

Effort Estimate​

PhaseEffort
Phase 1: Screenshot script~2h
Phase 2: Diff report generator~1h
Phase 3: CI workflow~2h
Phase 4: Cleanup workflow~30min
Phase 5: Cloudflare Access~15min (manual, one-time)
Total~6h

Open Questions​

  1. Self-hosted runner vs ubuntu-latest? Docker Compose build on ubuntu-latest adds ~3-5 min to CI. A self-hosted runner on the Mac mini would be faster (images cached) but requires runner setup. Start with ubuntu-latest, optimize later.

  2. Which pages to screenshot? Start with the 11 pages already defined in visual-qa.ts. Detail pages (e.g., /authority-paths/:id) need known entity IDs — seed script must produce deterministic IDs, or we skip detail pages initially.

  3. Baseline storage? The plan compares main vs PR branch on every run (no stored baselines). This means both branches must build successfully. Alternative: store baseline screenshots as GitHub Actions artifacts and only rebuild the PR branch. Tradeoff: faster CI but baselines can drift.

References​

As-Built Notes​

Implemented: 2026-03-05 – 2026-03-08 via sv0-platform PRs #25, #26, #27, #28, #44, #46

The implementation follows the plan with the following deviations:

Cloudflare Pages project name​

The plan did not specify a project name. The project was created as sv0-reviews (renamed in PR #27 from an earlier name). The resulting report URL pattern is:

https://sv0-reviews.pages.dev/reports/pr-<number>/

Manual dispatch workflow​

PR #28 added a workflow_dispatch trigger to the visual review workflow to allow triggering reviews manually without a PR push. This was not in the original plan.

Arg order fix (PR #26)​

The reg-cli wrapper script had an argument order issue that was fixed in PR #26. The detail capture step was also adjusted.

Workflow stability fixes (PRs #44, #46)​

Two rounds of workflow fixes were needed for reliability: timeout handling and error recovery for the Cloudflare Pages upload step.

What shipped as planned​

  • reg-cli for screenshot comparison and interactive HTML diff reports
  • Playwright for before/after screenshot capture
  • Cloudflare Pages for hosting per-PR visual diff reports
  • Sticky PR comment with direct link to the report
  • Cleanup workflow on PR close (visual-review-cleanup.yml)
  • Cloudflare Access with GitHub SSO (reports are private to org members)