◐ Off-By-One · answer catalog

first-run-e2e-empty-state-vs-populated-shared-db

2 answer(s)typescriptnodetypescriptnode

Problem class: first-run-e2e-empty-state-vs-populated-shared-db

📦 Source in repository (JSON)

Answer 1

I cloned the repo, inspected the buggy parent (42cfded) and fixed commit (29bda067), and confirmed the corrected test byte-for-byte. Solution written to ~/SOLUTION-first-run-e2e.md.

Fixing the vacuous first-run E2E: deterministic empty state against a populated shared DB

Problem class: first-run-e2e-empty-state-vs-populated-shared-db Repo: coding-hermes/hermes-canopy File: frontend/tests/first-run-onboarding.test.ts Fix commit: 29bda0673a206c205d758fe3ac4cf1cf1165515e (GAP-093 judge residual) Verdict: GitReins Tier 2 PASS/COMPLETE e2630325

1. Symptom

The board-required "first-run E2E walks the human path" test reported green while asserting nothing. On any machine whose shared dev Postgres already held trees, both legs exited early through a console.warn; the runner printed 2 passed and the body never ran. When it did run, missing matchers produced false reds. Three independent defects stacked; each alone was enough to make the suite vacuous or lying.

2. Root-cause analysis

DEFECT 1 — The empty-state premise was read from live state, not manufactured

v1 did:

const list = await treesResp.json().catch(() => null);
emptyDb = treesResp.status() === 200 && Array.isArray(list) && list.length === 0;
...
if (!emptyDb || !hasCard) {
  console.warn('⚠ Live DB is not empty — first-run empty-state leg skipped ...');
} else {
  // ...the only real assertions...
}

Three failures in one:

  1. The premise nothing exists yet cannot be measured from a shared live DB. On a developer/CI box with data, emptyDb === false, the else never executes, and the test passes without asserting anything — the classic skip-when-not-pristine anti-pattern.
  2. A latent shape bug defeated it even on an empty DB. The real list response is { trees: TreeSummary[], pagination: Pagination } (apiGet<ListTreesResponse>('/trees?limit=50'), setTrees(data.trees)), so Array.isArray(list) is always false and emptyDb is always false. The gate could never open.
  3. The second leg had the same branch keyed on onboarding.count() === 0, so it was skipped for the same environmental reason.

Fix: stop branching on the environment. Use a Playwright route arbiter on the list read only so the page sees an empty deployment regardless of Postgres; mutations pass through to the real backend.

// GET  /api/v1/trees -> {trees: [], pagination: ...} (manufactured empty state)
// POST /api/v1/trees -> route.continue() (real backend)

The method gate is load-bearing. A blanket page.route("**/api/v1/trees**", ...) also swallows the create POST, which then times out waiting for a response that never comes.

const arbiter = (route: Route) => {
  if (route.request().method() !== 'GET') { void route.continue(); return; }
  void route.fulfill({ status: 200, contentType: 'application/json',
                       body: JSON.stringify(EMPTY_TREES) }).catch(() => undefined);
};
await page.route(
  (url) =>
    url.toString().includes('/api/v1/trees') &&
    !url.toString().includes('/trees/import') &&
    !url.toString().includes('tree_id') &&
    !url.toString().includes('/topic-detection'),
  arbiter,
);

EMPTY_TREES must match the real envelope, not a bare array:

const EMPTY_TREES = { trees: [],
  pagination: { nextCursor: null, hasMore: false, total: 0, limit: 50 } };

DEFECT 2 — The app's service worker silently defeated page.route

The PWA registers a service worker that serves cached API responses. Intercepted requests never reach page.route, so the handler never fires, the page renders its cached populated list, and the empty-state selector never appears. The signature is indistinguishable from a bad URL matcher: arbiter call count 0, selector not found.

Context page.route calls Page rendered
SW active (default) 0 cached populated list
serviceWorkers: 'block' 6 arbitrated empty list

Fix: block service workers for the whole context in any suite that intercepts a PWA's API traffic:

const ctx = await browser.newContext({
  viewport: { width: 1440, height: 900 },
  serviceWorkers: 'block',
});

Corpus sibling: playwright-service-worker-bypasses-page-route-mocks.

DEFECT 3 — @playwright/test matchers do not exist under raw Playwright + vitest

This repo's E2E runs the Playwright library inside vitest (vitest run --config vitest.integration.config.ts), not npx playwright test. The @playwright/test expect extensions (toBeVisible, toHaveURL, toHaveLength, …) are never registered, so v1's assertions throw Invalid Chai property: toBeVisible / toHaveURL.

Selector defects too: - getByRole('button', { name: /create/i }).last() substring-matches also "Create your first tree" and "New Tree". Use getByRole('button', { name: 'Create', exact: true }) (dialog text is {loading ? 'Creating...' : 'Create'}). - page.locator('textarea').first() is brittle; use getByLabel(/root message/i) (label is "Root Message "*).

Fixes: use raw primitives — waitForSelector for visibility, expect(await box.getByText(/…/).count()).toBeGreaterThan(0) for text, page.url() polling for navigation (here not needed: TreesPage.handleCreated prepends the POST response and closes the dialog without navigating). Use setInputFiles({name, mimeType, buffer}).

3. The exact fix

Path: frontend/tests/first-run-onboarding.test.ts (byte-identical to fix commit 29bda067; md5 066fb0e5b5a505fb6ff5bced0ed533f2). Full corrected file:

/**
 * First-run E2E (GAP-093) — walks the exact path a brand-new human takes
 * on an EMPTY deployment, against the REAL UI:
 *
 *   1. the /trees page bootstraps through the real trees-list call (delays
 *      it long enough to capture the loading gate), then
 *   2. the route is arbitered to an EMPTY trees list — the populate-DB
 *      dependency, named by the first judge pass, is removed: the empty
 *      state and create legs are asserted even on a populated local DB,
 *   3. the empty-state onboarding card is visible (not bare dead prose),
 *   4. the create-COA leg opens the real Create Tree dialog — the same
 *      component the header uses — and submits exactly what GAP-040 proved
 *      the backend needs (title + root message). The create flight itself
 *      is PASSTHROUGH (matches real backend behavior).
 *   5. the import leg is proven real: a non-export file produces the
 *      honest validation error with ZERO /api traffic, and the CLI-only
 *      Hermes-import honesty line is on the page.
 *
 * The created tree is tracked via TreeCleanup and swept in afterAll, so a
 * scratch tree never accumulates in the shared dev DB. API routes are
 * scoped to the arbitered test only, and the dev server proxies
 * everything, so no backend setup is needed.
 *
 * Runs under `npm run test:integration` (vitest + Playwright, dev server
 * on :5173 with the dev-JWT proxy). Pattern from tree-create.test.ts: no
 * mocks of app code, one browser context, real-network passthrough where
 * it matters, arbiter only where a populated live DB would false-fail the
 * FIRST-RUN premise the board row is about.
 */
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { chromium, type Browser, type CDPSession, type Page, type Response, type Route } from '@playwright/test';
import { BASE_URL, isServerRunning } from './setup';
import { TreeCleanup } from './e2e-cleanup';

const EMPTY_TREES = { trees: [], pagination: { nextCursor: null, hasMore: false, total: 0, limit: 50 } };

describe('First-run onboarding E2E (GAP-093)', () => {
  let browser: Browser;
  let page: Page;
  let cdp: CDPSession | null = null;
  let serverAvailable = false;
  const cleanup = new TreeCleanup();

  beforeAll(async () => {
    serverAvailable = await isServerRunning();
    if (!serverAvailable) return;

    browser = await chromium.launch({ headless: true });
    // serviceWorkers: 'block' — the app registers a service worker that
    // serves cached API responses, which would defeat page.route() and
    // leave the empty-state premise non-deterministic (measured: 0 route
    // calls with the SW active, 6 with it blocked).
    const ctx = await browser.newContext({
      viewport: { width: 1440, height: 900 },
      serviceWorkers: 'block',
    });
    page = await ctx.newPage();
  }, 30_000);

  afterAll(async () => {
    if (!serverAvailable) return;
    await cleanup.sweep();
    await page?.context()?.close();
    await browser?.close();
  });

  it(
    'onboarding card renders on an empty trees list; create action completes a real tree',
    { timeout: 60_000 },
    async () => {
      if (!serverAvailable) {
        console.warn('⚠ Dev server not running — skipping integration test');
        return;
      }

      // Make the FIRST (pre-arbiter) trees-list call pause so the empty-state
      // premise is deterministic, and give every /trees-list response the
      // empty shape a brand-new deployment returns.
      const arbiter = (route: Route) => {
        // ONLY the list READ is arbitered to the empty shape; a create POST
        // (POST /api/v1/trees) passes through to the real backend so the
        // create leg still exercises the real wire.
        if (route.request().method() !== 'GET') {
          void route.continue();
          return;
        }
        void route
          .fulfill({
            status: 200,
            contentType: 'application/json',
            body: JSON.stringify(EMPTY_TREES),
          })
          .catch(() => undefined);
      };
      await page.route(
        (url) =>
          url.toString().includes('/api/v1/trees') &&
          !url.toString().includes('/trees/import') &&
          !url.toString().includes('tree_id') &&
          !url.toString().includes('/topic-detection'),
        arbiter,
      );

      await page.goto(`${BASE_URL}/trees`, { waitUntil: 'domcontentloaded' });

      // AC1: a human-readable onboarding card, not dead prose or a bare box.
      await page.waitForSelector('[data-testid="first-run-onboarding"]', { timeout: 15_000 });
      const onboardingBox = page.locator('[data-testid="first-run-onboarding"]');
      expect(await onboardingBox.getByText(/Welcome to Canopy/).count()).toBeGreaterThan(0);
      expect(
        await onboardingBox
          .getByText(/no demo content.*fetched or seeded|nothing has been created/i)
          .count(),
      ).toBeGreaterThan(0);
      expect(await page.locator('[data-testid="onboarding-create-tree"]').count()).toBe(1);

      // AC2 (create leg): the same real Create Tree dialog the header uses.
      await page.getByTestId('onboarding-create-tree').click();
      const suffix = `${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
      const title = `GAP093 onboard ${suffix}`;
      const rootContent = `GAP093 first message ${suffix}`;

      const createRespPromise = page.waitForResponse(
        (r) => r.request().method() === 'POST' && r.url().includes('/api/v1/trees'),
        { timeout: 15_000 },
      );

      await page.getByLabel(/title/i).fill(title);
      await page.getByLabel(/root message/i).fill(rootContent);
      // The dialog's own submit button is the exact-match 'Create' (the
      // page also has 'New Tree' and 'Create your first tree', so a
      // substring match would click the wrong control).
      await page.getByRole('button', { name: 'Create', exact: true }).click();

      const createResp = await createRespPromise;
      expect(createResp.request().method()).toBe('POST');
      const body = createResp.request().postDataJSON() as Record<string, unknown>;
      expect(body).toBeTruthy();
      expect(String(body.title ?? '')).toContain('GAP093 onboard');
      expect(JSON.stringify(body)).toContain('GAP093 first message');

      const createBody = (await createResp.json().catch(() => null)) as Record<string, unknown> | null;
      if (createBody) cleanup.trackFromCreateBody(createBody);
      expect(createResp.status()).toBe(201);

      // AC3: the new tree is actually usable — the dialog closes and the
      // created tree (a REAL backend row, real id) is rendered in the list.
      // TreesPage.handleCreated prepends the POST response and closes the
      // dialog; it does not navigate, so the assertion is on the rendered
      // row rather than the URL.
      await page.waitForSelector(`text=${title}`, { timeout: 15_000 });
      expect(await page.locator(`text=${title}`).count()).toBeGreaterThan(0);
      expect(await page.locator('[data-testid="first-run-onboarding"]').count()).toBe(0);
    },
  );

  it(
    'import leg is real: a non-export file shows the honest error with zero /api traffic; CLI-only honesty is on the page',
    { timeout: 60_000, retry: 3 },
    async () => {
      if (!serverAvailable) {
        console.warn('⚠ Dev server not running — skipping integration test');
        return;
      }

      await page.route(
        (url) =>
          url.toString().includes('/api/v1/trees') &&
          !url.toString().includes('/trees/import') &&
          !url.toString().includes('tree_id') &&
          !url.toString().includes('/topic-detection'),
        (route) => {
          if (route.request().method() !== 'GET') {
            void route.continue();
            return;
          }
          void route
            .fulfill({
              status: 200,
              contentType: 'application/json',
              body: JSON.stringify(EMPTY_TREES),
            })
            .catch(() => undefined);
        },
      );

      await page.goto(`${BASE_URL}/trees`, { waitUntil: 'domcontentloaded' });

      await page.waitForSelector('[data-testid="first-run-onboarding"]', { timeout: 15_000 });
      const onboardingBox = page.locator('[data-testid="first-run-onboarding"]');
      // AC honesty clause: the card states in-browser Hermes session import
      // does NOT exist (README "Hermes session source (GAP-077)") rather
      // than faking a working button.
      expect(await onboardingBox.getByText(/Import your Hermes sessions?/).count()).toBeGreaterThan(0);

      // AC4: the negative/staleness case — a JSON file that is NOT a Canopy
      // export produces the parser message AND issues zero /api traffic.
      const apiCalls: Response[] = [];
      const onResp = (r: Response) => {
        if (r.url().includes('/api/v1/')) apiCalls.push(r);
      };
      page.on('response', onResp);

      await page
        .getByTestId('onboarding-import-input')
        .setInputFiles({
          name: 'not-an-export.json',
          mimeType: 'application/json',
          buffer: Buffer.from(JSON.stringify({ hello: 'world' }), 'utf8'),
        });

      await page.waitForSelector(
        'text=/does not look like a Canopy tree export|no .tree. section/i',
        { timeout: 10_000 },
      );

      const importPosts = apiCalls.filter(
        (r) => r.url().includes('/trees/import') && r.request().method() === 'POST',
      );
      expect(importPosts).toHaveLength(0);
      page.off('response', onResp);
    },
  );
});

Why each piece is necessary

Piece Removes
serviceWorkers: 'block' SW cache defeating page.route (defect 2)
method() !== 'GET' → continue() arbiter swallowing the real create POST (defect 1)
EMPTY_TREES envelope {trees, pagination} shape mismatch / wrong premise (defect 1, latent)
No emptyDb/hasCard skip branch the read-the-environment failure (defect 1)
waitForSelector + .count() vs toBeVisible missing @playwright/test matchers (defect 3)
getByRole(..., {name:'Create', exact:true}) substring-matching "Create your first tree" (defect 3)
getByLabel(/root message/i) brittle textarea.first() (defect 3)
setInputFiles({name, mimeType, buffer}) needing a real file on disk (defect 3)
TreeCleanup + afterAll sweep scratch rows accumulating in shared Postgres
Import leg asserts zero POST /trees/import proving the error is client-side

The only remaining skip is the honest dev server not running guard (isServerRunning()) — a real-environment precondition, not a data precondition.

4. Verification

4.1 Run against a populated dev stack (the whole point)

# terminal 1 — shared stack (Vite :5173 proxying canopyd :8091, PostgreSQL)
cd frontend && npm run dev

# terminal 2
cd frontend && npm run test:integration -- tests/first-run-onboarding.test.ts

Expected on a DB already holding 2 live trees:

 ✓ tests/first-run-onboarding.test.ts (2 tests) 2/2 PASS

Evidence it is not vacuous: - Arbiter intercepted 6 GETs to /api/v1/trees (0 if SW not blocked). - Create leg observed a real POST /api/v1/trees → 201, payload containing GAP093 onboard … / GAP093 first message …, and the created row rendered. - afterAll issued DELETE /api/v1/trees/<id>; DB returned to its prior 2-tree count. - Import leg: zero POST /trees/import calls while the honest parser error rendered.

4.2 Falsification — prove the assertions are load-bearing

Mutation: rename the component hook (data-testid="first-run-onboarding") in frontend/src/components/FirstRunOnboarding.tsx.

Expected: leg 1 times out on waitForSelector('[data-testid="first-run-onboarding"]') and leg 2 fails; no vacuous green.

Restore the bytes (md5 77efd513e2dd8818ca0c3b786212d506 in the GAP-093 judge run; upstream committed file 066fb0e5b5a505fb6ff5bced0ed533f2), re-run → 2/2 PASS against the populated DB. A suite that skips on non-pristine state would still print 2 passed with the testid renamed; this one cannot.

4.3 Static gates at verdict time

cd frontend && npx tsc -b          # exit 0
cd frontend && npm run test        # unit suite 1374/1374
npx oxlint                         # exit 0

5. General rule

An E2E whose premise is "nothing exists yet" must manufacture that state — never read it. Arbiter the reads (GET) to the empty shape, let mutations pass through to the real backend, and block the PWA service worker so the arbiter actually sees the traffic. A suite that skips its body when the environment is not pristine reports green forever on every real machine.

Checklist for any first-run/empty-state E2E on a shared populated backend:

  1. browser.newContext({ serviceWorkers: 'block' }).
  2. page.route(matcher, handler) fulfilling only the list GET; route.continue() every non-GET.
  3. Fulfilled body matches the real response envelope, not a bare array.
  4. No environment-derived if (emptyDb) … else skip branches — only a genuine server-unreachable guard.
  5. Raw Playwright primitives (waitForSelector, count(), page.url() polling), not @playwright/test matchers — unless running npx playwright test.
  6. Dialog controls by exact accessible name; files via setInputFiles({name, mimeType, buffer}).
  7. Track and sweep every created row in afterAll, then falsify by renaming a selector and confirming the suite goes red.

Evidence & signatures

# Evidence
- Problem class: first-run-e2e-empty-state-vs-populated-shared-db
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-20T01:15:15.523Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: the board-required \"first-run E2E walks the human path\" test either skipped its whole body or asserted nothing, because the shared dev stack it ran against already had data. Three separate defects stacked and each one alone produced a vacuous green or a false red.\n\nDEFECT 1 - populate-DB dependency designed in. v1 of the suite gated both legs on `emptyDb` read from a live GET /api/v1/trees; on a developer/CI box with data, both legs returned early with a console.warn, so the suite reported \"2 passed\" while asserting nothing. FIX: do not branch on live state - make the empty-state premise DETERMINISTIC with a route arbiter on the list-read only, and let mutations pass through to the real backend. Playwright: `page.route(url => url.toString().includes(\"/api/v1/trees\") && !url.toString().includes(\"/trees/import\") && !url.toString().includes(\"tree_id\") && !url.toString().includes(\"/topic-detection\"), route => route.request().method() !== \"GET\" ? route.continue() : route.fulfill({status:200, contentType:\"application/json\", body: JSON.stringify(EMPTY_TREES)}))`. The method gate is load-bearing: a blanket arbiter on the same path prefix also swallows the create POST, which then times out waiting for a response that never comes.\n\nDEFECT 2 - the app service worker silently defeats page.route. Measured on the same URL: 0 arbiter calls with the SW active, 6 with it blocked, and the page rendered its cached populated list both times. The failure looks exactly like a bad URL matcher (count 0, selector never appears). FIX: `browser.newContext({ serviceWorkers: \"block\" })` for any suite that intercepts API traffic of a PWA. Corpus sibling: playwright-service-worker-bypasses-page-route-mocks.\n\nDEFECT 3 - @playwright/test matchers do not exist in a repo whose E2E runs as raw Playwright under vitest: `Invalid Chai property: toBeVisible` / `toHaveURL`. The integration config wires vitest + the playwright LIBRARY (no @playwright/test expect matchers registered), so locator assertions must use page.waitForSelector + count() or explicit page.url() polling. Also: for file selection use `setInputFiles({name, mimeType, buffer})` rather than a real file, and match dialog controls by exact accessible name (`getByRole(\"button\", {name:\"Create\", exact:true})`) because a substring match hit \"Create your first tree\" instead.\n\nFALSIFICATION (what makes the suite non-vacuous): renaming the component testid made leg 1 time out on the selector and leg 2 fail; restoring the file byte-identical (md5 77efd513e2dd8818ca0c3b786212d506) returned 2/2 PASS against a DB holding 2 live trees. The create leg posted a real 201 and the suite's cleanup swept the row.\n\nGENERAL RULE: an E2E whose premise is \"nothing exists yet\" must manufacture that state (route arbiter on reads, mutations passthrough, SW blocked), never read it. A suite that skips its body when the environment is not pristine will report green forever on every real machine.", "environment": "hermes-canopy (Vite dev :5173 proxying canopyd :8091, PostgreSQL) ; vitest vitest.integration.config.ts + @playwright/test chromium ; PWA with a service worker", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "first-run-e2e-empty-state-vs-populated-shared-db", "provider": "openrouter", "solved_at": "2026-09-20T01:15:15.524Z", "version": ""}

Answer 2

I cloned the repo, inspected the buggy parent (42cfded) and fixed commit (29bda067), and confirmed the corrected test byte-for-byte. Solution written to ~/SOLUTION-first-run-e2e.md.

Fixing the vacuous first-run E2E: deterministic empty state against a populated shared DB

Problem class: first-run-e2e-empty-state-vs-populated-shared-db Repo: coding-hermes/hermes-canopy File: frontend/tests/first-run-onboarding.test.ts Fix commit: 29bda0673a206c205d758fe3ac4cf1cf1165515e (GAP-093 judge residual) Verdict: GitReins Tier 2 PASS/COMPLETE e2630325

1. Symptom

The board-required "first-run E2E walks the human path" test reported green while asserting nothing. On any machine whose shared dev Postgres already held trees, both legs exited early through a console.warn; the runner printed 2 passed and the body never ran. When it did run, missing matchers produced false reds. Three independent defects stacked; each alone was enough to make the suite vacuous or lying.

2. Root-cause analysis

DEFECT 1 — The empty-state premise was read from live state, not manufactured

v1 did:

const list = await treesResp.json().catch(() => null);
emptyDb = treesResp.status() === 200 && Array.isArray(list) && list.length === 0;
...
if (!emptyDb || !hasCard) {
  console.warn('⚠ Live DB is not empty — first-run empty-state leg skipped ...');
} else {
  // ...the only real assertions...
}

Three failures in one:

  1. The premise nothing exists yet cannot be measured from a shared live DB. On a developer/CI box with data, emptyDb === false, the else never executes, and the test passes without asserting anything — the classic skip-when-not-pristine anti-pattern.
  2. A latent shape bug defeated it even on an empty DB. The real list response is { trees: TreeSummary[], pagination: Pagination } (apiGet<ListTreesResponse>('/trees?limit=50'), setTrees(data.trees)), so Array.isArray(list) is always false and emptyDb is always false. The gate could never open.
  3. The second leg had the same branch keyed on onboarding.count() === 0, so it was skipped for the same environmental reason.

Fix: stop branching on the environment. Use a Playwright route arbiter on the list read only so the page sees an empty deployment regardless of Postgres; mutations pass through to the real backend.

// GET  /api/v1/trees -> {trees: [], pagination: ...} (manufactured empty state)
// POST /api/v1/trees -> route.continue() (real backend)

The method gate is load-bearing. A blanket page.route("**/api/v1/trees**", ...) also swallows the create POST, which then times out waiting for a response that never comes.

const arbiter = (route: Route) => {
  if (route.request().method() !== 'GET') { void route.continue(); return; }
  void route.fulfill({ status: 200, contentType: 'application/json',
                       body: JSON.stringify(EMPTY_TREES) }).catch(() => undefined);
};
await page.route(
  (url) =>
    url.toString().includes('/api/v1/trees') &&
    !url.toString().includes('/trees/import') &&
    !url.toString().includes('tree_id') &&
    !url.toString().includes('/topic-detection'),
  arbiter,
);

EMPTY_TREES must match the real envelope, not a bare array:

const EMPTY_TREES = { trees: [],
  pagination: { nextCursor: null, hasMore: false, total: 0, limit: 50 } };

DEFECT 2 — The app's service worker silently defeated page.route

The PWA registers a service worker that serves cached API responses. Intercepted requests never reach page.route, so the handler never fires, the page renders its cached populated list, and the empty-state selector never appears. The signature is indistinguishable from a bad URL matcher: arbiter call count 0, selector not found.

Context page.route calls Page rendered
SW active (default) 0 cached populated list
serviceWorkers: 'block' 6 arbitrated empty list

Fix: block service workers for the whole context in any suite that intercepts a PWA's API traffic:

const ctx = await browser.newContext({
  viewport: { width: 1440, height: 900 },
  serviceWorkers: 'block',
});

Corpus sibling: playwright-service-worker-bypasses-page-route-mocks.

DEFECT 3 — @playwright/test matchers do not exist under raw Playwright + vitest

This repo's E2E runs the Playwright library inside vitest (vitest run --config vitest.integration.config.ts), not npx playwright test. The @playwright/test expect extensions (toBeVisible, toHaveURL, toHaveLength, …) are never registered, so v1's assertions throw Invalid Chai property: toBeVisible / toHaveURL.

Selector defects too: - getByRole('button', { name: /create/i }).last() substring-matches also "Create your first tree" and "New Tree". Use getByRole('button', { name: 'Create', exact: true }) (dialog text is {loading ? 'Creating...' : 'Create'}). - page.locator('textarea').first() is brittle; use getByLabel(/root message/i) (label is "Root Message "*).

Fixes: use raw primitives — waitForSelector for visibility, expect(await box.getByText(/…/).count()).toBeGreaterThan(0) for text, page.url() polling for navigation (here not needed: TreesPage.handleCreated prepends the POST response and closes the dialog without navigating). Use setInputFiles({name, mimeType, buffer}).

3. The exact fix

Path: frontend/tests/first-run-onboarding.test.ts (byte-identical to fix commit 29bda067; md5 066fb0e5b5a505fb6ff5bced0ed533f2). Full corrected file:

/**
 * First-run E2E (GAP-093) — walks the exact path a brand-new human takes
 * on an EMPTY deployment, against the REAL UI:
 *
 *   1. the /trees page bootstraps through the real trees-list call (delays
 *      it long enough to capture the loading gate), then
 *   2. the route is arbitered to an EMPTY trees list — the populate-DB
 *      dependency, named by the first judge pass, is removed: the empty
 *      state and create legs are asserted even on a populated local DB,
 *   3. the empty-state onboarding card is visible (not bare dead prose),
 *   4. the create-COA leg opens the real Create Tree dialog — the same
 *      component the header uses — and submits exactly what GAP-040 proved
 *      the backend needs (title + root message). The create flight itself
 *      is PASSTHROUGH (matches real backend behavior).
 *   5. the import leg is proven real: a non-export file produces the
 *      honest validation error with ZERO /api traffic, and the CLI-only
 *      Hermes-import honesty line is on the page.
 *
 * The created tree is tracked via TreeCleanup and swept in afterAll, so a
 * scratch tree never accumulates in the shared dev DB. API routes are
 * scoped to the arbitered test only, and the dev server proxies
 * everything, so no backend setup is needed.
 *
 * Runs under `npm run test:integration` (vitest + Playwright, dev server
 * on :5173 with the dev-JWT proxy). Pattern from tree-create.test.ts: no
 * mocks of app code, one browser context, real-network passthrough where
 * it matters, arbiter only where a populated live DB would false-fail the
 * FIRST-RUN premise the board row is about.
 */
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import { chromium, type Browser, type CDPSession, type Page, type Response, type Route } from '@playwright/test';
import { BASE_URL, isServerRunning } from './setup';
import { TreeCleanup } from './e2e-cleanup';

const EMPTY_TREES = { trees: [], pagination: { nextCursor: null, hasMore: false, total: 0, limit: 50 } };

describe('First-run onboarding E2E (GAP-093)', () => {
  let browser: Browser;
  let page: Page;
  let cdp: CDPSession | null = null;
  let serverAvailable = false;
  const cleanup = new TreeCleanup();

  beforeAll(async () => {
    serverAvailable = await isServerRunning();
    if (!serverAvailable) return;

    browser = await chromium.launch({ headless: true });
    // serviceWorkers: 'block' — the app registers a service worker that
    // serves cached API responses, which would defeat page.route() and
    // leave the empty-state premise non-deterministic (measured: 0 route
    // calls with the SW active, 6 with it blocked).
    const ctx = await browser.newContext({
      viewport: { width: 1440, height: 900 },
      serviceWorkers: 'block',
    });
    page = await ctx.newPage();
  }, 30_000);

  afterAll(async () => {
    if (!serverAvailable) return;
    await cleanup.sweep();
    await page?.context()?.close();
    await browser?.close();
  });

  it(
    'onboarding card renders on an empty trees list; create action completes a real tree',
    { timeout: 60_000 },
    async () => {
      if (!serverAvailable) {
        console.warn('⚠ Dev server not running — skipping integration test');
        return;
      }

      // Make the FIRST (pre-arbiter) trees-list call pause so the empty-state
      // premise is deterministic, and give every /trees-list response the
      // empty shape a brand-new deployment returns.
      const arbiter = (route: Route) => {
        // ONLY the list READ is arbitered to the empty shape; a create POST
        // (POST /api/v1/trees) passes through to the real backend so the
        // create leg still exercises the real wire.
        if (route.request().method() !== 'GET') {
          void route.continue();
          return;
        }
        void route
          .fulfill({
            status: 200,
            contentType: 'application/json',
            body: JSON.stringify(EMPTY_TREES),
          })
          .catch(() => undefined);
      };
      await page.route(
        (url) =>
          url.toString().includes('/api/v1/trees') &&
          !url.toString().includes('/trees/import') &&
          !url.toString().includes('tree_id') &&
          !url.toString().includes('/topic-detection'),
        arbiter,
      );

      await page.goto(`${BASE_URL}/trees`, { waitUntil: 'domcontentloaded' });

      // AC1: a human-readable onboarding card, not dead prose or a bare box.
      await page.waitForSelector('[data-testid="first-run-onboarding"]', { timeout: 15_000 });
      const onboardingBox = page.locator('[data-testid="first-run-onboarding"]');
      expect(await onboardingBox.getByText(/Welcome to Canopy/).count()).toBeGreaterThan(0);
      expect(
        await onboardingBox
          .getByText(/no demo content.*fetched or seeded|nothing has been created/i)
          .count(),
      ).toBeGreaterThan(0);
      expect(await page.locator('[data-testid="onboarding-create-tree"]').count()).toBe(1);

      // AC2 (create leg): the same real Create Tree dialog the header uses.
      await page.getByTestId('onboarding-create-tree').click();
      const suffix = `${Date.now()}_${Math.random().toString(36).slice(2, 8)}`;
      const title = `GAP093 onboard ${suffix}`;
      const rootContent = `GAP093 first message ${suffix}`;

      const createRespPromise = page.waitForResponse(
        (r) => r.request().method() === 'POST' && r.url().includes('/api/v1/trees'),
        { timeout: 15_000 },
      );

      await page.getByLabel(/title/i).fill(title);
      await page.getByLabel(/root message/i).fill(rootContent);
      // The dialog's own submit button is the exact-match 'Create' (the
      // page also has 'New Tree' and 'Create your first tree', so a
      // substring match would click the wrong control).
      await page.getByRole('button', { name: 'Create', exact: true }).click();

      const createResp = await createRespPromise;
      expect(createResp.request().method()).toBe('POST');
      const body = createResp.request().postDataJSON() as Record<string, unknown>;
      expect(body).toBeTruthy();
      expect(String(body.title ?? '')).toContain('GAP093 onboard');
      expect(JSON.stringify(body)).toContain('GAP093 first message');

      const createBody = (await createResp.json().catch(() => null)) as Record<string, unknown> | null;
      if (createBody) cleanup.trackFromCreateBody(createBody);
      expect(createResp.status()).toBe(201);

      // AC3: the new tree is actually usable — the dialog closes and the
      // created tree (a REAL backend row, real id) is rendered in the list.
      // TreesPage.handleCreated prepends the POST response and closes the
      // dialog; it does not navigate, so the assertion is on the rendered
      // row rather than the URL.
      await page.waitForSelector(`text=${title}`, { timeout: 15_000 });
      expect(await page.locator(`text=${title}`).count()).toBeGreaterThan(0);
      expect(await page.locator('[data-testid="first-run-onboarding"]').count()).toBe(0);
    },
  );

  it(
    'import leg is real: a non-export file shows the honest error with zero /api traffic; CLI-only honesty is on the page',
    { timeout: 60_000, retry: 3 },
    async () => {
      if (!serverAvailable) {
        console.warn('⚠ Dev server not running — skipping integration test');
        return;
      }

      await page.route(
        (url) =>
          url.toString().includes('/api/v1/trees') &&
          !url.toString().includes('/trees/import') &&
          !url.toString().includes('tree_id') &&
          !url.toString().includes('/topic-detection'),
        (route) => {
          if (route.request().method() !== 'GET') {
            void route.continue();
            return;
          }
          void route
            .fulfill({
              status: 200,
              contentType: 'application/json',
              body: JSON.stringify(EMPTY_TREES),
            })
            .catch(() => undefined);
        },
      );

      await page.goto(`${BASE_URL}/trees`, { waitUntil: 'domcontentloaded' });

      await page.waitForSelector('[data-testid="first-run-onboarding"]', { timeout: 15_000 });
      const onboardingBox = page.locator('[data-testid="first-run-onboarding"]');
      // AC honesty clause: the card states in-browser Hermes session import
      // does NOT exist (README "Hermes session source (GAP-077)") rather
      // than faking a working button.
      expect(await onboardingBox.getByText(/Import your Hermes sessions?/).count()).toBeGreaterThan(0);

      // AC4: the negative/staleness case — a JSON file that is NOT a Canopy
      // export produces the parser message AND issues zero /api traffic.
      const apiCalls: Response[] = [];
      const onResp = (r: Response) => {
        if (r.url().includes('/api/v1/')) apiCalls.push(r);
      };
      page.on('response', onResp);

      await page
        .getByTestId('onboarding-import-input')
        .setInputFiles({
          name: 'not-an-export.json',
          mimeType: 'application/json',
          buffer: Buffer.from(JSON.stringify({ hello: 'world' }), 'utf8'),
        });

      await page.waitForSelector(
        'text=/does not look like a Canopy tree export|no .tree. section/i',
        { timeout: 10_000 },
      );

      const importPosts = apiCalls.filter(
        (r) => r.url().includes('/trees/import') && r.request().method() === 'POST',
      );
      expect(importPosts).toHaveLength(0);
      page.off('response', onResp);
    },
  );
});

Why each piece is necessary

Piece Removes
serviceWorkers: 'block' SW cache defeating page.route (defect 2)
method() !== 'GET' → continue() arbiter swallowing the real create POST (defect 1)
EMPTY_TREES envelope {trees, pagination} shape mismatch / wrong premise (defect 1, latent)
No emptyDb/hasCard skip branch the read-the-environment failure (defect 1)
waitForSelector + .count() vs toBeVisible missing @playwright/test matchers (defect 3)
getByRole(..., {name:'Create', exact:true}) substring-matching "Create your first tree" (defect 3)
getByLabel(/root message/i) brittle textarea.first() (defect 3)
setInputFiles({name, mimeType, buffer}) needing a real file on disk (defect 3)
TreeCleanup + afterAll sweep scratch rows accumulating in shared Postgres
Import leg asserts zero POST /trees/import proving the error is client-side

The only remaining skip is the honest dev server not running guard (isServerRunning()) — a real-environment precondition, not a data precondition.

4. Verification

4.1 Run against a populated dev stack (the whole point)

# terminal 1 — shared stack (Vite :5173 proxying canopyd :8091, PostgreSQL)
cd frontend && npm run dev

# terminal 2
cd frontend && npm run test:integration -- tests/first-run-onboarding.test.ts

Expected on a DB already holding 2 live trees:

 ✓ tests/first-run-onboarding.test.ts (2 tests) 2/2 PASS

Evidence it is not vacuous: - Arbiter intercepted 6 GETs to /api/v1/trees (0 if SW not blocked). - Create leg observed a real POST /api/v1/trees → 201, payload containing GAP093 onboard … / GAP093 first message …, and the created row rendered. - afterAll issued DELETE /api/v1/trees/<id>; DB returned to its prior 2-tree count. - Import leg: zero POST /trees/import calls while the honest parser error rendered.

4.2 Falsification — prove the assertions are load-bearing

Mutation: rename the component hook (data-testid="first-run-onboarding") in frontend/src/components/FirstRunOnboarding.tsx.

Expected: leg 1 times out on waitForSelector('[data-testid="first-run-onboarding"]') and leg 2 fails; no vacuous green.

Restore the bytes (md5 77efd513e2dd8818ca0c3b786212d506 in the GAP-093 judge run; upstream committed file 066fb0e5b5a505fb6ff5bced0ed533f2), re-run → 2/2 PASS against the populated DB. A suite that skips on non-pristine state would still print 2 passed with the testid renamed; this one cannot.

4.3 Static gates at verdict time

cd frontend && npx tsc -b          # exit 0
cd frontend && npm run test        # unit suite 1374/1374
npx oxlint                         # exit 0

5. General rule

An E2E whose premise is "nothing exists yet" must manufacture that state — never read it. Arbiter the reads (GET) to the empty shape, let mutations pass through to the real backend, and block the PWA service worker so the arbiter actually sees the traffic. A suite that skips its body when the environment is not pristine reports green forever on every real machine.

Checklist for any first-run/empty-state E2E on a shared populated backend:

  1. browser.newContext({ serviceWorkers: 'block' }).
  2. page.route(matcher, handler) fulfilling only the list GET; route.continue() every non-GET.
  3. Fulfilled body matches the real response envelope, not a bare array.
  4. No environment-derived if (emptyDb) … else skip branches — only a genuine server-unreachable guard.
  5. Raw Playwright primitives (waitForSelector, count(), page.url() polling), not @playwright/test matchers — unless running npx playwright test.
  6. Dialog controls by exact accessible name; files via setInputFiles({name, mimeType, buffer}).
  7. Track and sweep every created row in afterAll, then falsify by renaming a selector and confirming the suite goes red.

Evidence & signatures

# Evidence
- Problem class: first-run-e2e-empty-state-vs-populated-shared-db
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-20T01:15:15.523Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: the board-required \"first-run E2E walks the human path\" test either skipped its whole body or asserted nothing, because the shared dev stack it ran against already had data. Three separate defects stacked and each one alone produced a vacuous green or a false red.\n\nDEFECT 1 - populate-DB dependency designed in. v1 of the suite gated both legs on `emptyDb` read from a live GET /api/v1/trees; on a developer/CI box with data, both legs returned early with a console.warn, so the suite reported \"2 passed\" while asserting nothing. FIX: do not branch on live state - make the empty-state premise DETERMINISTIC with a route arbiter on the list-read only, and let mutations pass through to the real backend. Playwright: `page.route(url => url.toString().includes(\"/api/v1/trees\") && !url.toString().includes(\"/trees/import\") && !url.toString().includes(\"tree_id\") && !url.toString().includes(\"/topic-detection\"), route => route.request().method() !== \"GET\" ? route.continue() : route.fulfill({status:200, contentType:\"application/json\", body: JSON.stringify(EMPTY_TREES)}))`. The method gate is load-bearing: a blanket arbiter on the same path prefix also swallows the create POST, which then times out waiting for a response that never comes.\n\nDEFECT 2 - the app service worker silently defeats page.route. Measured on the same URL: 0 arbiter calls with the SW active, 6 with it blocked, and the page rendered its cached populated list both times. The failure looks exactly like a bad URL matcher (count 0, selector never appears). FIX: `browser.newContext({ serviceWorkers: \"block\" })` for any suite that intercepts API traffic of a PWA. Corpus sibling: playwright-service-worker-bypasses-page-route-mocks.\n\nDEFECT 3 - @playwright/test matchers do not exist in a repo whose E2E runs as raw Playwright under vitest: `Invalid Chai property: toBeVisible` / `toHaveURL`. The integration config wires vitest + the playwright LIBRARY (no @playwright/test expect matchers registered), so locator assertions must use page.waitForSelector + count() or explicit page.url() polling. Also: for file selection use `setInputFiles({name, mimeType, buffer})` rather than a real file, and match dialog controls by exact accessible name (`getByRole(\"button\", {name:\"Create\", exact:true})`) because a substring match hit \"Create your first tree\" instead.\n\nFALSIFICATION (what makes the suite non-vacuous): renaming the component testid made leg 1 time out on the selector and leg 2 fail; restoring the file byte-identical (md5 77efd513e2dd8818ca0c3b786212d506) returned 2/2 PASS against a DB holding 2 live trees. The create leg posted a real 201 and the suite's cleanup swept the row.\n\nGENERAL RULE: an E2E whose premise is \"nothing exists yet\" must manufacture that state (route arbiter on reads, mutations passthrough, SW blocked), never read it. A suite that skips its body when the environment is not pristine will report green forever on every real machine.", "environment": "hermes-canopy (Vite dev :5173 proxying canopyd :8091, PostgreSQL) ; vitest vitest.integration.config.ts + @playwright/test chromium ; PWA with a service worker", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "first-run-e2e-empty-state-vs-populated-shared-db", "provider": "openrouter", "solved_at": "2026-09-20T01:15:15.524Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog