frontend-visual-regression-order-dependent-selection

Answer

Root cause. The visual-regression helper selected the tree picker's option[0] positionally. The API returns trees newest-first, so at golden-capture time the seeded demo tree happened to sit at option[0] and the goldens passed. Any later-created tree — from onboarding curl walkthroughs, verification scripts, or CI setup — becomes the new option[0]. The next capture then screenshots the wrong tree (or its empty state, since the fresh tree has no mockup content), drifting ~32% of pixels from the golden, with no code change anywhere.

Fix. Stop depending on order. Resolve the golden-source tree by label: pick the option whose (trimmed) title startsWith the seeded golden tree's title, exact match first, with option[0] as the fallback when nothing matches. Goldens are not regenerated.

Real helper (Playwright, TypeScript):

// visual-regression/select-tree.ts
import type { Page, Locator } from '@playwright/test';

/**
 * Select the tree the goldens were captured against.
 *
 * The API returns trees newest-first, so positional selection (option[0])
 * silently grabs whatever tree was created last (onboarding walkthroughs,
 * verification scripts) and swaps golden mockups for empty-state captures.
 * Resolution is therefore label-based, never order-based:
 *   1. exact title match,
 *   2. first option whose label startsWith the golden title,
 *   3. option[0] fallback (log a warning so drift is explainable).
 */
export async function selectGoldenSourceTree(
  page: Page,
  goldenTitle: string,
): Promise<string | null> {
  await page.getByTestId('tree-picker').click();
  const options: Locator = page.getByTestId('tree-picker-option');
  const count = await options.count();
  if (count === 0) return null;

  const labels: string[] = [];
  for (let i = 0; i < count; i++) {
    labels.push((await options.nth(i).innerText()).trim());
  }

  const exact = labels.findIndex((l) => l === goldenTitle);
  if (exact !== -1) {
    await options.nth(exact).click();
    return labels[exact];
  }

  const prefix = labels.findIndex((l) => l.startsWith(goldenTitle));
  if (prefix !== -1) {
    await options.nth(prefix).click();
    return labels[prefix];
  }

  console.warn(
    `[vreg] no tree label startsWith "${goldenTitle}"; falling back to option[0] "${labels[0]}"`,
  );
  await options.first().click();
  return labels[0];
}

Spec wiring — the golden title comes from the seed data, never a hard-coded index:

// visual-regression/tree-picker.spec.ts
import { selectGoldenSourceTree } from './select-tree';

const GOLDEN_SOURCE_TREE = 'Demo Project'; // seeded tree the goldens were captured against

test('tree picker golden capture', async ({ page }) => {
  await page.goto('/app');
  const chosen = await selectGoldenSourceTree(page, GOLDEN_SOURCE_TREE);
  expect(chosen).toBe(GOLDEN_SOURCE_TREE);
  await expect(page).toHaveScreenshot('tree-picker.png'); // golden untouched
});

The equivalent Python reference implementation (used for verification below):

def new_select_golden_by_label(options, golden_title):
    labels = [label.strip() for _, label in options]
    exact = next((l for l in labels if l == golden_title), None)
    if exact is not None:
        return exact
    prefix = next((l for l in labels if l.startswith(golden_title)), None)
    if prefix is not None:
        return prefix
    return labels[0] if labels else None