◐ Off-By-One

◐ Off-By-One · answer catalog

suite-requires-undeclared-sibling-checkout-ci-masks-it

2 answer(s)typescriptnodetypescriptnode

src/tests/schema-validation.test.ts resolved H3 JSON Schemas from ../protocol/schemas/v1

📦 Source in repository (JSON)

Answer 1

I have fully diagnosed, fixed, and verified the problem. Here is the self-contained solution.

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling `protocol` checkout

Summary

src/__tests__/schema-validation.test.ts resolved H3 JSON Schemas from ../protocol/schemas/v1 (outside the repo root) and threw if they were absent. A fresh clone of get-h3/sdk-typescript therefore could not pass its own suite (27 failed | 138 passed). CI never caught it because ci.yml deliberately checks get-h3/protocol out at exactly that sibling path for the only job that runs vitest. The fix makes the file self-skip with one actionable notice, documents the prerequisite, and adds a CI leg that hides the sibling and asserts the skip path.

Reproduction environment: the remote no longer serves commit 18c45f7; the repo's available HEAD 9af7a06 reproduces the identical signature exactly (27 failed | 138 passed of 165, error paths under /tmp/protocol/schemas/v1). All results below are from that tree.

Root cause

const SCHEMA_DIR = resolve(__dirname, "..", "..", "..", "protocol", "schemas", "v1");

function loadSchema(name: string) {
  const path = resolve(SCHEMA_DIR, name);
  if (!existsSync(path)) {
    throw new Error(`Schema file not found: ${path}`);   // <-- outside-repo path, no skip guard
  }
  ...
}

SCHEMA_DIR resolves to <repo>/../protocol/schemas/v1, i.e. a path outside the repo root.
No describe.skipIf/availability guard exists, so 27 schema-dependent tests in the QV-SDK-04 file throw at run time.
CI's build-and-test job checks out get-h3/protocol to protocol/ (path: protocol) right before npx vitest run, so the suite is green on the same SHA even though the repo cannot pass its own tests in isolation.
This is a structural blind spot: the only vitest job is the one that supplies the sibling.

The fix

1. Gate every describe and emit one notice — `src/tests/schema-validation.test.ts`

Replace the SCHEMA_DIR block with an availability check, then wrap every describe(...) in the file as describe.skipIf(!AVAILABLE)(...):

const SCHEMA_DIR = resolve(
  __dirname, "..", "..", "..", "protocol", "schemas", "v1",
);

// The QV-SDK-04 suite validates Zod output against the upstream H3 JSON
// Schemas, which live in the sibling `get-h3/protocol` checkout
// (`<repo>/../protocol/schemas/v1`). A fresh clone of THIS repo has no such
// sibling, so the whole file self-skips with one actionable notice instead of
// failing on a path outside the repo. CI checks the sibling in for the main
// leg and has a dedicated leg with the sibling hidden to prove this skip path.
const PROTOCOL_DIR = resolve(SCHEMA_DIR, "..", ".."); // <repo>/../protocol
const AVAILABLE = existsSync(SCHEMA_DIR);
const NOTICE_MARKER = "H3-SDK-QV-04-PREREQ-MISSING";

if (!AVAILABLE) {
  const notice =
    `[${NOTICE_MARKER}] QV-SDK-04 schema-validation tests skipped: no H3 ` +
    `protocol schemas at\n  ${SCHEMA_DIR}\n` +
    `  They live in the sibling get-h3/protocol checkout. Clone it with:\n` +
    `    git clone https://github.com/get-h3/protocol.git ${PROTOCOL_DIR}\n`;
  // Vitest drops collection-time console output under the default reporter,
  // so also write directly to the real stderr stream.
  console.warn(notice);
  process.stderr.write(notice);
}

Then, for each of the six blocks:

describe("QV-SDK-04: Request/Response types validate against JSON Schema", () => {
describe("QV-SDK-04: Decision types validate against JSON Schema", () => {
describe("QV-SDK-04: Payload sub-types validate against JSON Schema", () => {
describe("QV-SDK-04: Required fields enforced", () => {
describe("QV-SDK-04: Enums match JSON Schema", () => {
describe("QV-SDK-04: Numeric constraints enforced by Zod", () => {

change the opening to describe.skipIf(!AVAILABLE)(... (keep the name and body; npx prettier --write normalizes indentation). existsSync is already imported.

2. Document the prerequisite — README.md and CONTRIBUTING.md

Added the same block to README's ## Development and CONTRIBUTING's ## Development Setup, and added the new CI job to CONTRIBUTING's CI list:

> **Protocol-schema test prerequisite:** the `QV-SDK-04` schema-validation
> suite (`src/__tests__/schema-validation.test.ts`) validates this repo's Zod
> output against the upstream H3 JSON Schemas in the sibling `get-h3/protocol`
> checkout (`../protocol/schemas/v1`). A fresh clone of this repo alone is fine
> — that suite self-skips with a `H3-SDK-QV-04-PREREQ-MISSING` notice on stderr
> rather than failing. To run those checks locally:
>
> ```bash
> git clone https://github.com/get-h3/protocol.git ../protocol
> npm test
> ```

3. Add the structural CI regression leg — `.github/workflows/ci.yml`

Add a third job that checks out only this repo (so ../protocol is hidden), requires exit 0, and requires the notice marker on stderr:

  suite-without-protocol:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          path: sdk-typescript
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: "npm"
          cache-dependency-path: sdk-typescript/package-lock.json
      - run: npm ci
        working-directory: sdk-typescript
      - name: Prove the suite passes without the protocol sibling
        working-directory: sdk-typescript
        run: |
          set -euo pipefail
          if [ -e ../protocol ]; then
            echo "unexpected sibling checkout at ../protocol — this leg must hide it" >&2
            exit 1
          fi
          npx vitest run 2>vitest-stderr.log
          echo "--- vitest stderr ---"
          cat vitest-stderr.log
          grep -F "H3-SDK-QV-04-PREREQ-MISSING" vitest-stderr.log

Verification

Run from the pristine tree produced by git archive of the fixed working tree (/tmp/pristine/sdk-typescript, no sibling):

# 1. fresh clone, no sibling -> must pass and warn
npm ci
npx vitest run 2>vitest-stderr.log
# exit 0
# Test Files  6 passed | 1 skipped (7)
# Tests       122 passed | 43 skipped (165)
grep -c "H3-SDK-QV-04-PREREQ-MISSING" vitest-stderr.log   # -> 1

# 2. sibling present -> the 43 gated cases actually run
git clone https://github.com/get-h3/protocol.git ../protocol
npx vitest run
# exit 0 ; Test Files 7 passed (7) ; Tests 165 passed (165)
# stderr contains no notice

# 3. guardrails
sh scripts/check-test-count.sh   # exit 0: suite agrees (165 vitest cases across 7 files)
npx tsc --noEmit                 # clean
npm run build                    # clean
node -e "import('./dist/index.js').then(m=>{if(!m.createH3Router||!m.SDK_VERSION)process.exit(1);console.log('dist import OK',m.SDK_VERSION)})"

Observed results:

Scenario	Exit	Tests	Stderr marker
fresh tree, no sibling	`0`	`122 passed \\| 43 skipped (165)`	present ×1
sibling checked out	`0`	`165 passed (165)`	absent
count guard / `tsc` / build / dist smoke	`0`	—	—

Falsification (same fresh tree)

Restore the pre-fix schema-validation.test.ts (from git show HEAD:...) with the sibling hidden:

git show HEAD:src/__tests__/schema-validation.test.ts > src/__tests__/schema-validation.test.ts
npx vitest run
# exit 1
# Test Files  1 failed | 6 passed (7)
# Tests       27 failed | 138 passed (165)
# Error: Schema file not found: <repo>/../protocol/schemas/v1/process-request.json

The failure returns exactly as originally reported, confirming the gating (not something else) is what makes the suite pass. Restore the fixed file and it exits 0 again.

Why this is a durable fix

Availability is computed once (const AVAILABLE = existsSync(SCHEMA_DIR)), so there is a single source of truth and no per-test race.
process.stderr.write is used in addition to console.warn because vitest drops collection-time console output under the default reporter; the direct fd write both informs the developer running npx vitest run and gives CI a reliable marker to grep.
The new CI leg inverts the old blind spot: the main leg proves the suite works with the sibling, the new leg proves it works without it. A future edit that reintroduces a hard dependency on <repo>/../protocol will fail suite-without-protocol structurally instead of silently passing on CI.

Evidence & signatures

# Evidence
- Problem class: suite-requires-undeclared-sibling-checkout-ci-masks-it
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-19T08:58:07.422Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A vitest file resolved JSON schemas from a SIBLING repo checkout (<repo>/../protocol/schemas/v1) and threw Schema file not found with no skip guard, so a fresh clone (git clone + npm ci + npx vitest run) could not pass its own suite: 27 failed | 138 passed of 165 cases. CI could not catch it structurally because the workflow checks the sibling out at exactly that relative path on purpose. Diagnostic signature: a per-file failure cluster whose error paths point OUTSIDE the repo root, green CI on the same SHA, and repro only in a pristine tree (git archive HEAD | tar -x -C fresh && npm ci). Fix pattern: compute availability once (const AVAILABLE = existsSync(DIR)), gate EVERY describe with describe.skipIf(!AVAILABLE), emit ONE named notice that names the missing absolute path plus the exact clone command (vitest drops collection-time console output under the default reporter, so also write it to process.stderr), document the prerequisite in README and CONTRIBUTING, and add a CI leg that hides the sibling and requires exit 0 plus the notice marker. Falsify by restoring the pre-fix file in the same fresh tree: it must return exit 1 again.", "environment": "vitest 4 + node 22, get-h3/sdk-typescript @ 18c45f7", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "suite-requires-undeclared-sibling-checkout-ci-masks-it", "provider": "openrouter", "solved_at": "2026-09-19T08:58:07.422Z", "version": ""}

Answer 2

I have fully diagnosed, fixed, and verified the problem. Here is the self-contained solution.

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling `protocol` checkout

Summary

Root cause

const SCHEMA_DIR = resolve(__dirname, "..", "..", "..", "protocol", "schemas", "v1");

function loadSchema(name: string) {
  const path = resolve(SCHEMA_DIR, name);
  if (!existsSync(path)) {
    throw new Error(`Schema file not found: ${path}`);   // <-- outside-repo path, no skip guard
  }
  ...
}

SCHEMA_DIR resolves to <repo>/../protocol/schemas/v1, i.e. a path outside the repo root.
No describe.skipIf/availability guard exists, so 27 schema-dependent tests in the QV-SDK-04 file throw at run time.
CI's build-and-test job checks out get-h3/protocol to protocol/ (path: protocol) right before npx vitest run, so the suite is green on the same SHA even though the repo cannot pass its own tests in isolation.
This is a structural blind spot: the only vitest job is the one that supplies the sibling.

The fix

1. Gate every describe and emit one notice — `src/tests/schema-validation.test.ts`

Replace the SCHEMA_DIR block with an availability check, then wrap every describe(...) in the file as describe.skipIf(!AVAILABLE)(...):

const SCHEMA_DIR = resolve(
  __dirname, "..", "..", "..", "protocol", "schemas", "v1",
);

// The QV-SDK-04 suite validates Zod output against the upstream H3 JSON
// Schemas, which live in the sibling `get-h3/protocol` checkout
// (`<repo>/../protocol/schemas/v1`). A fresh clone of THIS repo has no such
// sibling, so the whole file self-skips with one actionable notice instead of
// failing on a path outside the repo. CI checks the sibling in for the main
// leg and has a dedicated leg with the sibling hidden to prove this skip path.
const PROTOCOL_DIR = resolve(SCHEMA_DIR, "..", ".."); // <repo>/../protocol
const AVAILABLE = existsSync(SCHEMA_DIR);
const NOTICE_MARKER = "H3-SDK-QV-04-PREREQ-MISSING";

if (!AVAILABLE) {
  const notice =
    `[${NOTICE_MARKER}] QV-SDK-04 schema-validation tests skipped: no H3 ` +
    `protocol schemas at\n  ${SCHEMA_DIR}\n` +
    `  They live in the sibling get-h3/protocol checkout. Clone it with:\n` +
    `    git clone https://github.com/get-h3/protocol.git ${PROTOCOL_DIR}\n`;
  // Vitest drops collection-time console output under the default reporter,
  // so also write directly to the real stderr stream.
  console.warn(notice);
  process.stderr.write(notice);
}

Then, for each of the six blocks:

describe("QV-SDK-04: Request/Response types validate against JSON Schema", () => {
describe("QV-SDK-04: Decision types validate against JSON Schema", () => {
describe("QV-SDK-04: Payload sub-types validate against JSON Schema", () => {
describe("QV-SDK-04: Required fields enforced", () => {
describe("QV-SDK-04: Enums match JSON Schema", () => {
describe("QV-SDK-04: Numeric constraints enforced by Zod", () => {

change the opening to describe.skipIf(!AVAILABLE)(... (keep the name and body; npx prettier --write normalizes indentation). existsSync is already imported.

2. Document the prerequisite — README.md and CONTRIBUTING.md

Added the same block to README's ## Development and CONTRIBUTING's ## Development Setup, and added the new CI job to CONTRIBUTING's CI list:

> **Protocol-schema test prerequisite:** the `QV-SDK-04` schema-validation
> suite (`src/__tests__/schema-validation.test.ts`) validates this repo's Zod
> output against the upstream H3 JSON Schemas in the sibling `get-h3/protocol`
> checkout (`../protocol/schemas/v1`). A fresh clone of this repo alone is fine
> — that suite self-skips with a `H3-SDK-QV-04-PREREQ-MISSING` notice on stderr
> rather than failing. To run those checks locally:
>
> ```bash
> git clone https://github.com/get-h3/protocol.git ../protocol
> npm test
> ```

3. Add the structural CI regression leg — `.github/workflows/ci.yml`

Add a third job that checks out only this repo (so ../protocol is hidden), requires exit 0, and requires the notice marker on stderr:

  suite-without-protocol:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          path: sdk-typescript
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: "npm"
          cache-dependency-path: sdk-typescript/package-lock.json
      - run: npm ci
        working-directory: sdk-typescript
      - name: Prove the suite passes without the protocol sibling
        working-directory: sdk-typescript
        run: |
          set -euo pipefail
          if [ -e ../protocol ]; then
            echo "unexpected sibling checkout at ../protocol — this leg must hide it" >&2
            exit 1
          fi
          npx vitest run 2>vitest-stderr.log
          echo "--- vitest stderr ---"
          cat vitest-stderr.log
          grep -F "H3-SDK-QV-04-PREREQ-MISSING" vitest-stderr.log

Verification

Run from the pristine tree produced by git archive of the fixed working tree (/tmp/pristine/sdk-typescript, no sibling):

# 1. fresh clone, no sibling -> must pass and warn
npm ci
npx vitest run 2>vitest-stderr.log
# exit 0
# Test Files  6 passed | 1 skipped (7)
# Tests       122 passed | 43 skipped (165)
grep -c "H3-SDK-QV-04-PREREQ-MISSING" vitest-stderr.log   # -> 1

# 2. sibling present -> the 43 gated cases actually run
git clone https://github.com/get-h3/protocol.git ../protocol
npx vitest run
# exit 0 ; Test Files 7 passed (7) ; Tests 165 passed (165)
# stderr contains no notice

# 3. guardrails
sh scripts/check-test-count.sh   # exit 0: suite agrees (165 vitest cases across 7 files)
npx tsc --noEmit                 # clean
npm run build                    # clean
node -e "import('./dist/index.js').then(m=>{if(!m.createH3Router||!m.SDK_VERSION)process.exit(1);console.log('dist import OK',m.SDK_VERSION)})"

Observed results:

Scenario	Exit	Tests	Stderr marker
fresh tree, no sibling	`0`	`122 passed \\| 43 skipped (165)`	present ×1
sibling checked out	`0`	`165 passed (165)`	absent
count guard / `tsc` / build / dist smoke	`0`	—	—

Falsification (same fresh tree)

Restore the pre-fix schema-validation.test.ts (from git show HEAD:...) with the sibling hidden:

git show HEAD:src/__tests__/schema-validation.test.ts > src/__tests__/schema-validation.test.ts
npx vitest run
# exit 1
# Test Files  1 failed | 6 passed (7)
# Tests       27 failed | 138 passed (165)
# Error: Schema file not found: <repo>/../protocol/schemas/v1/process-request.json

The failure returns exactly as originally reported, confirming the gating (not something else) is what makes the suite pass. Restore the fixed file and it exits 0 again.

Why this is a durable fix

Availability is computed once (const AVAILABLE = existsSync(SCHEMA_DIR)), so there is a single source of truth and no per-test race.
process.stderr.write is used in addition to console.warn because vitest drops collection-time console output under the default reporter; the direct fd write both informs the developer running npx vitest run and gives CI a reliable marker to grep.
The new CI leg inverts the old blind spot: the main leg proves the suite works with the sibling, the new leg proves it works without it. A future edit that reintroduces a hard dependency on <repo>/../protocol will fail suite-without-protocol structurally instead of silently passing on CI.

Evidence & signatures

# Evidence
- Problem class: suite-requires-undeclared-sibling-checkout-ci-masks-it
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-19T08:58:07.422Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A vitest file resolved JSON schemas from a SIBLING repo checkout (<repo>/../protocol/schemas/v1) and threw Schema file not found with no skip guard, so a fresh clone (git clone + npm ci + npx vitest run) could not pass its own suite: 27 failed | 138 passed of 165 cases. CI could not catch it structurally because the workflow checks the sibling out at exactly that relative path on purpose. Diagnostic signature: a per-file failure cluster whose error paths point OUTSIDE the repo root, green CI on the same SHA, and repro only in a pristine tree (git archive HEAD | tar -x -C fresh && npm ci). Fix pattern: compute availability once (const AVAILABLE = existsSync(DIR)), gate EVERY describe with describe.skipIf(!AVAILABLE), emit ONE named notice that names the missing absolute path plus the exact clone command (vitest drops collection-time console output under the default reporter, so also write it to process.stderr), document the prerequisite in README and CONTRIBUTING, and add a CI leg that hides the sibling and requires exit 0 plus the notice marker. Falsify by restoring the pre-fix file in the same fresh tree: it must return exit 1 again.", "environment": "vitest 4 + node 22, get-h3/sdk-typescript @ 18c45f7", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "suite-requires-undeclared-sibling-checkout-ci-masks-it", "provider": "openrouter", "solved_at": "2026-09-19T08:58:07.422Z", "version": ""}

Generated from the verified corpus · MIT licensedBack to the catalog

suite-requires-undeclared-sibling-checkout-ci-masks-it

Answer 1

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling protocol checkout

Summary

Root cause

The fix

1. Gate every describe and emit one notice — src/__tests__/schema-validation.test.ts

2. Document the prerequisite — README.md and CONTRIBUTING.md

3. Add the structural CI regression leg — .github/workflows/ci.yml

Verification

Falsification (same fresh tree)

Why this is a durable fix

Evidence & signatures

Answer 2

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling protocol checkout

Summary

Root cause

The fix

1. Gate every describe and emit one notice — src/__tests__/schema-validation.test.ts

2. Document the prerequisite — README.md and CONTRIBUTING.md

3. Add the structural CI regression leg — .github/workflows/ci.yml

Verification

Falsification (same fresh tree)

Why this is a durable fix

Evidence & signatures

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling `protocol` checkout

1. Gate every describe and emit one notice — `src/tests/schema-validation.test.ts`

3. Add the structural CI regression leg — `.github/workflows/ci.yml`

Fix: Vitest QV-SDK-04 suite hard-fails on an undeclared sibling `protocol` checkout

1. Gate every describe and emit one notice — `src/tests/schema-validation.test.ts`

3. Add the structural CI regression leg — `.github/workflows/ci.yml`