ci-needs-gating-amplifies-unit-flake
Diagnosis complete and verified against the live repo API. Artifacts written:
~/INT-CI-017-solution.md — the full solution~/ci-unit-tests-split.patch — ready-to-apply patch (applies cleanly to 666356d, reproduces fe8e1be exactly)~/verify_ci.py — self-contained structural checker (44/44 pass)~/ci_before.yml, ~/ci_after.yml — fetched reference treesKey findings, independently confirmed from api.github.com per-job data:
| run | build-and-test | unit-tests | gitreins-guard | regression | root-suite | run conclusion |
|---|---|---|---|---|---|---|
35253802213 (defect) |
failure | — | skipped | skipped | skipped | failure |
35264998279 (before) |
failure | — | skipped | skipped | skipped | failure |
35265000613 (after) |
success | failure | success | skipped* | skipped* | failure |
* skipped only by the unchanged if: github.event_name == 'push' && ref == 'refs/heads/main'.
Root cause: build-and-test conflated the compile gate and the assertion layer; regression/gitreins-guard/root-suite all carried needs: build-and-test, so a unit flake propagated as SKIPPED across the fan-out (amplification = 3). GitHub's run-level failure hides skipped jobs.
Fix: move Unit tests into a new unit-tests job (needs: build-and-test, own checkout/setup-go/buf prerequisites, timeout-minutes: 10, no continue-on-error), while the coverage jobs keep needs: build-and-test. The gate stays red (run conclusion failure in both A/B runs); only the coverage cliff is removed.
The markdown contains the full root-cause analysis, DAG before/after, exact YAML + patch, the A/B evidence table, gh run view ... --json jobs reproduction commands, the structural checker, and the transferable rule:
A job that only ASSERTS must never be the
needsancestor of jobs that PRODUCE coverage. Put the compile gate at the root; hang assertion and coverage jobs off it independently; keep the assertion job free ofcontinue-on-error; verify via per-job conclusions, never run-level.
# Evidence - Problem class: ci-needs-gating-amplifies-unit-flake - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T19:34:38.313Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: one flaky unit-test assertion in a single GitHub Actions job silently strips the ENTIRE downstream CI coverage of a push. Measured on a real push: run 35253802213 (commit 58546f5) had `build-and-test` = failure at step `Unit tests`, and the jobs `regression`, `gitreins-guard` and `root-suite` all = SKIPPED. The run-level conclusion is `failure`, but the expensive coverage layers never executed, and nothing in the board/CI record distinguishes that push from a fully covered one.\n\nROOT CAUSE (mechanism, not blame): the workflow had ONE Tier-0 job (`build-and-test`) that did both the compile-surface checks (protobuf generation, go build, CLI smoke, freshness/version/tag authority, go vet) AND the unit tests, and every downstream job carried `needs: build-and-test`. In GitHub Actions a job whose dependency failed is SKIPPED, so the assertion layer (unit tests) was gating the artifact/report layers (regression suite, root-suite, Tier-1 guard). Any flake in the assertion layer therefore cost three jobs, not one. Amplification factor = number of jobs hanging off the same `needs` edge.\n\nFIX (minimal, no weakening): split the assertion layer from the artifact layer instead of raising retries or muting the gate. The `Unit tests` step was removed from `build-and-test` and now runs in its OWN job `unit-tests` with `needs: build-and-test` (its own checkout/setup-go/buf-setup/buf generate prerequisites, `timeout-minutes: 10`). `gitreins-guard`, `regression` and `root-suite` KEEP `needs: build-and-test`, because what they actually need is a COMPILING TREE, not passing assertions. The new job deliberately carries NO `continue-on-error`: a genuine unit-test failure still fails the workflow run (GitHub fails the run when any job fails), so the red gate is intact and only the coverage cliff is gone. Diff: 1 file, +30/-3.\n\nVERIFICATION (A/B, same probe, one commit apart - the falsifiable shape that makes the claim checkable): a temporary test that fails ONLY under `-short` (so it can be told apart from the guard's full-suite run) was pushed on two throwaway branches and the workflow was dispatched on each:\n - BEFORE, base 666356d (pre-fix), run 35264998279 -> build-and-test RED at step `Unit tests`; root-suite, gitreins-guard, regression ALL SKIPPED.\n - AFTER, base fe8e1be (fix), run 35265000613 -> build-and-test SUCCESS; `unit-tests` RED at step `Unit tests`; `gitreins-guard` RAN and PASSED (conclusion success); root-suite/regression skipped only by their pre-existing `if: github.event_name == 'push' && ref == refs/heads/main` condition, which is unchanged by design.\n - Run-level conclusion was `failure` in BOTH runs, which is the proof that the gate did not get weaker while the coverage was restored.\n - Throwaway branches deleted after the runs; no probe code was merged.\n Plus a structural check on the parsed workflow YAML: 27/27 assertions (only one job added; every pre-existing step preserved exactly once; no job gained `continue-on-error`; `if:`/`timeout-minutes`/env unchanged; `on:` triggers unchanged).\n\nTRANSFERABLE RULE: a job that only ASSERTS (tests, lint, coverage) must never be the `needs` ancestor of jobs that PRODUCE coverage (integration/E2E/security suites). Put the compile/build gate at the root of the graph, hang assertion and coverage jobs off it independently, and keep the assertion job free of `continue-on-error` so the workflow still goes red. Verify the fix by dispatching the workflow with a deliberately failing assertion and reading the per-job conclusions (`gh run view <id> --json jobs --jq '.jobs[]|{name,conclusion}'`) - not by trusting the run-level conclusion, which cannot see a skipped job.", "environment": "", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ci-needs-gating-amplifies-unit-flake", "provider": "openrouter", "solved_at": "2026-09-17T19:34:38.313Z", "version": ""}Diagnosis complete and verified against the live repo API. Artifacts written:
~/INT-CI-017-solution.md — the full solution~/ci-unit-tests-split.patch — ready-to-apply patch (applies cleanly to 666356d, reproduces fe8e1be exactly)~/verify_ci.py — self-contained structural checker (44/44 pass)~/ci_before.yml, ~/ci_after.yml — fetched reference treesKey findings, independently confirmed from api.github.com per-job data:
| run | build-and-test | unit-tests | gitreins-guard | regression | root-suite | run conclusion |
|---|---|---|---|---|---|---|
35253802213 (defect) |
failure | — | skipped | skipped | skipped | failure |
35264998279 (before) |
failure | — | skipped | skipped | skipped | failure |
35265000613 (after) |
success | failure | success | skipped* | skipped* | failure |
* skipped only by the unchanged if: github.event_name == 'push' && ref == 'refs/heads/main'.
Root cause: build-and-test conflated the compile gate and the assertion layer; regression/gitreins-guard/root-suite all carried needs: build-and-test, so a unit flake propagated as SKIPPED across the fan-out (amplification = 3). GitHub's run-level failure hides skipped jobs.
Fix: move Unit tests into a new unit-tests job (needs: build-and-test, own checkout/setup-go/buf prerequisites, timeout-minutes: 10, no continue-on-error), while the coverage jobs keep needs: build-and-test. The gate stays red (run conclusion failure in both A/B runs); only the coverage cliff is removed.
The markdown contains the full root-cause analysis, DAG before/after, exact YAML + patch, the A/B evidence table, gh run view ... --json jobs reproduction commands, the structural checker, and the transferable rule:
A job that only ASSERTS must never be the
needsancestor of jobs that PRODUCE coverage. Put the compile gate at the root; hang assertion and coverage jobs off it independently; keep the assertion job free ofcontinue-on-error; verify via per-job conclusions, never run-level.
# Evidence - Problem class: ci-needs-gating-amplifies-unit-flake - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-17T19:34:38.313Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: one flaky unit-test assertion in a single GitHub Actions job silently strips the ENTIRE downstream CI coverage of a push. Measured on a real push: run 35253802213 (commit 58546f5) had `build-and-test` = failure at step `Unit tests`, and the jobs `regression`, `gitreins-guard` and `root-suite` all = SKIPPED. The run-level conclusion is `failure`, but the expensive coverage layers never executed, and nothing in the board/CI record distinguishes that push from a fully covered one.\n\nROOT CAUSE (mechanism, not blame): the workflow had ONE Tier-0 job (`build-and-test`) that did both the compile-surface checks (protobuf generation, go build, CLI smoke, freshness/version/tag authority, go vet) AND the unit tests, and every downstream job carried `needs: build-and-test`. In GitHub Actions a job whose dependency failed is SKIPPED, so the assertion layer (unit tests) was gating the artifact/report layers (regression suite, root-suite, Tier-1 guard). Any flake in the assertion layer therefore cost three jobs, not one. Amplification factor = number of jobs hanging off the same `needs` edge.\n\nFIX (minimal, no weakening): split the assertion layer from the artifact layer instead of raising retries or muting the gate. The `Unit tests` step was removed from `build-and-test` and now runs in its OWN job `unit-tests` with `needs: build-and-test` (its own checkout/setup-go/buf-setup/buf generate prerequisites, `timeout-minutes: 10`). `gitreins-guard`, `regression` and `root-suite` KEEP `needs: build-and-test`, because what they actually need is a COMPILING TREE, not passing assertions. The new job deliberately carries NO `continue-on-error`: a genuine unit-test failure still fails the workflow run (GitHub fails the run when any job fails), so the red gate is intact and only the coverage cliff is gone. Diff: 1 file, +30/-3.\n\nVERIFICATION (A/B, same probe, one commit apart - the falsifiable shape that makes the claim checkable): a temporary test that fails ONLY under `-short` (so it can be told apart from the guard's full-suite run) was pushed on two throwaway branches and the workflow was dispatched on each:\n - BEFORE, base 666356d (pre-fix), run 35264998279 -> build-and-test RED at step `Unit tests`; root-suite, gitreins-guard, regression ALL SKIPPED.\n - AFTER, base fe8e1be (fix), run 35265000613 -> build-and-test SUCCESS; `unit-tests` RED at step `Unit tests`; `gitreins-guard` RAN and PASSED (conclusion success); root-suite/regression skipped only by their pre-existing `if: github.event_name == 'push' && ref == refs/heads/main` condition, which is unchanged by design.\n - Run-level conclusion was `failure` in BOTH runs, which is the proof that the gate did not get weaker while the coverage was restored.\n - Throwaway branches deleted after the runs; no probe code was merged.\n Plus a structural check on the parsed workflow YAML: 27/27 assertions (only one job added; every pre-existing step preserved exactly once; no job gained `continue-on-error`; `if:`/`timeout-minutes`/env unchanged; `on:` triggers unchanged).\n\nTRANSFERABLE RULE: a job that only ASSERTS (tests, lint, coverage) must never be the `needs` ancestor of jobs that PRODUCE coverage (integration/E2E/security suites). Put the compile/build gate at the root of the graph, hang assertion and coverage jobs off it independently, and keep the assertion job free of `continue-on-error` so the workflow still goes red. Verify the fix by dispatching the workflow with a deliberately failing assertion and reading the per-job conclusions (`gh run view <id> --json jobs --jq '.jobs[]|{name,conclusion}'`) - not by trusting the run-level conclusion, which cannot see a skipped job.", "environment": "", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "ci-needs-gating-amplifies-unit-flake", "provider": "openrouter", "solved_at": "2026-09-17T19:34:38.313Z", "version": ""}