ci-go-test-per-package-timeout-edge

Answer

Diagnosis chain (why it was a timeout, not a hang or a regression):

1. Board-only commit — git diff HEAD^ HEAD --stat on the failing run showed only tasks.md touched. No code delta → code regression is impossible; the red must be environmental or config.
2. Exact 60.008s — Go's -timeout=60s fires at 60s and the test binary gets a few ms of scheduler/teardown slack before GitHub marks the step failed. The log tail shows the canonical signature: panic: test timed out after 1m0s followed by FAIL internal/db 60.008s / FAIL internal/handler 60.008s. A genuine hang (deadlock, select{}, blocked network read) does not stop at exactly the budget — it gets killed because of the budget. A suite that finishes in ~55–59s on a fast runner crossing 60s on a slow runner stops at exactly 60.008s.
3. Workflow unchanged since last 3 green runs — no config drift; the variable is runner speed variance on internal/db + internal/handler (PG spin-up, migrations, integration fixtures — the two slowest suites).
4. Root cause — -timeout in go test ./... is applied per test binary (per package), not to the whole run. The budget had zero headroom above the slowest suites' real duration on shared GitHub runners.

The fix — raise the per-package budget in the Test (short) step from 60s → 300s, matching the local make test guard, keeping the job-level timeout-minutes as the total-run watchdog (still fails fast on a real hang).

.github/workflows/ci.yml (before):

  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache: true

      - name: Start PostgreSQL
        run: |
          docker run -d --name pg -p 5432:5432 \
            -e POSTGRES_PASSWORD=test postgres:16

      - name: Test (short)
        run: go test -count=1 -timeout=60s ./...

After:

      - name: Test (short)
        # -timeout is PER-PACKAGE, not per-run: go test ./... applies the full
        # budget to each test binary, and packages run in parallel (-p defaults
        # to GOMAXPROCS), so internal/db + internal/handler (PG integration)
        # contended for CPU and sat right at the 60s edge on variable-speed
        # GitHub runners. 300s matches the local `make test` guard budget and
        # still kills a genuine hang; timeout-minutes: 20 caps the whole job.
        run: go test -count=1 -timeout=300s ./...

Supporting guard in Makefile (so local and CI budgets can't drift again):

# Local guard budget — CI's Test (short) step must stay >= this value.
test:
    go test -count=1 -timeout=300s ./...

Rules this encodes for the future: - Headroom rule: CI per-package -timeout ≥ 5× the slowest suite's observed CI duration (measured from green runs) and ≥ the local budget — whichever is larger. Observed 55s → 300s. - Fail-fast on real hangs: a deadlocked test consumes the whole 300s per package, and the job-level timeout-minutes: 20 bounds the total; the 2m54s actual run means the raised budget has zero wall-clock cost. - Rejected alternatives (each considered and dismissed): retrying flaky tests (actions/github-script retry wrapper) — wrong tool, nothing is flaky; splitting the PG suites into a separate job — more moving parts, the whole run is 3 minutes anyway; -parallel 1 — reduces contention but changes test semantics and hides the real fix.