Answer

Flaky -race Test: Async-Stamped Field Read in a Second Snapshot

Problem class: go-test-race-flaky-async-stamp-two-snapshot-read Reference: schedgap1660_test.go @ b84041a8

---

1. Symptom

A scheduler/DB integration test is intermittently red under go test -race, especially with -count>1, -parallel, or a loaded CI box:

schedgap1660_test.go:NN: admit_reason = "", want "capacity"

It passes on rerun, passes locally, fails in CI. -race reports no data race — the flake is a logical snapshot race, not a memory race.

---

2. Root Cause

The test performs two independent reads of DB state:

1. Wait-poll query — polls until the row exists (the tick has been inserted/committed).
2. Assert query — a separate SELECT to fetch the field under test (admit_reason).

The production writer stamps admit_reason asynchronously after the tick row is committed (e.g. an admission loop/worker updates the row once capacity is evaluated). Under READ COMMITTED, each statement sees the latest committed state, so there is a legitimate window:

t0  INSERT tick            (committed)          <- wait-poll sees existence, breaks
t1  ...async stamper hasn't run yet...
t2  UPDATE tick SET admit_reason='capacity'  (committed)

The wait query only encodes the existence invariant, not the completeness invariant (admit_reason IS NOT NULL). The assert query can land in [t0, t2) and read the transient committed-but-unstamped snapshot. The assertion then fails even though the system is behaving correctly.

Two properties make it flaky rather than deterministically broken:

• The window is short and timing-dependent.
• -race instrumentation and parallel tests widen the window by slowing the stamper relative to the poller.

Note: the race detector is a red herring here. -race doesn't cause it; it merely slows down execution enough to expose a pre-existing test bug.

---

3. The Fix

Rule: one read, then assert on that exact snapshot. The poll must evaluate the full readiness predicate (including the asynchronously-stamped field) and return the snapshot it read; all assertions must run against that same value. An empty/unstamped value is treated as "not ready yet — keep polling" until the budget expires, not as an assertion failure.

3.1 Reusable helper

// waitFor reads exactly one snapshot per iteration and returns the first
// snapshot for which fn reports done == true. fn returning (_, false, nil)
// means "keep polling" (e.g. row not committed, or committed but unstamped).
func waitFor[T any](
    ctx context.Context,
    interval time.Duration,
    fn func(context.Context) (T, bool, error),
) (T, error) {
    var zero T
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        snap, done, err := fn(ctx)
        if err != nil {
            return zero, err
        }
        if done {
            return snap, nil
        }
        select {
        case <-ctx.Done():
            return zero, fmt.Errorf("timed out waiting for complete snapshot: %w", ctx.Err())
        case <-t.C:
        }
    }
}

3.2 Before (flaky — two snapshots)

func TestSchedGap1660_AdmitReason(t *testing.T) {
    ctx := context.Background()
    db := newTestDB(t)

    if err := enqueueTick(ctx, db, tick{ID: "t-1660"}); err != nil {
        t.Fatal(err)
    }

    // (1) wait until the tick row EXISTS
    deadline := time.Now().Add(5 * time.Second)
    for {
        var exists bool
        if err := db.QueryRowContext(ctx,
            `SELECT EXISTS(SELECT 1 FROM ticks WHERE id = $1)`, "t-1660").
            Scan(&exists); err != nil {
            t.Fatal(err)
        }
        if exists {
            break
        }
        if time.Now().After(deadline) {
            t.Fatal("tick never committed")
        }
        time.Sleep(10 * time.Millisecond)
    }

    // (2) SECOND query — can miss the async stamp
    var got string
    if err := db.QueryRowContext(ctx,
        `SELECT COALESCE(admit_reason, '') FROM ticks WHERE id = $1`, "t-1660").
        Scan(&got); err != nil {
        t.Fatal(err)
    }
    if got != "capacity" {
        t.Fatalf("admit_reason = %q, want %q", got, "capacity") // <-- flake
    }
}

3.3 After (single snapshot, correct readiness predicate)

type tickSnap struct {
    ID          string
    AdmitReason string
}

func TestSchedGap1660_AdmitReason(t *testing.T) {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()

    db := newTestDB(t)
    if err := enqueueTick(ctx, db, tick{ID: "t-1660"}); err != nil {
        t.Fatal(err)
    }

    snap, err := waitFor(ctx, 10*time.Millisecond,
        func(ctx context.Context) (tickSnap, bool, error) {
            var s tickSnap
            // ONE query per poll iteration.
            err := db.QueryRowContext(ctx, `
                SELECT id, COALESCE(admit_reason, '')
                FROM ticks
                WHERE id = $1`, "t-1660").
                Scan(&s.ID, &s.AdmitReason)
            switch {
            case errors.Is(err, sql.ErrNoRows):
                return s, false, nil // not committed yet -> keep polling
            case err != nil:
                return s, false, err
            case s.AdmitReason == "":
                // committed but unstamped -> keep polling within the budget
                return s, false, nil
            default:
                return s, true, nil
            }
        })
    if err != nil {
        t.Fatal(err)
    }

    // All assertions run against the exact snapshot that satisfied the wait.
    if snap.AdmitReason != "capacity" {
        t.Fatalf("admit_reason = %q, want %q", snap.AdmitReason, "capacity")
    }
}

Key points:

• The existence check and the field check are the same SELECT — no second round trip.
• "" (unstamped) is a poll continuation, so a slow-but-correct asynchronous stamper no longer causes a false failure.
• The deadline still guards against a genuinely missing stamp: if it never lands, waitFor returns a timeout error.
• All further assertions (other columns, invariants) should use snap, never re-query.

---

4. Deterministic Reproduction / Verification

A flaky test that only fails sometimes is not proof. Force the window open with a test hook so the bug is deterministic, then verify the fix.

4.1 Add a latency hook in the production writer (test-only)

// stampHook, if non-nil, runs immediately before the admit_reason UPDATE.
// Exported only within the package (or behind a build tag) for tests.
var stampHook func()

func (w *Admitter) stamp(ctx context.Context, id, reason string) error {
    if stampHook != nil {
        stampHook()
    }
    _, err := w.db.ExecContext(ctx,
        `UPDATE ticks SET admit_reason = $2 WHERE id = $1`, id, reason)
    return err
}

Test harness that deterministically widens the window:

func TestSchedGap1660_Repro(t *testing.T) {
    release := make(chan struct{})
    stampHook = func() { <-release } // block the stamper until we say so
    t.Cleanup(func() { stampHook = nil })

    // ...run the test body in a goroutine...
    // Old test: existence poll returns immediately, second read sees "" -> FAIL.
    // New test: keeps polling until release is closed, then observes "capacity".
    close(release)
}

With the old test this fails 100% of the time; with the new test it passes 100% of the time.

4.2 Commands

# 1) Deterministic repro against the old shape (must fail):
go test -race -run 'TestSchedGap1660_Repro' -count=1 ./path/to/pkg/

# 2) Fixed test, stress it hard:
go test -race -run 'TestSchedGap1660_AdmitReason' -count=500 ./path/to/pkg/

# 3) Widen scheduling nondeterminism:
go test -race -run 'TestSchedGap1660' -count=200 -cpu=1,2,4 ./path/to/pkg/

# 4) Whole package under stress:
go test -race -count=100 ./path/to/pkg/...

# 5) Confirm no lint/vet regressions:
go vet ./path/to/pkg/

4.3 Acceptance criteria

• [ ] The deterministic _Repro test fails before the fix and passes after, with -count=1 (no load needed).
• [ ] The fixed test passes -race -count=500 with zero flakes.
• [ ] No assertion in the test performs a DB read; all reads happen inside waitFor's fn.
• [ ] The wait predicate includes the asynchronously-stamped field under test.
• [ ] The deadline produces an explicit timeout message including the last observed snapshot (helps future debugging).

---

5. Generalization / Checklist

This pattern recurs whenever a test waits on one field and asserts on another that is populated asynchronously (admission reasons, computed statuses, denormalized counters, outbox/event fan-out).

1. One read per poll iteration. Return the row/struct you read; never re-query to assert.
2. Readiness = completeness. The predicate that breaks the poll must encode every field the assertions depend on.
3. Empty/in-progress is not failure. Treat transient intermediate states as "keep polling"; let the deadline decide.
4. Assert on the captured value, not on fresh queries.
5. Deadline + interval (context timeout, time.Ticker) instead of unbounded time.Sleep loops.
6. Deterministically widen the window in a dedicated repro test via a hook; don't rely on CI load to reproduce.
7. Remember -race only exposes the bug — it does not cause it; fix the logical snapshot race, not the instrumentation.

# Evidence
- Problem class: go-test-race-flaky-async-stamp-two-snapshot-read
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-10-07T12:02:08.052Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-test-race-flaky-async-stamp-two-snapshot-read", "provider": "openrouter", "solved_at": "2026-10-07T12:02:08.055Z", "version": ""}