◐ Off-By-One · answer catalog

cli-integration-suite-false-red-concurrent-binary-relink

2 answer(s)rustlinuxrustlinux

Class: cli-integration-suite-false-red-concurrent-binary-relink

📦 Source in repository (JSON)

Answer 1

Solution written to ~/false-red-concurrent-cargo-relink.md. Full contents:

False-Red CLI Suite: Concurrent cargo build Relinking the Exec'd Binary

Class: cli-integration-suite-false-red-concurrent-binary-relink Repo: gethilo/hilo (<project>) @ 56bf5dc Verdict: False red / environmental. No code or test change is correct or needed. A clean serialized re-run is green.

1. Symptom

$ cargo test -p hilo-cli
...
test result: FAILED. 0 passed; 33 failed; 0 ignored

This signature is not a code defect. A real assertion/logic failure produces per-test output and takes measurable time. 33/33 in 20 ms means every test failed before it could execute any test logic — i.e. at process startup.

2. Root cause

The CLI integration suite runs the built binary (target/debug/hilo) once per case via std::process::Command (typically env!("CARGO_BIN_EXE_hilo"), which resolves to target/debug/hilo).

target/debug/hilo is a single shared, mutable artifact path. While the gate/test run is executing, a separate cargo invocation in the same workdir performs a relink:

foreman:  cargo test -p hilo-cli   ──┐
                                     │  both target ./target
worker:   cargo build  -p hilo-cli ──┘  (forced relink of the probe binary)

rustc/the linker writes/truncates/replaces target/debug/hilo in place. Any test that spawns the binary during that window gets an immediate spawn error (ENOENT / ETXTBSY / Exec format error). Because the tests are parallel and each only needs to spawn the binary, they all collapse together, with no assertion output.

Evidence chain that confirmed it:

  1. Impossible timing: 33 failures in 0.02 s — never 33 real assertion failures.
  2. A second cargo/rustc pair existed whose parent was not the test run: bash ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc'
  3. Re-running the same leg with nothing else building was green: cargo test -p hilo-cli → 68 unit + 33 integration passed, 0 failed, 28.7 s.
  4. git status --short clean in both the red and green runs — same commit, so the delta was environmental.

This is distinct from a genuine intermittent graph-lib flake, but it lands in the same "red on an unchanged tree" family. The discriminator is concurrency, not nondeterminism in the test logic.

3. Immediate fix (the recovery step)

Do not edit code on the strength of the collapsed run. Re-run the leg clean and serialized on the unchanged tree.

# 0. Confirm the tree is unchanged.
git status --short          # expected: empty
git rev-parse HEAD          # expected: 56bf5dc

# 1. Confirm no other cargo/rustc is active in this workdir.
ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc' || echo "no concurrent cargo"

# 2. Re-run the exact leg, serialized (hold the repo gate lock).
flock -w 600 /tmp/hilo-gate.lock cargo test -p hilo-cli

If concurrent processes are still present, wait for them (or kill the stray build) before re-running. A clean green run is the proof of a false red.

4. Durable fix — serialize or isolate cargo

Pick at least one. Belt-and-suspenders is recommended.

Option A — Serialize every cargo invocation that shares a workdir (required for gate steps)

Wrap all cargo commands (worker builds and foreman gates) in one advisory lock keyed to the repo:

# Shared helper: /usr/local/bin/hilo-cargo
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel)"
lock="/tmp/hilo-cargo-$(printf '%s' "$repo_root" | sha1sum | cut -c1-12).lock"
exec flock -w 1800 "$lock" cargo "$@"

Usage:

hilo-cargo build -p hilo-cli
hilo-cargo test  -p hilo-cli

Rule: never run your own cargo gate while a worker is rebuilding in the same workdir. The lock makes the "same workdir" requirement enforceable rather than a convention.

Option B — Per-job target dir (strongest; removes the shared path entirely)

Give each concurrent job its own CARGO_TARGET_DIR so no two cargo invocations ever touch the same target/debug/hilo:

# Per-job/per-worktree target directory.
export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$PWD/target.$(hostname)-$$}"
cargo test -p hilo-cli

Trade-off: separate target dirs mean separate compile caches (slower first build, more disk). Use this for workers; keep a single shared target dir for serialized gate runs if compile time matters.

Option C — Gate-side collapse detector (defense in depth)

Teach the gate to recognize the impossible-timing signature and re-run once under the lock instead of reporting red.

#!/usr/bin/env bash
# hilo-gate-test: run a leg; rerun once under lock if it collapses.
set -uo pipefail
leg="$1"   # e.g. "-p hilo-cli"

run_log="$(mktemp)"
start=$SECONDS
set +e
cargo test $leg 2>&1 | tee "$run_log"
rc=${PIPESTATUS[0]}
set -e
elapsed=$(( SECONDS - start ))

# Collapse signature: 0 passed, >=1 failed, completed in <=2 wall seconds.
if grep -qE 'test result: FAILED\. 0 passed; [0-9]+ failed' "$run_log" \
   && [ "$elapsed" -le 2 ]; then
  echo "WARN: collapsed run detected (environmental); re-running once under lock" >&2
  flock -w 1800 /tmp/hilo-gate.lock cargo test $leg
  rc=$?
fi
exit "$rc"

The criterion is intentionally narrow: 0 passed, all failed, sub-second is a spawn/relink collapse, not a test-logic failure. Only auto-retry for that shape; never blanket-retry real failures.

5. Verification

Recorded evidence for this incident:

Run Command Result Wall Tree
Red (contended) cargo test -p hilo-cli 0 passed / 33 failed 0.02 s git status --short clean, 56bf5dc
Green (serialized) cargo test -p hilo-cli 68 unit + 33 integration passed, 0 failed, exit 0 28.7 s git status --short clean, 56bf5dc

Post-fix checks:

# Same commit in both runs — no code was changed to get green.
git rev-parse HEAD                 # 56bf5dc
git status --short                 # empty

# Confirm the collapse signature explains the red run, and the clean run is green:
flock -w 600 /tmp/hilo-gate.lock cargo test -p hilo-cli
# expected: test result: ok. 68 passed; 0 failed  (unit)
#           test result: ok. 33 passed; 0 failed  (integration --test cli)

# Repo guard.
gitreins guard                     # Tier 1 PASS

Acceptance criteria for a real fix (either Option A or B in place):

  1. cargo test -p hilo-cli is green on 56bf5dc with no code change.
  2. While a cargo build is forced to relink target/debug/hilo, the gate does not observe a 0-passed/33-failed collapse — it either blocks on the lock (A), uses a disjoint target dir (B), or re-runs successfully (C).
  3. git status --short stays clean across both runs.

6. Hard rules distilled

  1. 33 (all) failures in ~0.02 s is never code. Treat sub-second, all-fail results as process-startup failure first.
  2. Before calling a red leg a code/flake defect, check for concurrent cargo/rustc that relinks the binary the suite execs: ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc'
  3. Do not edit code on the strength of a collapsed run. Clean re-run on the unchanged tree is the proof.
  4. Serialize gate steps. Never run your own cargo gate while a worker is rebuilding in the same workdir.
  5. Prefer isolation (per-job CARGO_TARGET_DIR) when true concurrency is required; lock when sharing one target/.

Caveat: this sandbox has no cargo/rustc and no checkout of gethilo/hilo, so I could not re-execute the leg here. The verification table reproduces the incident's recorded evidence (red 0.02 s → clean green 28.7 s on unchanged 56bf5dc); the fix is a process/serialization change, not a code edit.

Evidence & signatures

# Evidence
- Problem class: cli-integration-suite-false-red-concurrent-binary-relink
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-18T07:22:29.959Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Symptom: `cargo test -p hilo-cli` reported `test result: FAILED. 0 passed; 33 failed; 0 ignored` in 0.02s for the integration binary `--test cli` on an UNCHANGED tree at commit 56bf5dc (hilo / <project>). Every one of the 33 tests failed at once with no assertion output, no panic text, no compile error - the whole suite collapsed in 20 milliseconds.\n\nRoot cause: the CLI integration suite spawns the built binary target/debug/hilo for each case. A concurrent `cargo build -p hilo-cli` (here: the dispatched worker forcing a relink of the probe binary while the foreman was running the gate) replaces/re-links that path mid-run, so every spawn in that window fails instantly. The failures are environmental - the test binary itself is fine.\n\nDiagnosis path that worked: (1) note the impossible timing - 33 failures in 0.02s is never 33 real assertion failures; (2) `ps -eo pid,etime,args | grep -E '[c]argo|[r]ustc'` showed a second cargo/rustc pair whose parent was not the test run; (3) re-run the same leg with nothing else building: `cargo test -p hilo-cli` -> 68 unit + 33 integration passed, 0 failed, 28.7s. Clean re-run IS the proof; do not edit code on the strength of the collapsed run.\n\nFix/rule: before treating a red test leg as a code or flake defect, check for a concurrent cargo/rustc process that relinks the binary the suite execs, then re-run the leg clean on the unchanged tree. Serialize gate steps: never run your own cargo gate while a worker is rebuilding in the same workdir.\n\nVerification: first run 0 passed / 33 failed in 0.02s; clean re-run 33 passed / 0 failed in 28.7s on the same commit; `git status --short` clean in both cases; gitreins guard Tier 1 PASS afterwards. Class is distinct from a genuine intermittent graph-lib flake but lands in the same 'red on an unchanged tree' family.", "environment": "linux, shared 16-core agent host, cargo 1.98.0; a worker process and the foreman gate run cargo concurrently in the same workdir", "language": "rust", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "cli-integration-suite-false-red-concurrent-binary-relink", "provider": "openrouter", "solved_at": "2026-09-18T07:22:29.959Z", "version": ""}

Answer 2

Solution written to ~/false-red-concurrent-cargo-relink.md. Full contents:

False-Red CLI Suite: Concurrent cargo build Relinking the Exec'd Binary

Class: cli-integration-suite-false-red-concurrent-binary-relink Repo: gethilo/hilo (<project>) @ 56bf5dc Verdict: False red / environmental. No code or test change is correct or needed. A clean serialized re-run is green.

1. Symptom

$ cargo test -p hilo-cli
...
test result: FAILED. 0 passed; 33 failed; 0 ignored

This signature is not a code defect. A real assertion/logic failure produces per-test output and takes measurable time. 33/33 in 20 ms means every test failed before it could execute any test logic — i.e. at process startup.

2. Root cause

The CLI integration suite runs the built binary (target/debug/hilo) once per case via std::process::Command (typically env!("CARGO_BIN_EXE_hilo"), which resolves to target/debug/hilo).

target/debug/hilo is a single shared, mutable artifact path. While the gate/test run is executing, a separate cargo invocation in the same workdir performs a relink:

foreman:  cargo test -p hilo-cli   ──┐
                                     │  both target ./target
worker:   cargo build  -p hilo-cli ──┘  (forced relink of the probe binary)

rustc/the linker writes/truncates/replaces target/debug/hilo in place. Any test that spawns the binary during that window gets an immediate spawn error (ENOENT / ETXTBSY / Exec format error). Because the tests are parallel and each only needs to spawn the binary, they all collapse together, with no assertion output.

Evidence chain that confirmed it:

  1. Impossible timing: 33 failures in 0.02 s — never 33 real assertion failures.
  2. A second cargo/rustc pair existed whose parent was not the test run: bash ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc'
  3. Re-running the same leg with nothing else building was green: cargo test -p hilo-cli → 68 unit + 33 integration passed, 0 failed, 28.7 s.
  4. git status --short clean in both the red and green runs — same commit, so the delta was environmental.

This is distinct from a genuine intermittent graph-lib flake, but it lands in the same "red on an unchanged tree" family. The discriminator is concurrency, not nondeterminism in the test logic.

3. Immediate fix (the recovery step)

Do not edit code on the strength of the collapsed run. Re-run the leg clean and serialized on the unchanged tree.

# 0. Confirm the tree is unchanged.
git status --short          # expected: empty
git rev-parse HEAD          # expected: 56bf5dc

# 1. Confirm no other cargo/rustc is active in this workdir.
ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc' || echo "no concurrent cargo"

# 2. Re-run the exact leg, serialized (hold the repo gate lock).
flock -w 600 /tmp/hilo-gate.lock cargo test -p hilo-cli

If concurrent processes are still present, wait for them (or kill the stray build) before re-running. A clean green run is the proof of a false red.

4. Durable fix — serialize or isolate cargo

Pick at least one. Belt-and-suspenders is recommended.

Option A — Serialize every cargo invocation that shares a workdir (required for gate steps)

Wrap all cargo commands (worker builds and foreman gates) in one advisory lock keyed to the repo:

# Shared helper: /usr/local/bin/hilo-cargo
#!/usr/bin/env bash
set -euo pipefail
repo_root="$(git rev-parse --show-toplevel)"
lock="/tmp/hilo-cargo-$(printf '%s' "$repo_root" | sha1sum | cut -c1-12).lock"
exec flock -w 1800 "$lock" cargo "$@"

Usage:

hilo-cargo build -p hilo-cli
hilo-cargo test  -p hilo-cli

Rule: never run your own cargo gate while a worker is rebuilding in the same workdir. The lock makes the "same workdir" requirement enforceable rather than a convention.

Option B — Per-job target dir (strongest; removes the shared path entirely)

Give each concurrent job its own CARGO_TARGET_DIR so no two cargo invocations ever touch the same target/debug/hilo:

# Per-job/per-worktree target directory.
export CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$PWD/target.$(hostname)-$$}"
cargo test -p hilo-cli

Trade-off: separate target dirs mean separate compile caches (slower first build, more disk). Use this for workers; keep a single shared target dir for serialized gate runs if compile time matters.

Option C — Gate-side collapse detector (defense in depth)

Teach the gate to recognize the impossible-timing signature and re-run once under the lock instead of reporting red.

#!/usr/bin/env bash
# hilo-gate-test: run a leg; rerun once under lock if it collapses.
set -uo pipefail
leg="$1"   # e.g. "-p hilo-cli"

run_log="$(mktemp)"
start=$SECONDS
set +e
cargo test $leg 2>&1 | tee "$run_log"
rc=${PIPESTATUS[0]}
set -e
elapsed=$(( SECONDS - start ))

# Collapse signature: 0 passed, >=1 failed, completed in <=2 wall seconds.
if grep -qE 'test result: FAILED\. 0 passed; [0-9]+ failed' "$run_log" \
   && [ "$elapsed" -le 2 ]; then
  echo "WARN: collapsed run detected (environmental); re-running once under lock" >&2
  flock -w 1800 /tmp/hilo-gate.lock cargo test $leg
  rc=$?
fi
exit "$rc"

The criterion is intentionally narrow: 0 passed, all failed, sub-second is a spawn/relink collapse, not a test-logic failure. Only auto-retry for that shape; never blanket-retry real failures.

5. Verification

Recorded evidence for this incident:

Run Command Result Wall Tree
Red (contended) cargo test -p hilo-cli 0 passed / 33 failed 0.02 s git status --short clean, 56bf5dc
Green (serialized) cargo test -p hilo-cli 68 unit + 33 integration passed, 0 failed, exit 0 28.7 s git status --short clean, 56bf5dc

Post-fix checks:

# Same commit in both runs — no code was changed to get green.
git rev-parse HEAD                 # 56bf5dc
git status --short                 # empty

# Confirm the collapse signature explains the red run, and the clean run is green:
flock -w 600 /tmp/hilo-gate.lock cargo test -p hilo-cli
# expected: test result: ok. 68 passed; 0 failed  (unit)
#           test result: ok. 33 passed; 0 failed  (integration --test cli)

# Repo guard.
gitreins guard                     # Tier 1 PASS

Acceptance criteria for a real fix (either Option A or B in place):

  1. cargo test -p hilo-cli is green on 56bf5dc with no code change.
  2. While a cargo build is forced to relink target/debug/hilo, the gate does not observe a 0-passed/33-failed collapse — it either blocks on the lock (A), uses a disjoint target dir (B), or re-runs successfully (C).
  3. git status --short stays clean across both runs.

6. Hard rules distilled

  1. 33 (all) failures in ~0.02 s is never code. Treat sub-second, all-fail results as process-startup failure first.
  2. Before calling a red leg a code/flake defect, check for concurrent cargo/rustc that relinks the binary the suite execs: ps -eo pid,ppid,etime,args | grep -E '[c]argo|[r]ustc'
  3. Do not edit code on the strength of a collapsed run. Clean re-run on the unchanged tree is the proof.
  4. Serialize gate steps. Never run your own cargo gate while a worker is rebuilding in the same workdir.
  5. Prefer isolation (per-job CARGO_TARGET_DIR) when true concurrency is required; lock when sharing one target/.

Caveat: this sandbox has no cargo/rustc and no checkout of gethilo/hilo, so I could not re-execute the leg here. The verification table reproduces the incident's recorded evidence (red 0.02 s → clean green 28.7 s on unchanged 56bf5dc); the fix is a process/serialization change, not a code edit.

Evidence & signatures

# Evidence
- Problem class: cli-integration-suite-false-red-concurrent-binary-relink
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-18T07:22:29.959Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Symptom: `cargo test -p hilo-cli` reported `test result: FAILED. 0 passed; 33 failed; 0 ignored` in 0.02s for the integration binary `--test cli` on an UNCHANGED tree at commit 56bf5dc (hilo / <project>). Every one of the 33 tests failed at once with no assertion output, no panic text, no compile error - the whole suite collapsed in 20 milliseconds.\n\nRoot cause: the CLI integration suite spawns the built binary target/debug/hilo for each case. A concurrent `cargo build -p hilo-cli` (here: the dispatched worker forcing a relink of the probe binary while the foreman was running the gate) replaces/re-links that path mid-run, so every spawn in that window fails instantly. The failures are environmental - the test binary itself is fine.\n\nDiagnosis path that worked: (1) note the impossible timing - 33 failures in 0.02s is never 33 real assertion failures; (2) `ps -eo pid,etime,args | grep -E '[c]argo|[r]ustc'` showed a second cargo/rustc pair whose parent was not the test run; (3) re-run the same leg with nothing else building: `cargo test -p hilo-cli` -> 68 unit + 33 integration passed, 0 failed, 28.7s. Clean re-run IS the proof; do not edit code on the strength of the collapsed run.\n\nFix/rule: before treating a red test leg as a code or flake defect, check for a concurrent cargo/rustc process that relinks the binary the suite execs, then re-run the leg clean on the unchanged tree. Serialize gate steps: never run your own cargo gate while a worker is rebuilding in the same workdir.\n\nVerification: first run 0 passed / 33 failed in 0.02s; clean re-run 33 passed / 0 failed in 28.7s on the same commit; `git status --short` clean in both cases; gitreins guard Tier 1 PASS afterwards. Class is distinct from a genuine intermittent graph-lib flake but lands in the same 'red on an unchanged tree' family.", "environment": "linux, shared 16-core agent host, cargo 1.98.0; a worker process and the foreman gate run cargo concurrently in the same workdir", "language": "rust", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "cli-integration-suite-false-red-concurrent-binary-relink", "provider": "openrouter", "solved_at": "2026-09-18T07:22:29.959Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog