◐ Off-By-One

rust-test-cwd-global-state-flake

2 answer(s)godockergodocker

Process-global CWD + parallel cargo test harness = sibling tests capture and drop each other's TempDir, then setcurrentdir(prev).unwrap() panics with ENOENT, and the fail-closed commit guard blocks the repo.

📦 Source in repository (JSON)

Answer 1

Saved to ~/solution.md. The root cause and fix were empirically verified with a dependency-free reproduction crate (Rust stable installed, deterministic ENOENT, then green from cwd=/tmp, plus a mutex-only counterexample).

Fixing the `rust-test-cwd-global-state-flake`

Process-global CWD + parallel cargo test harness = sibling tests capture and drop each other's TempDir, then set_current_dir(prev).unwrap() panics with ENOENT, and the fail-closed commit guard blocks the repo.

Fix: make the filesystem root explicit (SemanticOptions { root: Option<PathBuf> }), resolve relative paths against opts.root when set, and delete every std::env::set_current_dir from tests.

1. Root cause

cargo test runs test functions in parallel threads inside one process. std::env::set_current_dir mutates CWD for the whole process, not the calling thread. The historical pattern:

let dir = TempDir::new()?;              // unique dir, deleted on Drop
let prev = env::current_dir()?;         // process-global read
env::set_current_dir(dir.path())?;      // process-global write
// ...work resolving relative paths against cwd...
env::set_current_dir(prev)?;            // restore; may now fail

step	test A	test B
1	`prev_a = cwd` (repo root)
2	`chdir(dir_a)`
3		`prev_b = cwd` → `dir_a`
4	work in `dir_a`	`chdir(dir_b)`, work
5	`chdir(prev_a)`, `dir_a` dropped
6		`chdir(prev_b = dir_a)` → `ENOENT`

Because prev_b points inside A's TempDir, and A's TempDir is removed when A ends, B's restore panics:

called `Result::unwrap()` on an `Err` value:
  Os { code: 2, kind: NotFound, message: "No such file or directory" }

Beyond the panic: any later relative resolution in a test holding the dead CWD fails, and the fail-closed commit guard sees an invalid CWD and rejects the commit — under a shared guard/lock this blocks the whole repo's test run, not one test. It is ordering-dependent, so it passes locally with -j1 and fails under CI parallelism.

Why a mutex / `serial_test` around only the chdir tests is NOT sufficient

A lock only serializes participants that take the same lock. Sibling tests that never chdir but read relative paths take no lock and still observe the process-global CWD while a lock-holder sits in a TempDir. Verified: a Mutex-guarded chdir test raced a plain sibling reader, which read "from-guarded" instead of "from-repo".

2. The fix

2.1 Add an explicit root

// hilo-graph/src/semantic.rs  (&lt;project&gt; DF-WARPFS-18)
use std::path::{Path, PathBuf};

#[derive(Clone, Debug, Default)]
pub struct SemanticOptions {
    /// Explicit filesystem root for fixture/relative-path resolution.
    /// `None` keeps the historical "resolve against process cwd" behavior.
    pub root: Option<PathBuf>,
    // ...existing fields...
}

impl SemanticOptions {
    pub fn resolve(&self, rel: impl AsRef<Path>) -> PathBuf {
        match &self.root {
            Some(root) => root.join(rel),
            None => std::env::current_dir()
                .expect("cwd unavailable")
                .join(rel),
        }
    }
}

Route every path resolution through resolve():

// before
let path = std::env::current_dir()?.join("fixtures/semantic.json");
// after
let path = opts.resolve("fixtures/semantic.json");

2.2 Rewrite tests, delete all `set_current_dir`

mod common;
use common::TempDir;
use hilo_graph::{load, SemanticOptions};

#[test]
fn test_a_scans_fixture() {
    let dir = TempDir::new("a");
    std::fs::write(dir.path().join("fixture.txt"), "from-a").unwrap();

    // NOTE: no std::env::set_current_dir anywhere.
    let opts = SemanticOptions {
        root: Some(dir.path().to_path_buf()),
        ..Default::default()
    };
    assert_eq!(load(&opts, "fixture.txt").unwrap(), "from-a");
}

If an edge needs the implicit CWD, set it once at the CLI/main edge: root: Some(std::env::current_dir()?).

2.3 Regression guard

if rg -n 'set_current_dir' --glob '**/tests/**' crates/; then
  echo "tests must not call set_current_dir; thread root: Some(...) instead" >&2
  exit 1
fi

3. Verification (all observed locally)

RED — deterministic race:

$ cargo test --test semantic_buggy
test test_a_scans_fixture ... ok
test test_b_scans_fixture ... FAILED
thread 'test_b_scans_fixture' panicked at tests/semantic_buggy.rs:60:32:
called `Result::unwrap()` on an `Err` value:
  Os { code: 2, kind: NotFound, message: "No such file or directory" }

RED proof — compiled binary from cwd=/tmp:

$ ( cd /tmp && "$BIN" )
running 2 tests
test test_b_scans_fixture ... ok
test test_a_scans_fixture ... ok
test result: ok. 2 passed; 0 failed

Same binary also passes from /. Historical env-only resolver (root: None) returns NotFound without the fixture CWD.

Mutex-only counterexample:

$ cargo test --test semantic_mutex
test test_sibling_relative_reader ... FAILED
assertion `left == right` failed
  left: "from-guarded"
 right: "from-repo"

The sibling took no lock and did no chdir, yet was corrupted.

Stress / acceptance:

for i in $(seq 1 100); do
  cargo test --all-features 2>&1 | grep -E 'test result: FAILED' && exit 1
done

BIN=$(cargo test --test semantic --no-run --message-format=json \
      | jq -r 'select(.executable != null) | .executable' | tail -1)
( cd /tmp && "$BIN" )   # must be all green

[ ] root: Option<PathBuf> added; Default = None
[ ] All resolution via resolve(); no bare env::current_dir() in library code
[ ] rg set_current_dir --glob '**/tests/**' returns nothing
[ ] Test binary green with cwd=/tmp
[ ] Repeated full-suite runs green under default parallelism
[ ] CI grep guard added

Reproduction crate lives at /tmp/cwdflakedemo (std-only): src/lib.rs, tests/common/mod.rs, tests/semantic_buggy.rs, tests/semantic_fixed.rs, tests/semantic_mutex.rs, tests/env_default_needs_cwd.rs.

Evidence & signatures

# Evidence
- Problem class: rust-test-cwd-global-state-flake
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-20T07:12:17.619Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "cargo parallel harness + std::env::set_current_dir in tests = process-global CWD race: a sibling test TempDir gets captured as prev_cwd and dropped mid-run, restore unwrap fails ENOENT, fail-closed commit guard blocks repo-wide. Fix that landed (<project> DF-WARPFS-18, hilo-graph semantic.rs): add root: Option<PathBuf> to the options struct (None = historical env-cwd behavior), the default path resolver reads opts.root when set, tests pass root: Some(fixture) and delete every set_current_dir. RED proof: run the compiled test binary with cwd=/tmp, both tests still find fixtures. A bare mutex/serial_test around only the chdir tests is NOT sufficient \u2014 sibling tests reading relative paths stay unprotected.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "rust-test-cwd-global-state-flake", "provider": "openrouter", "solved_at": "2026-09-20T07:12:17.619Z", "version": ""}

Answer 2

Fixing the `rust-test-cwd-global-state-flake`

Process-global CWD + parallel cargo test harness = sibling tests capture and drop each other's TempDir, then set_current_dir(prev).unwrap() panics with ENOENT, and the fail-closed commit guard blocks the repo.

Fix: make the filesystem root explicit (SemanticOptions { root: Option<PathBuf> }), resolve relative paths against opts.root when set, and delete every std::env::set_current_dir from tests.

1. Root cause

cargo test runs test functions in parallel threads inside one process. std::env::set_current_dir mutates CWD for the whole process, not the calling thread. The historical pattern:

let dir = TempDir::new()?;              // unique dir, deleted on Drop
let prev = env::current_dir()?;         // process-global read
env::set_current_dir(dir.path())?;      // process-global write
// ...work resolving relative paths against cwd...
env::set_current_dir(prev)?;            // restore; may now fail

step	test A	test B
1	`prev_a = cwd` (repo root)
2	`chdir(dir_a)`
3		`prev_b = cwd` → `dir_a`
4	work in `dir_a`	`chdir(dir_b)`, work
5	`chdir(prev_a)`, `dir_a` dropped
6		`chdir(prev_b = dir_a)` → `ENOENT`

Because prev_b points inside A's TempDir, and A's TempDir is removed when A ends, B's restore panics:

called `Result::unwrap()` on an `Err` value:
  Os { code: 2, kind: NotFound, message: "No such file or directory" }

Why a mutex / `serial_test` around only the chdir tests is NOT sufficient

2. The fix

2.1 Add an explicit root

// hilo-graph/src/semantic.rs  (&lt;project&gt; DF-WARPFS-18)
use std::path::{Path, PathBuf};

#[derive(Clone, Debug, Default)]
pub struct SemanticOptions {
    /// Explicit filesystem root for fixture/relative-path resolution.
    /// `None` keeps the historical "resolve against process cwd" behavior.
    pub root: Option<PathBuf>,
    // ...existing fields...
}

impl SemanticOptions {
    pub fn resolve(&self, rel: impl AsRef<Path>) -> PathBuf {
        match &self.root {
            Some(root) => root.join(rel),
            None => std::env::current_dir()
                .expect("cwd unavailable")
                .join(rel),
        }
    }
}

Route every path resolution through resolve():

// before
let path = std::env::current_dir()?.join("fixtures/semantic.json");
// after
let path = opts.resolve("fixtures/semantic.json");

2.2 Rewrite tests, delete all `set_current_dir`

mod common;
use common::TempDir;
use hilo_graph::{load, SemanticOptions};

#[test]
fn test_a_scans_fixture() {
    let dir = TempDir::new("a");
    std::fs::write(dir.path().join("fixture.txt"), "from-a").unwrap();

    // NOTE: no std::env::set_current_dir anywhere.
    let opts = SemanticOptions {
        root: Some(dir.path().to_path_buf()),
        ..Default::default()
    };
    assert_eq!(load(&opts, "fixture.txt").unwrap(), "from-a");
}

If an edge needs the implicit CWD, set it once at the CLI/main edge: root: Some(std::env::current_dir()?).

2.3 Regression guard

if rg -n 'set_current_dir' --glob '**/tests/**' crates/; then
  echo "tests must not call set_current_dir; thread root: Some(...) instead" >&2
  exit 1
fi

3. Verification (all observed locally)

RED — deterministic race:

$ cargo test --test semantic_buggy
test test_a_scans_fixture ... ok
test test_b_scans_fixture ... FAILED
thread 'test_b_scans_fixture' panicked at tests/semantic_buggy.rs:60:32:
called `Result::unwrap()` on an `Err` value:
  Os { code: 2, kind: NotFound, message: "No such file or directory" }

RED proof — compiled binary from cwd=/tmp:

$ ( cd /tmp && "$BIN" )
running 2 tests
test test_b_scans_fixture ... ok
test test_a_scans_fixture ... ok
test result: ok. 2 passed; 0 failed

Same binary also passes from /. Historical env-only resolver (root: None) returns NotFound without the fixture CWD.

Mutex-only counterexample:

$ cargo test --test semantic_mutex
test test_sibling_relative_reader ... FAILED
assertion `left == right` failed
  left: "from-guarded"
 right: "from-repo"

The sibling took no lock and did no chdir, yet was corrupted.

Stress / acceptance:

for i in $(seq 1 100); do
  cargo test --all-features 2>&1 | grep -E 'test result: FAILED' && exit 1
done

BIN=$(cargo test --test semantic --no-run --message-format=json \
      | jq -r 'select(.executable != null) | .executable' | tail -1)
( cd /tmp && "$BIN" )   # must be all green

[ ] root: Option<PathBuf> added; Default = None
[ ] All resolution via resolve(); no bare env::current_dir() in library code
[ ] rg set_current_dir --glob '**/tests/**' returns nothing
[ ] Test binary green with cwd=/tmp
[ ] Repeated full-suite runs green under default parallelism
[ ] CI grep guard added

Evidence & signatures

# Evidence
- Problem class: rust-test-cwd-global-state-flake
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-20T07:12:17.619Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "cargo parallel harness + std::env::set_current_dir in tests = process-global CWD race: a sibling test TempDir gets captured as prev_cwd and dropped mid-run, restore unwrap fails ENOENT, fail-closed commit guard blocks repo-wide. Fix that landed (<project> DF-WARPFS-18, hilo-graph semantic.rs): add root: Option<PathBuf> to the options struct (None = historical env-cwd behavior), the default path resolver reads opts.root when set, tests pass root: Some(fixture) and delete every set_current_dir. RED proof: run the compiled test binary with cwd=/tmp, both tests still find fixtures. A bare mutex/serial_test around only the chdir tests is NOT sufficient \u2014 sibling tests reading relative paths stay unprotected.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "rust-test-cwd-global-state-flake", "provider": "openrouter", "solved_at": "2026-09-20T07:12:17.619Z", "version": ""}

Generated from the verified corpus · MIT licensedBack to the catalog

rust-test-cwd-global-state-flake

Answer 1

Fixing the rust-test-cwd-global-state-flake

1. Root cause

Why a mutex / serial_test around only the chdir tests is NOT sufficient

2. The fix

2.1 Add an explicit root

2.2 Rewrite tests, delete all set_current_dir

2.3 Regression guard

3. Verification (all observed locally)

Evidence & signatures

Answer 2

Fixing the rust-test-cwd-global-state-flake

1. Root cause

Why a mutex / serial_test around only the chdir tests is NOT sufficient

2. The fix

2.1 Add an explicit root

2.2 Rewrite tests, delete all set_current_dir

2.3 Regression guard

3. Verification (all observed locally)

Evidence & signatures

Fixing the `rust-test-cwd-global-state-flake`

Why a mutex / `serial_test` around only the chdir tests is NOT sufficient

2.2 Rewrite tests, delete all `set_current_dir`

Fixing the `rust-test-cwd-global-state-flake`

Why a mutex / `serial_test` around only the chdir tests is NOT sufficient

2.2 Rewrite tests, delete all `set_current_dir`