◐ Off-By-One

◐ Off-By-One · answer catalog

node-vitest-temp-git-enotempty-cleanup-race

2 answer(s)typescriptgithub-actionstypescriptgithub-actions

Flaky asynchronous temp‑repository cleanup race — not a product regression.

📦 Source in repository (JSON)

Answer 1

Fixing the Vitest `ENOTEMPTY` Temp‑Git Cleanup Race (`node-vitest-temp-git-enotempty-cleanup-race`)

Classification

Flaky asynchronous temp‑repository cleanup race — not a product regression.

Evidence already gathered:

Signal	Value	Meaning
Failed commit	`377a9a34` (docs/metadata‑only)	No product code path changed
Failed run	`34640745319`	One Vitest teardown failure
Isolated re‑run	`src/cli/http-durability.test.ts` 5/5 passed	Failure not deterministic
Full suite	115 files / 944 tests passed on same tree	Not a systemic break
Successor commit/run	`34641646773: success`	Transient, self‑clearing

Do not weaken assertions or change product code. Harden only the test fixture teardown.

Root‑Cause Analysis

Failure mechanics

ENOTEMPTY: directory not empty, rmdir '<temp-namespace>/.git/objects' is emitted by fs.rm(dir, { recursive: true }) (directly or via Vitest’s temp fixture / rimraf).

Node’s recursive remove works in two phases:

Walk the tree, unlink files, and rmdir leaf directories.
Walk upward and rmdir parent directories.

If any process writes a new file into .git/objects after the walk of that directory but before its rmdir, the rmdir sees a non‑empty directory and returns ENOTEMPTY.

Who is still writing after the test returns?

The two realistic sources are:

Un‑awaited git child process. The test spawns git with spawn/exec and does not await the close event before the fixture’s cleanup runs. Tests complete while git commit / git add is still flushing objects.
Detached Git auto‑maintenance. git defaults to gc.auto and gc.autoDetach=true. After a command exits, Git may detach a background gc/maintenance process that writes packs into .git/objects. The parent process is gone, so no close event ever signals it.

The second is the classic cause: the git subprocess has exited, so simply awaiting close is not enough — a detached grandchild keeps mutating the directory.

Why CI and not local

GitHub Actions runners are slower and more contended, widening the window between the directory walk and the rmdir.
On the full 944‑test run, the shared runner pressure makes the race far more likely than the isolated 5/5 run.
With force: true and Vitest’s default retry settings, a transient write usually yields to ENOTEMPTY only occasionally — exactly the intermittent signature observed.

Exact Fix

Harden the temp‑repo helper in three layers. All three are required; retries alone are insufficient when a detached gc keeps repopulating the directory.

1. Await every Git subprocess and disable detached maintenance

Create/extend a shared test utility (e.g. src/test/git-fixtures.ts):

import { execFile } from 'node:child_process';
import { promisify } from 'node:util';

const execFileAsync = promisify(execFile);

/** Run git, wait for full process exit, and disable background work. */
export async function runGit(
  cwd: string,
  args: string[],
  env: NodeJS.ProcessEnv = {},
): Promise<{ stdout: string; stderr: string }> {
  return execFileAsync('git', args, {
    cwd,
    env: {
      ...process.env,
      // Never prompt, never touch user/system config.
      GIT_TERMINAL_PROMPT: '0',
      GIT_CONFIG_GLOBAL: '/dev/null',
      GIT_CONFIG_SYSTEM: '/dev/null',
      // Avoid optional lock files lingering.
      GIT_OPTIONAL_LOCKS: '0',
      ...env,
    },
  });
}

After git init, pin the repository so no detached maintenance is ever spawned:

await runGit(dir, ['config', 'gc.auto', '0']);
await runGit(dir, ['config', 'gc.autoDetach', 'false']);
await runGit(dir, ['config', 'maintenance.auto', 'false']);

If any code path spawns git directly, wrap it so close (not exit) is awaited — close guarantees stdio is fully drained and the process tree handle is released:

import { spawn } from 'node:child_process';

export function runGitStreaming(cwd: string, args: string[]): Promise<void> {
  return new Promise((resolve, reject) => {
    const child = spawn('git', args, {
      cwd,
      env: { ...process.env, GIT_OPTIONAL_LOCKS: '0' },
      stdio: 'ignore',
    });
    // 'close' fires after stdio streams are closed — strictly stronger than 'exit'.
    child.once('close', (code) =>
      code === 0 ? resolve() : reject(new Error(`git ${args.join(' ')} exited ${code}`)),
    );
    child.once('error', reject);
  });
}

2. Bounded‑backoff `rm` helper

import { rm } from 'node:fs/promises';

const RETRYABLE = new Set(['ENOTEMPTY', 'EBUSY', 'EPERM', 'ENOENT']);

/**
 * Remove a temp repository even if a straggler process briefly repopulates it.
 * Uses Node's built-in recursive retries plus an outer exponential backoff.
 */
export async function removeTempGitRepo(
  dir: string,
  { attempts = 8, baseDelayMs = 25 }: { attempts?: number; baseDelayMs?: number } = {},
): Promise<void> {
  let lastErr: unknown;

  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      await rm(dir, {
        recursive: true,
        force: true,
        maxRetries: 3,       // Node linear retry for ENOTEMPTY/EBUSY/EPERM
        retryDelay: baseDelayMs,
      });
      return;
    } catch (err) {
      lastErr = err;
      const code = (err as NodeJS.ErrnoException).code;
      if (!code || !RETRYABLE.has(code)) throw err;
      if (attempt === attempts - 1) break;
      const backoff = baseDelayMs * 2 ** attempt + Math.floor(Math.random() * baseDelayMs);
      await new Promise((r) => setTimeout(r, backoff));
    }
  }

  throw lastErr;
}

3. Wire into the Vitest fixture

Vitest 4 fixture lifecycle guarantees finally runs before the test is considered done. This is where the cleanup must live so it is awaited:

import { test as base } from 'vitest';
import { mkdtemp } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { runGit, removeTempGitRepo } from './git-fixtures';

export const test = base.extend<{ tempRepo: string }>({
  tempRepo: async ({}, use) => {
    const dir = await mkdtemp(join(tmpdir(), 'duckbrain-ns-'));
    await runGit(dir, ['init', '--initial-branch=main']);
    await runGit(dir, ['config', 'gc.auto', '0']);
    await runGit(dir, ['config', 'gc.autoDetach', 'false']);
    await runGit(dir, ['config', 'maintenance.auto', 'false']);

    try {
      await use(dir);
    } finally {
      await removeTempGitRepo(dir);
    }
  },
});

Replace ad‑hoc rm(tempDir, { recursive: true, force: true }) calls in the failing file with removeTempGitRepo(tempDir). Any spawn‑based Git call in the test must be awaited via runGitStreaming.

Alternative one‑liner if the fixture cannot be refactored immediately: pass maxRetries/retryDelay to the existing fs.rm call. This is necessary but not sufficient on its own while detached gc can still write, hence fixes 1 and 2 above.

Verification

A. Reproduce the race deterministically (proves the helper works)

Add a temporary regression test that injects a straggler writer during cleanup:

import { test, expect } from 'vitest';
import { mkdtemp, mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { removeTempGitRepo } from './git-fixtures';

test('removeTempGitRepo survives a straggler writing into .git/objects', async () => {
  const dir = await mkdtemp(join(tmpdir(), 'race-'));
  const objects = join(dir, '.git', 'objects');
  await mkdir(objects, { recursive: true });

  // Keep repopulating the directory while removal is in flight.
  const straggler = setInterval(() => {
    void writeFile(join(objects, `tmp_${Date.now()}_${Math.random()}`), 'x');
  }, 1);

  setTimeout(() => clearInterval(straggler), 300);
  await expect(removeTempGitRepo(dir)).resolves.toBeUndefined();
});

On the old plain‑rm path this test fails with ENOTEMPTY; on the hardened path it passes.

B. Local validation

# 1. Isolated failing file, repeated to defeat flakiness
npx vitest run src/cli/http-durability.test.ts --repeat 20

# 2. Full suite under CI-like contention
npx vitest run --pool=forks --maxWorkers=2

# 3. Typecheck / lint the new helper
npx tsc --noEmit && npx eslint src/test/git-fixtures.ts

C. CI validation (required gate)

Push to the failed branch and open/refresh the PR.
Confirm the first run on a docs/metadata‑only commit is green.
Re‑run the workflow at least twice (gh run rerun <id>) to exercise scheduling variance:

gh run rerun 34640745319 --failed
gh run watch

Require the successor run (34641646773 was already green; the new run must also be green) before merging.
If ENOTEMPTY reappears, escalate: capture lsof +D <temp-namespace> and ps -ef | grep '[g]it' from a failing job artifact to identify the remaining writer before relaxing anything.

Acceptance criteria

Hardened regression test passes 100% locally.
src/cli/http-durability.test.ts 20/20 repeated runs green.
Full 944‑test suite green in CI with no ENOTEMPTY.
No product code or assertions changed.
A green successor CI run is recorded for the fix commit.

Guardrails

Never fix this by deleting assertions, adding test.retry, or marking the file skipped. Those mask the race.
Keep the temp directory prefix (duckbrain-ns-) so CI artifacts and lsof triage remain possible.
If a future test needs a genuinely persistent repo, place it under the repo’s .cache/workspace, not the OS temp dir, so the cleanup fixture is not responsible for it.
Treat any recurrence on a non‑docs commit as a new signal worth investigating — but only after confirming the relying subprocess and maintenance settings above.

Evidence & signatures

# Evidence
- Problem class: node-vitest-temp-git-enotempty-cleanup-race
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T20:04:39.792Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A docs-only GitHub Actions run failed one Vitest teardown with ENOTEMPTY while removing a temporary namespace repository's .git/objects directory. The same committed tree had passed the full 115-file/944-test GitReins suite, and the isolated failing file immediately passed 5/5. A successor CI run on the next metadata-only commit passed. Classify this as an asynchronous temporary Git-repository cleanup race before changing product code: confirm the change cannot affect the failing code, rerun the exact failing file in isolation, and require a green successor run. If it repeats, harden the test cleanup by waiting for Git subprocess completion and retrying rm with bounded backoff rather than weakening assertions.", "environment": "GitHub Actions ubuntu runner; DuckBrain test suite", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "node-vitest-temp-git-enotempty-cleanup-race", "provider": "openrouter", "solved_at": "2026-09-11T20:04:39.792Z", "version": "Node 20.x, Vitest 4.1.10"}

Answer 2

Fixing the Vitest `ENOTEMPTY` Temp‑Git Cleanup Race (`node-vitest-temp-git-enotempty-cleanup-race`)

Classification

Flaky asynchronous temp‑repository cleanup race — not a product regression.

Evidence already gathered:

Signal	Value	Meaning
Failed commit	`377a9a34` (docs/metadata‑only)	No product code path changed
Failed run	`34640745319`	One Vitest teardown failure
Isolated re‑run	`src/cli/http-durability.test.ts` 5/5 passed	Failure not deterministic
Full suite	115 files / 944 tests passed on same tree	Not a systemic break
Successor commit/run	`34641646773: success`	Transient, self‑clearing

Do not weaken assertions or change product code. Harden only the test fixture teardown.

Root‑Cause Analysis

Failure mechanics

ENOTEMPTY: directory not empty, rmdir '<temp-namespace>/.git/objects' is emitted by fs.rm(dir, { recursive: true }) (directly or via Vitest’s temp fixture / rimraf).

Node’s recursive remove works in two phases:

Walk the tree, unlink files, and rmdir leaf directories.
Walk upward and rmdir parent directories.

If any process writes a new file into .git/objects after the walk of that directory but before its rmdir, the rmdir sees a non‑empty directory and returns ENOTEMPTY.

Who is still writing after the test returns?

The two realistic sources are:

Un‑awaited git child process. The test spawns git with spawn/exec and does not await the close event before the fixture’s cleanup runs. Tests complete while git commit / git add is still flushing objects.
Detached Git auto‑maintenance. git defaults to gc.auto and gc.autoDetach=true. After a command exits, Git may detach a background gc/maintenance process that writes packs into .git/objects. The parent process is gone, so no close event ever signals it.

The second is the classic cause: the git subprocess has exited, so simply awaiting close is not enough — a detached grandchild keeps mutating the directory.

Why CI and not local

GitHub Actions runners are slower and more contended, widening the window between the directory walk and the rmdir.
On the full 944‑test run, the shared runner pressure makes the race far more likely than the isolated 5/5 run.
With force: true and Vitest’s default retry settings, a transient write usually yields to ENOTEMPTY only occasionally — exactly the intermittent signature observed.

Exact Fix

Harden the temp‑repo helper in three layers. All three are required; retries alone are insufficient when a detached gc keeps repopulating the directory.

1. Await every Git subprocess and disable detached maintenance

Create/extend a shared test utility (e.g. src/test/git-fixtures.ts):

import { execFile } from 'node:child_process';
import { promisify } from 'node:util';

const execFileAsync = promisify(execFile);

/** Run git, wait for full process exit, and disable background work. */
export async function runGit(
  cwd: string,
  args: string[],
  env: NodeJS.ProcessEnv = {},
): Promise<{ stdout: string; stderr: string }> {
  return execFileAsync('git', args, {
    cwd,
    env: {
      ...process.env,
      // Never prompt, never touch user/system config.
      GIT_TERMINAL_PROMPT: '0',
      GIT_CONFIG_GLOBAL: '/dev/null',
      GIT_CONFIG_SYSTEM: '/dev/null',
      // Avoid optional lock files lingering.
      GIT_OPTIONAL_LOCKS: '0',
      ...env,
    },
  });
}

After git init, pin the repository so no detached maintenance is ever spawned:

await runGit(dir, ['config', 'gc.auto', '0']);
await runGit(dir, ['config', 'gc.autoDetach', 'false']);
await runGit(dir, ['config', 'maintenance.auto', 'false']);

If any code path spawns git directly, wrap it so close (not exit) is awaited — close guarantees stdio is fully drained and the process tree handle is released:

import { spawn } from 'node:child_process';

export function runGitStreaming(cwd: string, args: string[]): Promise<void> {
  return new Promise((resolve, reject) => {
    const child = spawn('git', args, {
      cwd,
      env: { ...process.env, GIT_OPTIONAL_LOCKS: '0' },
      stdio: 'ignore',
    });
    // 'close' fires after stdio streams are closed — strictly stronger than 'exit'.
    child.once('close', (code) =>
      code === 0 ? resolve() : reject(new Error(`git ${args.join(' ')} exited ${code}`)),
    );
    child.once('error', reject);
  });
}

2. Bounded‑backoff `rm` helper

import { rm } from 'node:fs/promises';

const RETRYABLE = new Set(['ENOTEMPTY', 'EBUSY', 'EPERM', 'ENOENT']);

/**
 * Remove a temp repository even if a straggler process briefly repopulates it.
 * Uses Node's built-in recursive retries plus an outer exponential backoff.
 */
export async function removeTempGitRepo(
  dir: string,
  { attempts = 8, baseDelayMs = 25 }: { attempts?: number; baseDelayMs?: number } = {},
): Promise<void> {
  let lastErr: unknown;

  for (let attempt = 0; attempt < attempts; attempt++) {
    try {
      await rm(dir, {
        recursive: true,
        force: true,
        maxRetries: 3,       // Node linear retry for ENOTEMPTY/EBUSY/EPERM
        retryDelay: baseDelayMs,
      });
      return;
    } catch (err) {
      lastErr = err;
      const code = (err as NodeJS.ErrnoException).code;
      if (!code || !RETRYABLE.has(code)) throw err;
      if (attempt === attempts - 1) break;
      const backoff = baseDelayMs * 2 ** attempt + Math.floor(Math.random() * baseDelayMs);
      await new Promise((r) => setTimeout(r, backoff));
    }
  }

  throw lastErr;
}

3. Wire into the Vitest fixture

Vitest 4 fixture lifecycle guarantees finally runs before the test is considered done. This is where the cleanup must live so it is awaited:

import { test as base } from 'vitest';
import { mkdtemp } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { runGit, removeTempGitRepo } from './git-fixtures';

export const test = base.extend<{ tempRepo: string }>({
  tempRepo: async ({}, use) => {
    const dir = await mkdtemp(join(tmpdir(), 'duckbrain-ns-'));
    await runGit(dir, ['init', '--initial-branch=main']);
    await runGit(dir, ['config', 'gc.auto', '0']);
    await runGit(dir, ['config', 'gc.autoDetach', 'false']);
    await runGit(dir, ['config', 'maintenance.auto', 'false']);

    try {
      await use(dir);
    } finally {
      await removeTempGitRepo(dir);
    }
  },
});

Alternative one‑liner if the fixture cannot be refactored immediately: pass maxRetries/retryDelay to the existing fs.rm call. This is necessary but not sufficient on its own while detached gc can still write, hence fixes 1 and 2 above.

Verification

A. Reproduce the race deterministically (proves the helper works)

Add a temporary regression test that injects a straggler writer during cleanup:

import { test, expect } from 'vitest';
import { mkdtemp, mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
import { removeTempGitRepo } from './git-fixtures';

test('removeTempGitRepo survives a straggler writing into .git/objects', async () => {
  const dir = await mkdtemp(join(tmpdir(), 'race-'));
  const objects = join(dir, '.git', 'objects');
  await mkdir(objects, { recursive: true });

  // Keep repopulating the directory while removal is in flight.
  const straggler = setInterval(() => {
    void writeFile(join(objects, `tmp_${Date.now()}_${Math.random()}`), 'x');
  }, 1);

  setTimeout(() => clearInterval(straggler), 300);
  await expect(removeTempGitRepo(dir)).resolves.toBeUndefined();
});

On the old plain‑rm path this test fails with ENOTEMPTY; on the hardened path it passes.

B. Local validation

# 1. Isolated failing file, repeated to defeat flakiness
npx vitest run src/cli/http-durability.test.ts --repeat 20

# 2. Full suite under CI-like contention
npx vitest run --pool=forks --maxWorkers=2

# 3. Typecheck / lint the new helper
npx tsc --noEmit && npx eslint src/test/git-fixtures.ts

C. CI validation (required gate)

Push to the failed branch and open/refresh the PR.
Confirm the first run on a docs/metadata‑only commit is green.
Re‑run the workflow at least twice (gh run rerun <id>) to exercise scheduling variance:

gh run rerun 34640745319 --failed
gh run watch

Require the successor run (34641646773 was already green; the new run must also be green) before merging.
If ENOTEMPTY reappears, escalate: capture lsof +D <temp-namespace> and ps -ef | grep '[g]it' from a failing job artifact to identify the remaining writer before relaxing anything.

Acceptance criteria

Hardened regression test passes 100% locally.
src/cli/http-durability.test.ts 20/20 repeated runs green.
Full 944‑test suite green in CI with no ENOTEMPTY.
No product code or assertions changed.
A green successor CI run is recorded for the fix commit.

Guardrails

Never fix this by deleting assertions, adding test.retry, or marking the file skipped. Those mask the race.
Keep the temp directory prefix (duckbrain-ns-) so CI artifacts and lsof triage remain possible.
If a future test needs a genuinely persistent repo, place it under the repo’s .cache/workspace, not the OS temp dir, so the cleanup fixture is not responsible for it.
Treat any recurrence on a non‑docs commit as a new signal worth investigating — but only after confirming the relying subprocess and maintenance settings above.

Evidence & signatures

# Evidence
- Problem class: node-vitest-temp-git-enotempty-cleanup-race
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T20:04:39.792Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A docs-only GitHub Actions run failed one Vitest teardown with ENOTEMPTY while removing a temporary namespace repository's .git/objects directory. The same committed tree had passed the full 115-file/944-test GitReins suite, and the isolated failing file immediately passed 5/5. A successor CI run on the next metadata-only commit passed. Classify this as an asynchronous temporary Git-repository cleanup race before changing product code: confirm the change cannot affect the failing code, rerun the exact failing file in isolation, and require a green successor run. If it repeats, harden the test cleanup by waiting for Git subprocess completion and retrying rm with bounded backoff rather than weakening assertions.", "environment": "GitHub Actions ubuntu runner; DuckBrain test suite", "language": "typescript", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "node-vitest-temp-git-enotempty-cleanup-race", "provider": "openrouter", "solved_at": "2026-09-11T20:04:39.792Z", "version": "Node 20.x, Vitest 4.1.10"}

Generated from the verified corpus · MIT licensedBack to the catalog

node-vitest-temp-git-enotempty-cleanup-race

Answer 1

Fixing the Vitest ENOTEMPTY Temp‑Git Cleanup Race (node-vitest-temp-git-enotempty-cleanup-race)

Classification

Root‑Cause Analysis

Failure mechanics

Who is still writing after the test returns?

Why CI and not local

Exact Fix

1. Await every Git subprocess and disable detached maintenance

2. Bounded‑backoff rm helper

3. Wire into the Vitest fixture

Verification

A. Reproduce the race deterministically (proves the helper works)

B. Local validation

C. CI validation (required gate)

Acceptance criteria

Guardrails

Evidence & signatures

Answer 2

Fixing the Vitest ENOTEMPTY Temp‑Git Cleanup Race (node-vitest-temp-git-enotempty-cleanup-race)

Classification

Root‑Cause Analysis

Failure mechanics

Who is still writing after the test returns?

Why CI and not local

Exact Fix

1. Await every Git subprocess and disable detached maintenance

2. Bounded‑backoff rm helper

3. Wire into the Vitest fixture

Verification

A. Reproduce the race deterministically (proves the helper works)

B. Local validation

C. CI validation (required gate)

Acceptance criteria

Guardrails

Evidence & signatures

Fixing the Vitest `ENOTEMPTY` Temp‑Git Cleanup Race (`node-vitest-temp-git-enotempty-cleanup-race`)

2. Bounded‑backoff `rm` helper

Fixing the Vitest `ENOTEMPTY` Temp‑Git Cleanup Race (`node-vitest-temp-git-enotempty-cleanup-race`)

2. Bounded‑backoff `rm` helper