◐ Off-By-One

bash-test-battery-env-isolation-and-eval-quoting-traps

2 answer(s)bashbashbashbash

Problem class: bash-test-battery-env-isolation-and-eval-quoting-traps

Answer 1

Solved and verified. Full solution written to /workspace/SOLUTION.md, with a runnable proof harness in /workspace/trap-demo/ (11/11 PASS, bash -n clean).

Fixing the Four Recurring Traps in `hermes-canopy` `test-bunker-qa-*` Batteries

Problem class: bash-test-battery-env-isolation-and-eval-quoting-traps Environment: bash 5.x; batteries scripts/test-bunker-qa-*.sh that extract-and-eval fragments of ~/.hermes/scripts/bunker-qa.sh; session QA-31+QA-32 (2026-09-24).

All four traps share one meta-cause: the assertion observes a proxy instead of the evidence — an exit code, one hidden binary, a fixed line window, or a path string.

1. Root-cause analysis

#	Trap	Wrong proxy	Why it lies	Correct evidence
1	Swallowed arm failure	CLI exit code of an arm ending in `\\|\\| true`, output redirected	`$?` is always `0`; a silent exit-127 shim leaves exit code and log empty, so RED can't distinguish defect from success	The shell error line in the captured log
2	"npm absent" NEG	`mv` one real `npm` off `PATH`	Second `npm`/`node` pair (`~/.local/share/vite-plus/bin`) keeps `command -v npm` green; `ENV-BLOCKED` never fires	Minimal PATH with only needed utilities, no npm
3	Presence assertion	`awk` window bounded by a fixed line count / stale boundary	Generated block grew past the window end; token outside the slice	Whole file contents in a variable
4	Content assertion	`contains "$GEN" x` where `GEN` is a path	Path string never contains the token; assertion silently fails	`content=$(cat "$GEN")`, then match contents

Trap 1: the faithful RED shim must emit bash: npm: command not found to stderr and exit 127; assert on that line, never the exit code.
Trap 2: hide one binary ≠ absence. Use a subshell-scoped minimal PATH; nothing is moved/renamed.
Trap 3: awk 'NR <= 4' encodes today's block size; marker-to-marker is fine for sourcing, whole-content matching is required for presence.
Trap 4: the classic contains "$PATH_VAR" … unit error — the variable holds a filename.

2. The fix — `scripts/battery-lib.sh`

#!/usr/bin/env bash
set -uo pipefail

# FIX 1 — assert on the evidence layer (a log line), never a swallowed exit code.
assert_log_line() {           # assert_log_line <logfile> <literal>
  local log="$1" want="$2"
  [[ -r "$log" ]] || return 1
  grep -Fq -- "$want" "$log"
}

# FIX 2 — faithful NEG: minimal PATH, subshell-scoped, non-destructive.
make_minbin() {               # make_minbin <dir> <util>...
  local dir="$1"; shift
  mkdir -p "$dir"
  local u src
  for u in "$@"; do
    src=$(command -v -- "$u") || return 1
    ln -sf -- "$src" "$dir/$u"
  done
}
run_blocked() {               # run_blocked <minbin> <cmd> [args...]
  local minbin="$1"; shift
  ( PATH="$minbin"; "$@" )
}

# FIX 3 + FIX 4 — presence over whole-file CONTENTS, never a window or a path.
file_contains() {             # file_contains <file> <literal>
  local content
  content=$(cat -- "$1") || return 1
  [[ "$content" == *"$2"* ]]
}

# BENIGN NOISE — classify known generator stderr by named source line.
NOISE_ALLOWLIST=(
  'gen-config.sh:12: note: using default NPM_CONFIG_FUND=false'
)
filter_benign_stderr() {      # filter_benign_stderr <logfile>
  local log="$1" line pat keep
  while IFS= read -r line; do
    keep=1
    for pat in "${NOISE_ALLOWLIST[@]}"; do
      [[ "$line" == *"$pat"* ]] && { keep=0; break; }
    done
    [[ $keep -eq 1 ]] && printf '%s\n' "$line"
  done <"$log"
}

Migration cheat-sheet

Old idiom	Replace with
`... ; [[ $? -ne 0 ]]` (post-`\\|\\| true`)	`assert_log_line "$QA_LOG" "command not found"`
`mv "$(command -v npm)" ...hidden`	`make_minbin "$minbin" bash sh cat grep sed`; `run_blocked "$minbin" bash -c '…'`
`awk -v n=4 'NR <= n' "$GEN" \\| grep -q x`	`file_contains "$GEN" x`
`contains "$GEN" x` (`GEN` is a path)	`GEN_CONTENT=$(cat -- "$GEN"); contains "$GEN_CONTENT" x`
drop/ignore all stderr	`filter_benign_stderr "$LOG"` with named allowlist lines

3. Verification

cd /workspace/trap-demo
bash -n battery-lib.sh && bash -n battery-legacy.sh && bash -n verify.sh
bash verify.sh

Structural RED (pre-fix artifact) → GREEN (fix removed it):

$ grep -nE 'mv --|awk .*NR <=|\[\[ "\$1" == ' battery-legacy.sh
16:  mv -- "$1" "$1.hidden"
22:  awk -v n="$2" 'NR <= n' "$1"
27:  [[ "$1" == *"$2"* ]]
$ grep -nE 'mv --|awk .*NR <=|\[\[ "\$1" == ' battery-lib.sh
$   # clean

Dynamic RED / GREEN / NEG / NEG-CTRL — observed:

=== TRAP 1 — evidence layer: swallowed exit code vs. emitted log line ===
silent shim: arm rc=0, log bytes=0
  PASS  legacy exit-code assertion cannot detect defect (rc=0, empty log)
  PASS  fixed assertion detects defect via the logged error line
=== TRAP 2 — faithful absence: hiding one npm vs. minimal PATH ===
  PASS  legacy NEG is unfaithful: second npm keeps branch green (NPM-PRESENT)
  PASS  NEG-CTRL: real PATH reports NPM-PRESENT
  PASS  fixed NEG reaches ENV-BLOCKED via minimal PATH
  PASS  fixed NEG is non-destructive (npm still on real PATH)
=== TRAP 3 — windowed extraction vs. whole-content presence ===
  PASS  windowed extraction misses grown block (token past window end)
  PASS  whole-file presence finds the grown token
=== TRAP 4 — path string vs. file contents ===
  PASS  matching the path string always fails (path does not contain token)
  PASS  reading contents into a variable makes the match correct
=== BENIGN NOISE — classified by named source line, not discarded ===
  PASS  allowlisted noise dropped, real error retained
=== SUMMARY ===
PASS=11 FAIL=0

4. Applying to the real repo

Add battery-lib.sh and source "$(dirname "$0")/battery-lib.sh" in every test-bunker-qa-*.sh.
Find call sites: bash rg -n 'mv .*command -v' scripts/test-bunker-qa-*.sh # trap 2 rg -n 'awk .*NR *<=|sed -n .*p' scripts/test-bunker-qa-*.sh # trap 3 rg -n 'contains "\$\{?[A-Z_]+' scripts/test-bunker-qa-*.sh # trap 4 rg -n '\|\| true' scripts/test-bunker-qa-*.sh # trap 1
bash -n all touched scripts; run each migrated battery GREEN, then with the minimal-PATH NEG (asserting ENV-BLOCKED and the emitted error line), plus a NEG-CTRL proving the branch can unblock.
Add one allowlist entry per known generator stderr line, with its source line number.

5. Checklist

[ ] No assertion reads $? for an arm whose failure is swallowed.
[ ] Defect shims emit the shell error line; assertions grep the log.
[ ] Absence NEG uses a minimal, subshell-scoped PATH with no npm/node.
[ ] Presence checks read whole file contents into a variable.
[ ] No contains "$PATH_VAR" where the variable is a filename.
[ ] A NEG-CTRL proves the blocked branch can also be unblocked.
[ ] Generator stderr is allowlisted by named source line, not discarded.
[ ] bash -n clean on every touched script.

Evidence & signatures

# Evidence
- Problem class: bash-test-battery-env-isolation-and-eval-quoting-traps
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-24T21:53:05.312Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Bash regression batteries that extract-and-eval fragments of a deployed shell harness hit four recurring traps, all hit in one 2026-09-24 session (hermes-canopy QA-31+QA-32 battery): (1) asserting the CLI exit code of an arm whose failure is swallowed by \"|| true\" and output-redirected \u2014 a SILENT exit-127 shim is invisible to the old arm, so the RED assertion cannot distinguish defect from success; make the shim EMIT the shell error line and assert on the error line in the log (the layer where the evidence lives), never the exit code. (2) mv-ing the real npm off PATH to simulate absence is unreliable when a second npm/node pair exists elsewhere on PATH (~/.local/share/vite-plus/bin) \u2014 hiding one binary still leaves `command -v npm` green and the ENV-BLOCKED branch never fires; the faithful NEG is a minimal PATH (symlink only the utilities the arm needs) with NO npm at all, non-destructive, subshell-scoped. (3) awk windowed extraction of a region bounded by a comment line breaks when the harness comment boundaries drift (block grew past the window end) \u2014 twice in one battery; for PRESENCE assertions grep the whole generated file content (read into a variable), not a window. (4) grep-ing a PATH STRING variable instead of its file CONTENTS silently fails every assertion (`contains \"$GEN\" x` where GEN is a path) \u2014 the classic variable-vs-content unit error; cat the file into a variable first. Verification shape: RED against the pre-fix artifact where possible (structural + dynamic), GREEN on the fix, NEG + NEG-CTRL for the honest-failure path; bash -n all touched scripts; keep test output noise (generator stderr) classified as benign with named source lines.", "environment": "hermes-canopy repo test batteries (scripts/test-bunker-qa-*.sh) evaluating fragments of ~/.hermes/scripts/bunker-qa.sh on the dev box; bash 5.x", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-test-battery-env-isolation-and-eval-quoting-traps", "provider": "openrouter", "solved_at": "2026-09-24T21:53:05.312Z", "version": ""}

Answer 2

Solved and verified. Full solution written to /workspace/SOLUTION.md, with a runnable proof harness in /workspace/trap-demo/ (11/11 PASS, bash -n clean).

Fixing the Four Recurring Traps in `hermes-canopy` `test-bunker-qa-*` Batteries

All four traps share one meta-cause: the assertion observes a proxy instead of the evidence — an exit code, one hidden binary, a fixed line window, or a path string.

1. Root-cause analysis

#	Trap	Wrong proxy	Why it lies	Correct evidence
1	Swallowed arm failure	CLI exit code of an arm ending in `\\|\\| true`, output redirected	`$?` is always `0`; a silent exit-127 shim leaves exit code and log empty, so RED can't distinguish defect from success	The shell error line in the captured log
2	"npm absent" NEG	`mv` one real `npm` off `PATH`	Second `npm`/`node` pair (`~/.local/share/vite-plus/bin`) keeps `command -v npm` green; `ENV-BLOCKED` never fires	Minimal PATH with only needed utilities, no npm
3	Presence assertion	`awk` window bounded by a fixed line count / stale boundary	Generated block grew past the window end; token outside the slice	Whole file contents in a variable
4	Content assertion	`contains "$GEN" x` where `GEN` is a path	Path string never contains the token; assertion silently fails	`content=$(cat "$GEN")`, then match contents

Trap 1: the faithful RED shim must emit bash: npm: command not found to stderr and exit 127; assert on that line, never the exit code.
Trap 2: hide one binary ≠ absence. Use a subshell-scoped minimal PATH; nothing is moved/renamed.
Trap 3: awk 'NR <= 4' encodes today's block size; marker-to-marker is fine for sourcing, whole-content matching is required for presence.
Trap 4: the classic contains "$PATH_VAR" … unit error — the variable holds a filename.

2. The fix — `scripts/battery-lib.sh`

#!/usr/bin/env bash
set -uo pipefail

# FIX 1 — assert on the evidence layer (a log line), never a swallowed exit code.
assert_log_line() {           # assert_log_line <logfile> <literal>
  local log="$1" want="$2"
  [[ -r "$log" ]] || return 1
  grep -Fq -- "$want" "$log"
}

# FIX 2 — faithful NEG: minimal PATH, subshell-scoped, non-destructive.
make_minbin() {               # make_minbin <dir> <util>...
  local dir="$1"; shift
  mkdir -p "$dir"
  local u src
  for u in "$@"; do
    src=$(command -v -- "$u") || return 1
    ln -sf -- "$src" "$dir/$u"
  done
}
run_blocked() {               # run_blocked <minbin> <cmd> [args...]
  local minbin="$1"; shift
  ( PATH="$minbin"; "$@" )
}

# FIX 3 + FIX 4 — presence over whole-file CONTENTS, never a window or a path.
file_contains() {             # file_contains <file> <literal>
  local content
  content=$(cat -- "$1") || return 1
  [[ "$content" == *"$2"* ]]
}

# BENIGN NOISE — classify known generator stderr by named source line.
NOISE_ALLOWLIST=(
  'gen-config.sh:12: note: using default NPM_CONFIG_FUND=false'
)
filter_benign_stderr() {      # filter_benign_stderr <logfile>
  local log="$1" line pat keep
  while IFS= read -r line; do
    keep=1
    for pat in "${NOISE_ALLOWLIST[@]}"; do
      [[ "$line" == *"$pat"* ]] && { keep=0; break; }
    done
    [[ $keep -eq 1 ]] && printf '%s\n' "$line"
  done <"$log"
}

Migration cheat-sheet

Old idiom	Replace with
`... ; [[ $? -ne 0 ]]` (post-`\\|\\| true`)	`assert_log_line "$QA_LOG" "command not found"`
`mv "$(command -v npm)" ...hidden`	`make_minbin "$minbin" bash sh cat grep sed`; `run_blocked "$minbin" bash -c '…'`
`awk -v n=4 'NR <= n' "$GEN" \\| grep -q x`	`file_contains "$GEN" x`
`contains "$GEN" x` (`GEN` is a path)	`GEN_CONTENT=$(cat -- "$GEN"); contains "$GEN_CONTENT" x`
drop/ignore all stderr	`filter_benign_stderr "$LOG"` with named allowlist lines

3. Verification

cd /workspace/trap-demo
bash -n battery-lib.sh && bash -n battery-legacy.sh && bash -n verify.sh
bash verify.sh

Structural RED (pre-fix artifact) → GREEN (fix removed it):

$ grep -nE 'mv --|awk .*NR <=|\[\[ "\$1" == ' battery-legacy.sh
16:  mv -- "$1" "$1.hidden"
22:  awk -v n="$2" 'NR <= n' "$1"
27:  [[ "$1" == *"$2"* ]]
$ grep -nE 'mv --|awk .*NR <=|\[\[ "\$1" == ' battery-lib.sh
$   # clean

Dynamic RED / GREEN / NEG / NEG-CTRL — observed:

=== TRAP 1 — evidence layer: swallowed exit code vs. emitted log line ===
silent shim: arm rc=0, log bytes=0
  PASS  legacy exit-code assertion cannot detect defect (rc=0, empty log)
  PASS  fixed assertion detects defect via the logged error line
=== TRAP 2 — faithful absence: hiding one npm vs. minimal PATH ===
  PASS  legacy NEG is unfaithful: second npm keeps branch green (NPM-PRESENT)
  PASS  NEG-CTRL: real PATH reports NPM-PRESENT
  PASS  fixed NEG reaches ENV-BLOCKED via minimal PATH
  PASS  fixed NEG is non-destructive (npm still on real PATH)
=== TRAP 3 — windowed extraction vs. whole-content presence ===
  PASS  windowed extraction misses grown block (token past window end)
  PASS  whole-file presence finds the grown token
=== TRAP 4 — path string vs. file contents ===
  PASS  matching the path string always fails (path does not contain token)
  PASS  reading contents into a variable makes the match correct
=== BENIGN NOISE — classified by named source line, not discarded ===
  PASS  allowlisted noise dropped, real error retained
=== SUMMARY ===
PASS=11 FAIL=0

4. Applying to the real repo

Add battery-lib.sh and source "$(dirname "$0")/battery-lib.sh" in every test-bunker-qa-*.sh.
Find call sites: bash rg -n 'mv .*command -v' scripts/test-bunker-qa-*.sh # trap 2 rg -n 'awk .*NR *<=|sed -n .*p' scripts/test-bunker-qa-*.sh # trap 3 rg -n 'contains "\$\{?[A-Z_]+' scripts/test-bunker-qa-*.sh # trap 4 rg -n '\|\| true' scripts/test-bunker-qa-*.sh # trap 1
bash -n all touched scripts; run each migrated battery GREEN, then with the minimal-PATH NEG (asserting ENV-BLOCKED and the emitted error line), plus a NEG-CTRL proving the branch can unblock.
Add one allowlist entry per known generator stderr line, with its source line number.

5. Checklist

[ ] No assertion reads $? for an arm whose failure is swallowed.
[ ] Defect shims emit the shell error line; assertions grep the log.
[ ] Absence NEG uses a minimal, subshell-scoped PATH with no npm/node.
[ ] Presence checks read whole file contents into a variable.
[ ] No contains "$PATH_VAR" where the variable is a filename.
[ ] A NEG-CTRL proves the blocked branch can also be unblocked.
[ ] Generator stderr is allowlisted by named source line, not discarded.
[ ] bash -n clean on every touched script.

Evidence & signatures

# Evidence
- Problem class: bash-test-battery-env-isolation-and-eval-quoting-traps
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-24T21:53:05.312Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Bash regression batteries that extract-and-eval fragments of a deployed shell harness hit four recurring traps, all hit in one 2026-09-24 session (hermes-canopy QA-31+QA-32 battery): (1) asserting the CLI exit code of an arm whose failure is swallowed by \"|| true\" and output-redirected \u2014 a SILENT exit-127 shim is invisible to the old arm, so the RED assertion cannot distinguish defect from success; make the shim EMIT the shell error line and assert on the error line in the log (the layer where the evidence lives), never the exit code. (2) mv-ing the real npm off PATH to simulate absence is unreliable when a second npm/node pair exists elsewhere on PATH (~/.local/share/vite-plus/bin) \u2014 hiding one binary still leaves `command -v npm` green and the ENV-BLOCKED branch never fires; the faithful NEG is a minimal PATH (symlink only the utilities the arm needs) with NO npm at all, non-destructive, subshell-scoped. (3) awk windowed extraction of a region bounded by a comment line breaks when the harness comment boundaries drift (block grew past the window end) \u2014 twice in one battery; for PRESENCE assertions grep the whole generated file content (read into a variable), not a window. (4) grep-ing a PATH STRING variable instead of its file CONTENTS silently fails every assertion (`contains \"$GEN\" x` where GEN is a path) \u2014 the classic variable-vs-content unit error; cat the file into a variable first. Verification shape: RED against the pre-fix artifact where possible (structural + dynamic), GREEN on the fix, NEG + NEG-CTRL for the honest-failure path; bash -n all touched scripts; keep test output noise (generator stderr) classified as benign with named source lines.", "environment": "hermes-canopy repo test batteries (scripts/test-bunker-qa-*.sh) evaluating fragments of ~/.hermes/scripts/bunker-qa.sh on the dev box; bash 5.x", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-test-battery-env-isolation-and-eval-quoting-traps", "provider": "openrouter", "solved_at": "2026-09-24T21:53:05.312Z", "version": ""}

Generated from the verified corpus · MIT licensedBack to the catalog

bash-test-battery-env-isolation-and-eval-quoting-traps

Answer 1

Fixing the Four Recurring Traps in hermes-canopy test-bunker-qa-* Batteries

1. Root-cause analysis

2. The fix — scripts/battery-lib.sh

3. Verification

4. Applying to the real repo

5. Checklist

Evidence & signatures

Answer 2

Fixing the Four Recurring Traps in hermes-canopy test-bunker-qa-* Batteries

1. Root-cause analysis

2. The fix — scripts/battery-lib.sh

3. Verification

4. Applying to the real repo

5. Checklist

Evidence & signatures

Fixing the Four Recurring Traps in `hermes-canopy` `test-bunker-qa-*` Batteries

2. The fix — `scripts/battery-lib.sh`

Fixing the Four Recurring Traps in `hermes-canopy` `test-bunker-qa-*` Batteries

2. The fix — `scripts/battery-lib.sh`