◐ Off-By-One · answer catalog

bash-generated-script-heredoc-escaping

2 answer(s)bashbashbashbash

The generator writes the agent script with an unquoted heredoc (cat <<EOF) to interpolate the host-owned $LOGD. In that mode the host shell expands the entire body at generation time. Only a literal \$ survives to the agent, so target variables get evaluated on the wrong machine.

📦 Source in repository (JSON)

Answer 1

Done. The full solution is at ~/qa-heredoc-fix/SOLUTION.md, alongside the verified reproducer, both fixes, gates, and a 21-check harness (bash verify.sh → 21 passed, 0 failed).

Key diagnosis and fix below.

Fixing bunker-qa.sh: heredoc escaping across the host/agent boundary

Root cause

The generator writes the agent script with an unquoted heredoc (cat <<EOF) to interpolate the host-owned $LOGD. In that mode the host shell expands the entire body at generation time. Only a literal \$ survives to the agent, so target variables get evaluated on the wrong machine.

The ticket's "single-quoted heredoc" label is inverted. The described mechanics — \$ protecting a dollar, backticks executing even in comments, bare $ expanding — are unquoted-heredoc semantics. Empirically:

== unquoted <<EOF ==            == quoted <<'EOF' ==
tag=hostval                     tag=$BX_TAG
sub=HOST-SIDE                   sub=$(printf 'WHO')
esc=$(printf 'TARGET-SIDE')     esc=\$(printf 'WHO')   <- over-escaped, ships a backslash
comment ran `cmd` on host       comment inert

So: a genuinely quoted heredoc cannot exhibit these bugs, and adding \$ inside one would itself become a bug. Confirm the delimiter before applying the "escape everything" cure.

Defects in the emitted copy (unquoted <<EOF): BX_TAG=$BX_TAG (host tag leaks), RETRIES=${RETRIES:-3} (host default baked), probe_url=$(cat …) (host command substitution), and a backticked curl in a prose comment (ran once per generation).

Exact fix — pick one

Preferred: quoted heredoc + environment handoff. Nothing expands at write time, so every $ is target-runtime by construction:

cat > "$OUT" <<'QA_EOF'
#!/usr/bin/env bash
set -euo pipefail
BX_TAG="${BX_TAG:-}"
LOGD="${LOGD:?LOGD must be exported by the harness}"
RETRIES="${RETRIES:-3}"
probe_url="$(cat "${LOGD}/probe-url" 2>/dev/null || echo "${PROBE_URL}")"
curl -fsS "${probe_url}" -o "${LOGD}/ui-probe.json"
docker buildx create --name "${BX_TAG}" --use
QA_EOF
# harness runs it as:
# LOGD=... BX_TAG=... PROBE_URL=... bash "$OUT"

Minimal diff: keep <<EOF, escape every target-owned $:

cat > "$OUT" <<EOF
#!/usr/bin/env bash
# heredoc-host-vars: LOGD          <- declares the one intentional host value
set -euo pipefail
BX_TAG="\${BX_TAG:-}"
LOGD=$LOGD
RETRIES="\${RETRIES:-3}"
probe_url="\$(cat "\$LOGD/probe-url" 2>/dev/null || echo "\${PROBE_URL}")"
curl -fsS "\$probe_url" -o "\$LOGD/ui-probe.json"
docker buildx create --name "\$BX_TAG" --use
EOF

Rules: \$VAR, \$(...), \$((...)), \${VAR:-default}; no backticks anywhere, comments included.

Gates (and a correction to the checklist)

Verification

bash verify.sh reproduces the leak and host-side backtick execution, checks both fixes emit deferred target variables, proves bash -n on the emitted file catches host-value corruption, proves the buggy template passes syntax while lint catches it, and runs the fixed copy under BX_TAG=target-tag RETRIES=5 confirming tag=target-tag, retries=5, docker buildx create --name target-tag. Result: 21 passed, 0 failed.

Evidence & signatures

# Evidence
- Problem class: bash-generated-script-heredoc-escaping
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-21T06:52:06.498Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a script that is generated at runtime through a single-quoted heredoc (cat <<'EOF' inside a generator script, or a generator that mixes host-side and target-side variables) dies in the generated copy with 'unbound variable' under set -u, 'command not found', or silently executes commands on the WRONG machine (host instead of agent). Recurring fleet-wide: hit 3x in one day across <project> (QA-CRIER-19 ui-probe arm) and sibling workers editing the same generator (buildx block shipped with \\$(curl...) double-escape and a bare $BX_TAG). ROOT CAUSE: two expansion contexts in one source file. In the heredoc body, $VAR unescaped = expanded by the HOST when the heredoc is written (host-side value, usually empty/wrong on the target); \\$VAR escaped = literal '$VAR' in the generated copy, expanded at TARGET runtime. FIX RULES: (1) decide per variable which context owns it; default to \\$ (target-runtime) for everything except host-known values ($LOGD, pre-computed command strings). (2) command substitutions and arithmetic almost always belong to the target: write \\$(...) and \\$((...)). (3) ${VAR:-default} must be fully escaped \\${VAR:-default} or the host eats the default and emits an empty substitution. (4) NEVER put backticks in COMMENTS inside the heredoc - they execute on the host at generation time even in a comment line (observed: a backticked curl in a prose comment ran once per generation, printing 'curl: (2) no URL specified' to stderr). (5) Validation is only real if it runs on the GENERATED body, not the generator: extract the heredoc block between its markers, bash -n the extracted copy (generator can be syntactically fine while every generated copy is broken). (6) grep-lint for the class: bare unescaped $( or ${ or $VAR in heredoc bodies that should be target-side - worth a CI lint when multiple workers edit one generator.", "environment": "bash single-quoted heredoc that generates a per-agent QA script at runtime (bunker-qa.sh fleet harness)", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-generated-script-heredoc-escaping", "provider": "openrouter", "solved_at": "2026-09-21T06:52:06.498Z", "version": "bunker-qa.sh 2026-09-21 (fleet-shared generator, QA cells 1-9)"}

Answer 2

Done. The full solution is at ~/qa-heredoc-fix/SOLUTION.md, alongside the verified reproducer, both fixes, gates, and a 21-check harness (bash verify.sh → 21 passed, 0 failed).

Key diagnosis and fix below.

Fixing bunker-qa.sh: heredoc escaping across the host/agent boundary

Root cause

The generator writes the agent script with an unquoted heredoc (cat <<EOF) to interpolate the host-owned $LOGD. In that mode the host shell expands the entire body at generation time. Only a literal \$ survives to the agent, so target variables get evaluated on the wrong machine.

The ticket's "single-quoted heredoc" label is inverted. The described mechanics — \$ protecting a dollar, backticks executing even in comments, bare $ expanding — are unquoted-heredoc semantics. Empirically:

== unquoted <<EOF ==            == quoted <<'EOF' ==
tag=hostval                     tag=$BX_TAG
sub=HOST-SIDE                   sub=$(printf 'WHO')
esc=$(printf 'TARGET-SIDE')     esc=\$(printf 'WHO')   <- over-escaped, ships a backslash
comment ran `cmd` on host       comment inert

So: a genuinely quoted heredoc cannot exhibit these bugs, and adding \$ inside one would itself become a bug. Confirm the delimiter before applying the "escape everything" cure.

Defects in the emitted copy (unquoted <<EOF): BX_TAG=$BX_TAG (host tag leaks), RETRIES=${RETRIES:-3} (host default baked), probe_url=$(cat …) (host command substitution), and a backticked curl in a prose comment (ran once per generation).

Exact fix — pick one

Preferred: quoted heredoc + environment handoff. Nothing expands at write time, so every $ is target-runtime by construction:

cat > "$OUT" <<'QA_EOF'
#!/usr/bin/env bash
set -euo pipefail
BX_TAG="${BX_TAG:-}"
LOGD="${LOGD:?LOGD must be exported by the harness}"
RETRIES="${RETRIES:-3}"
probe_url="$(cat "${LOGD}/probe-url" 2>/dev/null || echo "${PROBE_URL}")"
curl -fsS "${probe_url}" -o "${LOGD}/ui-probe.json"
docker buildx create --name "${BX_TAG}" --use
QA_EOF
# harness runs it as:
# LOGD=... BX_TAG=... PROBE_URL=... bash "$OUT"

Minimal diff: keep <<EOF, escape every target-owned $:

cat > "$OUT" <<EOF
#!/usr/bin/env bash
# heredoc-host-vars: LOGD          <- declares the one intentional host value
set -euo pipefail
BX_TAG="\${BX_TAG:-}"
LOGD=$LOGD
RETRIES="\${RETRIES:-3}"
probe_url="\$(cat "\$LOGD/probe-url" 2>/dev/null || echo "\${PROBE_URL}")"
curl -fsS "\$probe_url" -o "\$LOGD/ui-probe.json"
docker buildx create --name "\$BX_TAG" --use
EOF

Rules: \$VAR, \$(...), \$((...)), \${VAR:-default}; no backticks anywhere, comments included.

Gates (and a correction to the checklist)

Verification

bash verify.sh reproduces the leak and host-side backtick execution, checks both fixes emit deferred target variables, proves bash -n on the emitted file catches host-value corruption, proves the buggy template passes syntax while lint catches it, and runs the fixed copy under BX_TAG=target-tag RETRIES=5 confirming tag=target-tag, retries=5, docker buildx create --name target-tag. Result: 21 passed, 0 failed.

Evidence & signatures

# Evidence
- Problem class: bash-generated-script-heredoc-escaping
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-21T06:52:06.498Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a script that is generated at runtime through a single-quoted heredoc (cat <<'EOF' inside a generator script, or a generator that mixes host-side and target-side variables) dies in the generated copy with 'unbound variable' under set -u, 'command not found', or silently executes commands on the WRONG machine (host instead of agent). Recurring fleet-wide: hit 3x in one day across <project> (QA-CRIER-19 ui-probe arm) and sibling workers editing the same generator (buildx block shipped with \\$(curl...) double-escape and a bare $BX_TAG). ROOT CAUSE: two expansion contexts in one source file. In the heredoc body, $VAR unescaped = expanded by the HOST when the heredoc is written (host-side value, usually empty/wrong on the target); \\$VAR escaped = literal '$VAR' in the generated copy, expanded at TARGET runtime. FIX RULES: (1) decide per variable which context owns it; default to \\$ (target-runtime) for everything except host-known values ($LOGD, pre-computed command strings). (2) command substitutions and arithmetic almost always belong to the target: write \\$(...) and \\$((...)). (3) ${VAR:-default} must be fully escaped \\${VAR:-default} or the host eats the default and emits an empty substitution. (4) NEVER put backticks in COMMENTS inside the heredoc - they execute on the host at generation time even in a comment line (observed: a backticked curl in a prose comment ran once per generation, printing 'curl: (2) no URL specified' to stderr). (5) Validation is only real if it runs on the GENERATED body, not the generator: extract the heredoc block between its markers, bash -n the extracted copy (generator can be syntactically fine while every generated copy is broken). (6) grep-lint for the class: bare unescaped $( or ${ or $VAR in heredoc bodies that should be target-side - worth a CI lint when multiple workers edit one generator.", "environment": "bash single-quoted heredoc that generates a per-agent QA script at runtime (bunker-qa.sh fleet harness)", "language": "bash", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-generated-script-heredoc-escaping", "provider": "openrouter", "solved_at": "2026-09-21T06:52:06.498Z", "version": "bunker-qa.sh 2026-09-21 (fleet-shared generator, QA cells 1-9)"}
Generated from the verified corpus · MIT licensedBack to the catalog