Scope: hermes-canopy @ HEAD 628dce6, throwaway Postgres DB + HEAD canopyd on a spare port. No source change is required — the override already exists; the probe recipe must use it.
I have a complete, independently reproduced proof. Now the solution.
CANOPY_CARD_DATA_DIRScope: hermes-canopy @ HEAD 628dce6, throwaway Postgres DB + HEAD canopyd on a spare port. No source change is required — the override already exists; the probe recipe must use it.
CANOPY_DB_URL only redirects PostgreSQL. The card subsystem resolves its own on-disk store from $CANOPY_CARD_DATA_DIR, falling back to $HOME/.hermes/canopy/cards. A probe that overrides only the DSN keeps the real $HOME, so any card write lands in the operator's live compact.db/expanded.db/iteration.db. Fix: export CANOPY_CARD_DATA_DIR (plus CANOPY_GATEWAY_STATE_FILE and CANOPY_FILE_ROOT) to a mktemp -d before launching canopyd serve. Prove it afterwards with row count + mtime, never file-existence (a read-only SQLite open creates -wal/-shm artifacts).
DSN override is Postgres-only. internal/config/config.go:312-320 documents CANOPY_DB_URL as overriding the DB_* fields — nothing else.
Card store has an independent environment contract. internal/card/database.go:22-31:
go
func DataDir() string {
if dir := os.Getenv("CANOPY_CARD_DATA_DIR"); dir != "" {
return dir
}
home, _ := os.UserHomeDir()
if home == "" { home = "/tmp" }
return filepath.Join(home, ".hermes", "canopy", "cards")
}
It is returned verbatim and is never joined onto $HOME.
The manager is bound at process start. cmd/canopyd/main.go:418:
go
cardDBMgr := card.NewCardDBManager(card.DataDir())
So the variable must be present in the environment before canopyd serve starts; setting it after launch has no effect on that process.
Consequence. The "isolated-stack" recipe (throwaway PG DB + HEAD binary on a spare port) only overrides the DSN:
bash
env -i ... CANOPY_DB_URL="postgres://…/canopy_probe?sslmode=disable" "$BIN" serve
The probe process still runs as the operator, resolves $HOME/.hermes/canopy/cards, and the first POST /api/v1/cards (or any plugin/card-export path) opens compact.db read-write in the live store and appends rows/events. internal/gateway/service.go and the file-viewer root are shared the same way.
The override already exists at this HEAD (DF-HERMES-CANOPY-9, commit b2ca8bb; tests in internal/card/datadir_test.go, docs in docs/SCRATCH_INSTANCE.md:67, README.md:686, AGENTS.md:19). The gap is purely that the lightweight probe recipe never sets it.
Add the three per-store overrides to the probe's launch environment. Minimum change for cards:
+PROBE_STORE="$(mktemp -d /tmp/canopy-probe-store-XXXXXX)"
+
env -i PATH="$PATH" HOME="$HOME" \
HTTP_ADDR="<ip-address>:8099" \
CANOPY_DB_URL="postgres://canopy:canopy@<ip-address>:5437/canopy_probe?sslmode=disable" \
+ CANOPY_CARD_DATA_DIR="$PROBE_STORE/cards" \
+ CANOPY_GATEWAY_STATE_FILE="$PROBE_STORE/gateway/runs.jsonl" \
+ CANOPY_FILE_ROOT="$PROBE_STORE/files" \
JWT_SECRET="dev-secret-change-me" \
"$BIN" serve >"$PROBE_STORE/canopyd.log" 2>&1 &
Notes:
- CANOPY_CARD_DATA_DIR names a directory (one <type>.db per card type); CANOPY_GATEWAY_STATE_FILE names the file runs.jsonl.
- Using a scratch HOME also isolates cards, but the explicit overrides are the stricter form and keep the probe off the shared stores even when it must retain the real $HOME (e.g. tooling that reads other per-user state). See docs/SCRATCH_INSTANCE.md §2 "Store paths".
- canopyd card --data-dir and the CANOPY_CARD_DATA_DIR env var both redirect the card CLI, so export it for any CLI-side card step too.
Drop-in wrapper (reusable as the probe's launch primitive):
launch_isolated_probe() {
local port="$1" dsn="$2" bin="$3"
local store; store="$(mktemp -d /tmp/canopy-probe-store-XXXXXX)"
env -i PATH="$PATH" HOME="$store/home" \
HTTP_ADDR="<ip-address>:$port" \
CANOPY_DB_URL="$dsn" \
CANOPY_CARD_DATA_DIR="$store/cards" \
CANOPY_GATEWAY_STATE_FILE="$store/gateway/runs.jsonl" \
CANOPY_FILE_ROOT="$store/files" \
HERMES_WEBUI_GATEWAY_BASE_URL="http://<ip-address>:9" \
JWT_SECRET="dev-secret-change-me" LOG_FORMAT=json METRICS_ENABLED=false \
"$bin" serve >"$store/canopyd.log" 2>&1 &
echo "$!"; echo "$store" >&2 # caller keeps the store for verification/cleanup
}
Optional hardening (defence in depth, not required by this bug): at cmd/canopyd/main.go:418, log a warning when CANOPY_DB_URL is set and CANOPY_CARD_DATA_DIR is empty and HOME equals the real user home, e.g.
if os.Getenv("CANOPY_CARD_DATA_DIR") == "" && os.Getenv("CANOPY_DB_URL") != "" {
log.Warn().Str("card_dir", card.DataDir()).
Msg("CANOPY_DB_URL is set but the card store still follows $HOME; set CANOPY_CARD_DATA_DIR for an isolated probe")
}
The three required checks are row count, probe-id absence, and mtime ordering against the live compact.db. Do not use file existence: a read-only SQLite open of a WAL store creates compact.db-shm (32 KiB) and a zero-length compact.db-wal; that is not a write (internal/cardexport/export.go:530-537). Also use mode=ro, not immutable=1 (immutable gives a silent stale read when the WAL holds committed data).
LIVE="${LIVE_CARD_DIR:-$HOME/.hermes/canopy/cards}"
PROBE_ID="$1" # id from the POST /api/v1/cards 201 response
PROBE_START="$2" # epoch recorded immediately before canopyd started
# (a) live row count unchanged
sqlite3 "file:$LIVE/compact.db?mode=ro" 'SELECT COUNT(*) FROM cards;'
# (b) probe card id absent from the live store
sqlite3 "file:$LIVE/compact.db?mode=ro" \
"SELECT COUNT(*) FROM cards WHERE id='$PROBE_ID';" # must print 0
# (c) live .db mtime is older than the probe (and unchanged)
test "$(stat -c '%Y' "$LIVE/compact.db")" -lt "$PROBE_START" # must succeed
# positive control: the card really was written — into the probe store
sqlite3 "$CANOPY_CARD_DATA_DIR/compact.db" \
"SELECT id FROM cards WHERE id='$PROBE_ID';" # must print the id
I ran the real HEAD binary against a throwaway canopy_probe DB on :5437, spare port :8099, dev JWT, with a seeded baseline row in the guard store (~/.hermes/canopy/cards/compact.db):
== 2. snapshot the guard baseline
guard cards=1 compact.db mtime=1789772121
== 3. FIXED recipe: probe with CANOPY_CARD_DATA_DIR=/tmp/probe-proof/card-data
fixed probe card id: d6a6759b-83e5-448a-bc8f-5b44ebfffe1c
probe start epoch: 1789772123 (guard mtime 1789772121)
guard compact.db mtime after fixed probe: 1789772121
(a) guard card count after fixed probe: 1 (want 1)
(b) fixed probe id in guard: 0 (want 0)
(c) guard .db mtime < probe start: YES (mtime unchanged: YES)
probe store has the card: 1 (want 1)
== 4. NEGATIVE CONTROL: the same probe WITHOUT the override (buggy recipe)
buggy probe card id: 4fcb3cae-2dbe-4ac2-aea1-3e3b4ae6f910
guard card count after buggy probe: 2 (baseline was 1 -> leak)
buggy probe id in guard: 1
== 5. live :8091 after (read-only)
live API card count after: 5 (before 5)
PASS: buggy recipe leaks into the guard store; fixed recipe writes to the temp store instead
The negative control matters: with the exact same server and DB, removing CANOPY_CARD_DATA_DIR moves the guard store from 1 to 2 cards and makes the probe id present — this is the leak the fix removes.
kill -TERM "$PROBE_PID"; wait "$PROBE_PID" 2>/dev/null
PGPASSWORD=canopy psql -h <ip-address> -p 5437 -U canopy -d postgres \
-c "DROP DATABASE IF EXISTS canopy_probe WITH (FORCE)" # drop ONLY canopy_probe
rm -rf "$PROBE_STORE"
After the run: /health on the live :8091 was still 200 with schema_version 47 == embedded_migrations 47, the live API card count was unchanged (5), canopy_probe was dropped, and :8099 was free.
CANOPY_DB_URL isolates Postgres; the probe also needs CANOPY_CARD_DATA_DIR (and CANOPY_GATEWAY_STATE_FILE, CANOPY_FILE_ROOT) pointing at a mktemp -d, set before canopyd serve, and must be proven with row count + mtime, not file existence.
# Evidence - Problem class: canopy-isolated-stack-must-isolate-card-data-dir - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T22:56:17.285Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Proving a hermes-canopy HTTP fix on the isolated-stack recipe (throwaway PG DB + HEAD binary on a spare port) is NOT sufficient isolation: the card subsystem stores per-type SQLite databases under ~/.hermes/canopy/cards/ and the recipe only overrides CANOPY_DB_URL, so any card/plugin path in the probe writes into the LIVE user card store. Set CANOPY_CARD_DATA_DIR to a temp dir for the probe and verify afterwards by (a) card row count in the live compact.db unchanged, (b) the probe card id absent from it, (c) .db mtimes older than the probe. Note that a read-only sqlite open of the live DB creates 0-byte .db-wal/.db-shm artifacts - that is not a write; prove no write via mtime + row count, not via file existence.", "environment": "hermes-canopy HEAD 628dce6, postgres 5437, probe DB canopy_probe on :8099", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "canopy-isolated-stack-must-isolate-card-data-dir", "provider": "openrouter", "solved_at": "2026-09-18T22:56:17.285Z", "version": ""}I have a complete, independently reproduced proof. Now the solution.
CANOPY_CARD_DATA_DIRScope: hermes-canopy @ HEAD 628dce6, throwaway Postgres DB + HEAD canopyd on a spare port. No source change is required — the override already exists; the probe recipe must use it.
CANOPY_DB_URL only redirects PostgreSQL. The card subsystem resolves its own on-disk store from $CANOPY_CARD_DATA_DIR, falling back to $HOME/.hermes/canopy/cards. A probe that overrides only the DSN keeps the real $HOME, so any card write lands in the operator's live compact.db/expanded.db/iteration.db. Fix: export CANOPY_CARD_DATA_DIR (plus CANOPY_GATEWAY_STATE_FILE and CANOPY_FILE_ROOT) to a mktemp -d before launching canopyd serve. Prove it afterwards with row count + mtime, never file-existence (a read-only SQLite open creates -wal/-shm artifacts).
DSN override is Postgres-only. internal/config/config.go:312-320 documents CANOPY_DB_URL as overriding the DB_* fields — nothing else.
Card store has an independent environment contract. internal/card/database.go:22-31:
go
func DataDir() string {
if dir := os.Getenv("CANOPY_CARD_DATA_DIR"); dir != "" {
return dir
}
home, _ := os.UserHomeDir()
if home == "" { home = "/tmp" }
return filepath.Join(home, ".hermes", "canopy", "cards")
}
It is returned verbatim and is never joined onto $HOME.
The manager is bound at process start. cmd/canopyd/main.go:418:
go
cardDBMgr := card.NewCardDBManager(card.DataDir())
So the variable must be present in the environment before canopyd serve starts; setting it after launch has no effect on that process.
Consequence. The "isolated-stack" recipe (throwaway PG DB + HEAD binary on a spare port) only overrides the DSN:
bash
env -i ... CANOPY_DB_URL="postgres://…/canopy_probe?sslmode=disable" "$BIN" serve
The probe process still runs as the operator, resolves $HOME/.hermes/canopy/cards, and the first POST /api/v1/cards (or any plugin/card-export path) opens compact.db read-write in the live store and appends rows/events. internal/gateway/service.go and the file-viewer root are shared the same way.
The override already exists at this HEAD (DF-HERMES-CANOPY-9, commit b2ca8bb; tests in internal/card/datadir_test.go, docs in docs/SCRATCH_INSTANCE.md:67, README.md:686, AGENTS.md:19). The gap is purely that the lightweight probe recipe never sets it.
Add the three per-store overrides to the probe's launch environment. Minimum change for cards:
+PROBE_STORE="$(mktemp -d /tmp/canopy-probe-store-XXXXXX)"
+
env -i PATH="$PATH" HOME="$HOME" \
HTTP_ADDR="<ip-address>:8099" \
CANOPY_DB_URL="postgres://canopy:canopy@<ip-address>:5437/canopy_probe?sslmode=disable" \
+ CANOPY_CARD_DATA_DIR="$PROBE_STORE/cards" \
+ CANOPY_GATEWAY_STATE_FILE="$PROBE_STORE/gateway/runs.jsonl" \
+ CANOPY_FILE_ROOT="$PROBE_STORE/files" \
JWT_SECRET="dev-secret-change-me" \
"$BIN" serve >"$PROBE_STORE/canopyd.log" 2>&1 &
Notes:
- CANOPY_CARD_DATA_DIR names a directory (one <type>.db per card type); CANOPY_GATEWAY_STATE_FILE names the file runs.jsonl.
- Using a scratch HOME also isolates cards, but the explicit overrides are the stricter form and keep the probe off the shared stores even when it must retain the real $HOME (e.g. tooling that reads other per-user state). See docs/SCRATCH_INSTANCE.md §2 "Store paths".
- canopyd card --data-dir and the CANOPY_CARD_DATA_DIR env var both redirect the card CLI, so export it for any CLI-side card step too.
Drop-in wrapper (reusable as the probe's launch primitive):
launch_isolated_probe() {
local port="$1" dsn="$2" bin="$3"
local store; store="$(mktemp -d /tmp/canopy-probe-store-XXXXXX)"
env -i PATH="$PATH" HOME="$store/home" \
HTTP_ADDR="<ip-address>:$port" \
CANOPY_DB_URL="$dsn" \
CANOPY_CARD_DATA_DIR="$store/cards" \
CANOPY_GATEWAY_STATE_FILE="$store/gateway/runs.jsonl" \
CANOPY_FILE_ROOT="$store/files" \
HERMES_WEBUI_GATEWAY_BASE_URL="http://<ip-address>:9" \
JWT_SECRET="dev-secret-change-me" LOG_FORMAT=json METRICS_ENABLED=false \
"$bin" serve >"$store/canopyd.log" 2>&1 &
echo "$!"; echo "$store" >&2 # caller keeps the store for verification/cleanup
}
Optional hardening (defence in depth, not required by this bug): at cmd/canopyd/main.go:418, log a warning when CANOPY_DB_URL is set and CANOPY_CARD_DATA_DIR is empty and HOME equals the real user home, e.g.
if os.Getenv("CANOPY_CARD_DATA_DIR") == "" && os.Getenv("CANOPY_DB_URL") != "" {
log.Warn().Str("card_dir", card.DataDir()).
Msg("CANOPY_DB_URL is set but the card store still follows $HOME; set CANOPY_CARD_DATA_DIR for an isolated probe")
}
The three required checks are row count, probe-id absence, and mtime ordering against the live compact.db. Do not use file existence: a read-only SQLite open of a WAL store creates compact.db-shm (32 KiB) and a zero-length compact.db-wal; that is not a write (internal/cardexport/export.go:530-537). Also use mode=ro, not immutable=1 (immutable gives a silent stale read when the WAL holds committed data).
LIVE="${LIVE_CARD_DIR:-$HOME/.hermes/canopy/cards}"
PROBE_ID="$1" # id from the POST /api/v1/cards 201 response
PROBE_START="$2" # epoch recorded immediately before canopyd started
# (a) live row count unchanged
sqlite3 "file:$LIVE/compact.db?mode=ro" 'SELECT COUNT(*) FROM cards;'
# (b) probe card id absent from the live store
sqlite3 "file:$LIVE/compact.db?mode=ro" \
"SELECT COUNT(*) FROM cards WHERE id='$PROBE_ID';" # must print 0
# (c) live .db mtime is older than the probe (and unchanged)
test "$(stat -c '%Y' "$LIVE/compact.db")" -lt "$PROBE_START" # must succeed
# positive control: the card really was written — into the probe store
sqlite3 "$CANOPY_CARD_DATA_DIR/compact.db" \
"SELECT id FROM cards WHERE id='$PROBE_ID';" # must print the id
I ran the real HEAD binary against a throwaway canopy_probe DB on :5437, spare port :8099, dev JWT, with a seeded baseline row in the guard store (~/.hermes/canopy/cards/compact.db):
== 2. snapshot the guard baseline
guard cards=1 compact.db mtime=1789772121
== 3. FIXED recipe: probe with CANOPY_CARD_DATA_DIR=/tmp/probe-proof/card-data
fixed probe card id: d6a6759b-83e5-448a-bc8f-5b44ebfffe1c
probe start epoch: 1789772123 (guard mtime 1789772121)
guard compact.db mtime after fixed probe: 1789772121
(a) guard card count after fixed probe: 1 (want 1)
(b) fixed probe id in guard: 0 (want 0)
(c) guard .db mtime < probe start: YES (mtime unchanged: YES)
probe store has the card: 1 (want 1)
== 4. NEGATIVE CONTROL: the same probe WITHOUT the override (buggy recipe)
buggy probe card id: 4fcb3cae-2dbe-4ac2-aea1-3e3b4ae6f910
guard card count after buggy probe: 2 (baseline was 1 -> leak)
buggy probe id in guard: 1
== 5. live :8091 after (read-only)
live API card count after: 5 (before 5)
PASS: buggy recipe leaks into the guard store; fixed recipe writes to the temp store instead
The negative control matters: with the exact same server and DB, removing CANOPY_CARD_DATA_DIR moves the guard store from 1 to 2 cards and makes the probe id present — this is the leak the fix removes.
kill -TERM "$PROBE_PID"; wait "$PROBE_PID" 2>/dev/null
PGPASSWORD=canopy psql -h <ip-address> -p 5437 -U canopy -d postgres \
-c "DROP DATABASE IF EXISTS canopy_probe WITH (FORCE)" # drop ONLY canopy_probe
rm -rf "$PROBE_STORE"
After the run: /health on the live :8091 was still 200 with schema_version 47 == embedded_migrations 47, the live API card count was unchanged (5), canopy_probe was dropped, and :8099 was free.
CANOPY_DB_URL isolates Postgres; the probe also needs CANOPY_CARD_DATA_DIR (and CANOPY_GATEWAY_STATE_FILE, CANOPY_FILE_ROOT) pointing at a mktemp -d, set before canopyd serve, and must be proven with row count + mtime, not file existence.
# Evidence - Problem class: canopy-isolated-stack-must-isolate-card-data-dir - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-18T22:56:17.285Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "Proving a hermes-canopy HTTP fix on the isolated-stack recipe (throwaway PG DB + HEAD binary on a spare port) is NOT sufficient isolation: the card subsystem stores per-type SQLite databases under ~/.hermes/canopy/cards/ and the recipe only overrides CANOPY_DB_URL, so any card/plugin path in the probe writes into the LIVE user card store. Set CANOPY_CARD_DATA_DIR to a temp dir for the probe and verify afterwards by (a) card row count in the live compact.db unchanged, (b) the probe card id absent from it, (c) .db mtimes older than the probe. Note that a read-only sqlite open of the live DB creates 0-byte .db-wal/.db-shm artifacts - that is not a write; prove no write via mtime + row count, not via file existence.", "environment": "hermes-canopy HEAD 628dce6, postgres 5437, probe DB canopy_probe on :8099", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "canopy-isolated-stack-must-isolate-card-data-dir", "provider": "openrouter", "solved_at": "2026-09-18T22:56:17.285Z", "version": ""}