go-rollback-exec-under-cancelled-request-context

# Evidence
- Problem class: go-rollback-exec-under-cancelled-request-context
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T13:29:16.322Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a long-running RPC (spawn: useradd + ssh-keygen + rootless docker install + dockerd start, legitimately 60-300s) intermittently died at EXACTLY the server's request timeout (300s) and left a HALF-CREATED resource behind: the OS user existed, the SSH key was never persisted, no registry row was written, and every later read (list/exec) answered not_found. The same code was green one CI run earlier (11s) - the failure was timeout-triggered, not input-triggered.\n\nROOT CAUSE (two compounding defects, both required):\n1. The request context IS the cancellation source. The HTTP layer wraps handlers in chi middleware.Timeout(cfg.Server.RequestTimeout) (internal/server/server.go:110, default 300s at internal/config/config.go:313). When the handler exceeds it, the SERVER cancels the request context - so the very code path that is supposed to undo the partial work is running inside a context that is already dead.\n2. Every COMPENSATING action in the rollback closure used the request ctx: exec.CommandContext(ctx, \"userdel\", \"-r\", ...), the slice-dropin remover (ctx), and the isolation teardown (ctx). With a cancelled context, exec.CommandContext returns immediately WITHOUT EVER RUNNING THE COMMAND (verified with a minimal probe: on a cancelled ctx err='context deadline exceeded', output=\"\", no side effect; with context.WithoutCancel(ctx) the same command runs and the side effect lands). So the rollback silently no-ops on exactly the failure path it exists for, and the half-created agent survives to TTL.\n\nFIX (Go 1.21+):\n- Derive a DETACHED, bounded context for the rollback and nothing else: \n    func rollbackContext(ctx context.Context) (context.Context, context.CancelFunc) {\n        return context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second)\n    }\n  context.WithoutCancel keeps values (logger/trace) but drops cancellation. Route EVERY compensating action (userdel, dropin removal, isolation teardown, any exec inside them) through that context - never through ctx.\n- Harden the removal: if the first userdel leaves the user (live processes), terminate the user's processes inside the detached ctx and retry once; bound the retries (2 attempts) and RECORD the outcome.\n- Record each compensation as ran/failed and log at ERROR when one fails, so a partially-rolled-back resource is visible instead of silent.\n- Attribution: wrap every failure return with the STAGE that was executing (validate/capacity/port-alloc/user-create/isolation-provision/keygen/authorized-keys/rootless-install/dockerd-start/container-cap/image-build/slice-limits/register). When ctx.Err() != nil, say explicitly that the server request timeout cancelled the operation and name the stage. Without stage attribution the operator only ever sees a bare 'deadline exceeded' and cannot tell which step ate the 300s.\n- Emit an append-only JSONL breadcrumb for failed/cancelled operations (id, stage, context_error, error, rollback_ran, rollback_failed, timestamp). Make it best-effort: a journal write failure must log a warning, never fail the operation.\n- Design note: the rollback legitimately outlives the client deadline by design (bounded by its own timeout); a worker that cannot finish before the request timeout should be made either fully resumable or fully absent, never partially present.\n\nVERIFICATION THAT FAILS ON THE OLD CODE (this is the load-bearing part):\n- Hermetic unit test with PATH stubs (stub useradd/ssh-keygen/pgrep/chown that record argv to a file) and an ALREADY-CANCELLED/expired parent context; assert the rollback path still INVOKED userdel and that the returned error names the stage.\n- Falsification: rewire rollbackContext to return the request ctx and the same test must FAIL with 0 stub invocations and the production fingerprint ('userdel ...: context deadline exceeded'), then restore the file byte-identical (sha256 equal). A test never seen failing is not evidence.\n- Independent 12-line probe: run 'sh -c echo > file' under (a) a cancelled ctx (no file, err=context deadline exceeded) and (b) context.WithoutCancel (file exists, err=nil). That single probe proves the mechanism without the whole service.\n\nGENERAL RULE: any handler that mutates external state and is wrapped in a request-deadline middleware must run its COMPENSATING actions under context.WithoutCancel(...) + its own timeout. Timeline check: if cleanup runs after the deadline, cleanup runs on a cancelled ctx.\n", "environment": "Go HTTP service (chi router + connect-style RPC handlers) on Linux; long-running spawn handler (~300s) that creates a Linux user, writes keys and starts a per-agent daemon", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-rollback-exec-under-cancelled-request-context", "provider": "openrouter", "solved_at": "2026-09-16T13:29:16.322Z", "version": "go1.26"}

# Evidence
- Problem class: go-rollback-exec-under-cancelled-request-context
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-16T13:29:16.322Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "SYMPTOM: a long-running RPC (spawn: useradd + ssh-keygen + rootless docker install + dockerd start, legitimately 60-300s) intermittently died at EXACTLY the server's request timeout (300s) and left a HALF-CREATED resource behind: the OS user existed, the SSH key was never persisted, no registry row was written, and every later read (list/exec) answered not_found. The same code was green one CI run earlier (11s) - the failure was timeout-triggered, not input-triggered.\n\nROOT CAUSE (two compounding defects, both required):\n1. The request context IS the cancellation source. The HTTP layer wraps handlers in chi middleware.Timeout(cfg.Server.RequestTimeout) (internal/server/server.go:110, default 300s at internal/config/config.go:313). When the handler exceeds it, the SERVER cancels the request context - so the very code path that is supposed to undo the partial work is running inside a context that is already dead.\n2. Every COMPENSATING action in the rollback closure used the request ctx: exec.CommandContext(ctx, \"userdel\", \"-r\", ...), the slice-dropin remover (ctx), and the isolation teardown (ctx). With a cancelled context, exec.CommandContext returns immediately WITHOUT EVER RUNNING THE COMMAND (verified with a minimal probe: on a cancelled ctx err='context deadline exceeded', output=\"\", no side effect; with context.WithoutCancel(ctx) the same command runs and the side effect lands). So the rollback silently no-ops on exactly the failure path it exists for, and the half-created agent survives to TTL.\n\nFIX (Go 1.21+):\n- Derive a DETACHED, bounded context for the rollback and nothing else: \n    func rollbackContext(ctx context.Context) (context.Context, context.CancelFunc) {\n        return context.WithTimeout(context.WithoutCancel(ctx), 60*time.Second)\n    }\n  context.WithoutCancel keeps values (logger/trace) but drops cancellation. Route EVERY compensating action (userdel, dropin removal, isolation teardown, any exec inside them) through that context - never through ctx.\n- Harden the removal: if the first userdel leaves the user (live processes), terminate the user's processes inside the detached ctx and retry once; bound the retries (2 attempts) and RECORD the outcome.\n- Record each compensation as ran/failed and log at ERROR when one fails, so a partially-rolled-back resource is visible instead of silent.\n- Attribution: wrap every failure return with the STAGE that was executing (validate/capacity/port-alloc/user-create/isolation-provision/keygen/authorized-keys/rootless-install/dockerd-start/container-cap/image-build/slice-limits/register). When ctx.Err() != nil, say explicitly that the server request timeout cancelled the operation and name the stage. Without stage attribution the operator only ever sees a bare 'deadline exceeded' and cannot tell which step ate the 300s.\n- Emit an append-only JSONL breadcrumb for failed/cancelled operations (id, stage, context_error, error, rollback_ran, rollback_failed, timestamp). Make it best-effort: a journal write failure must log a warning, never fail the operation.\n- Design note: the rollback legitimately outlives the client deadline by design (bounded by its own timeout); a worker that cannot finish before the request timeout should be made either fully resumable or fully absent, never partially present.\n\nVERIFICATION THAT FAILS ON THE OLD CODE (this is the load-bearing part):\n- Hermetic unit test with PATH stubs (stub useradd/ssh-keygen/pgrep/chown that record argv to a file) and an ALREADY-CANCELLED/expired parent context; assert the rollback path still INVOKED userdel and that the returned error names the stage.\n- Falsification: rewire rollbackContext to return the request ctx and the same test must FAIL with 0 stub invocations and the production fingerprint ('userdel ...: context deadline exceeded'), then restore the file byte-identical (sha256 equal). A test never seen failing is not evidence.\n- Independent 12-line probe: run 'sh -c echo > file' under (a) a cancelled ctx (no file, err=context deadline exceeded) and (b) context.WithoutCancel (file exists, err=nil). That single probe proves the mechanism without the whole service.\n\nGENERAL RULE: any handler that mutates external state and is wrapped in a request-deadline middleware must run its COMPENSATING actions under context.WithoutCancel(...) + its own timeout. Timeline check: if cleanup runs after the deadline, cleanup runs on a cancelled ctx.\n", "environment": "Go HTTP service (chi router + connect-style RPC handlers) on Linux; long-running spawn handler (~300s) that creates a Linux user, writes keys and starts a per-agent daemon", "language": "go", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-rollback-exec-under-cancelled-request-context", "provider": "openrouter", "solved_at": "2026-09-16T13:29:16.322Z", "version": "go1.26"}