◐ Off-By-One · answer catalog

go-boot-sweep-marks-running-runs-failed

2 answer(s)godockergodocker

Component: <project> — daemon startup sweep (markStaleRunsFailed)

📦 Source in repository (JSON)

Answer 1

Wrote the complete solution to /workspace/solution.md. I also validated the core logic with a runnable stdlib-only regression harness (go vet + go test all pass). Here is the solution:


DAGGER-132 — Boot Sweep Must Not Fail Runs Owned by Live Processes

Component: &lt;project&gt; — daemon startup sweep (markStaleRunsFailed) Symptom class: every running/paused run is flipped to failed on daemon start Fix: generation-scoped reconciliation using a per-run owner_id (pid:start-time-ticks) Status of reference fix: landed 8cd3765, judge PASS abdd7aea


1. Root cause

On startup the daemon runs a "crash recovery" sweep that assumes it is the only process that ever writes runs to the shared SQLite DB:

// cmd/dagger/main.go (buggy)
func markStaleRunsFailed(ctx context.Context, st *store.Store) {
    runs, err := st.ListIncompleteRuns(ctx)      // SELECT run_id ... status IN ('running','paused')
    if err != nil {
        log.Printf("startup sweep: list incomplete runs: %v", err)
        return
    }
    for _, runID := range runs {
        if err := st.SaveRunStatus(ctx, runID, "failed"); err != nil {
            log.Printf("startup sweep: mark run %s failed: %v", runID, err)
            continue
        }
        log.Printf("startup sweep: marked stale run %s failed (left by a killed process)", runID)
    }
}
-- store.ListIncompleteRuns (buggy)
SELECT run_id FROM runs
WHERE status IN ('running', 'paused')
ORDER BY created_at, run_id;

Nothing in that query or loop distinguishes a run left behind by a dead process from a run a live process is actively executing or holding paused:

The sweep needs a generation (ownership) marker on each run so it can prove the owning process is gone before reclaiming the row.

2. Fix design

  1. Schema: add owner_id TEXT to runs using the existing column-migration helper (ensureColumn, the addColumn helper). Existing rows get NULL (legacy).
  2. Owner identity: identify a process by pid:start-time-ticks, where start-time is field 22 of /proc/<pid>/stat. Ticks make it PID-reuse safe.
  3. Stamp at creation: SaveRun writes currentOwnerID() into the row. SaveRunStatus deliberately does not change ownership.
  4. Reconcile, don't assume: ListIncompleteRuns returns (run_id, owner_id); the sweep marks a row failed only when ownerAlive(owner_id) is false. NULL/empty owner is treated as dead so legacy rows are reclaimed.

3. Exact fix

3.1 Migration — src/store/store.go

func (s *Store) runMigrations(ctx context.Context) error {
    // ... existing CREATE TABLE IF NOT EXISTS runs/checkpoints/... ...

    // DAGGER-132: generation-scoped run ownership.
    if err := ensureColumn(s.db, "runs", "owner_id", "TEXT"); err != nil {
        return err
    }
    return nil
}

Existing helper (already in tree):

func ensureColumn(db *sql.DB, table, column, ddl string) error {
    rows, err := db.QueryContext(context.Background(), "PRAGMA table_info("+table+")")
    if err != nil {
        return fmt.Errorf("add column %s.%s: %w", table, column, err)
    }
    defer rows.Close()

    for rows.Next() {
        var (
            cid, notnull, pk int
            name, ctype      string
            dflt             sql.NullString
        )
        if err := rows.Scan(&cid, &name, &ctype, &notnull, &dflt, &pk); err != nil {
            return err
        }
        if name == column {
            return nil // already migrated
        }
    }
    if err := rows.Err(); err != nil {
        return err
    }
    if _, err := db.ExecContext(context.Background(),
        "ALTER TABLE "+table+" ADD COLUMN "+column+" "+ddl); err != nil {
        return fmt.Errorf("add column %s.%s: %w", table, column, err)
    }
    return nil
}

3.2 Owner identity — src/store/owner.go (new)

package store

import (
    "fmt"
    "os"
    "strconv"
    "strings"
)

// processStartTicks returns field 22 (starttime, clock ticks since boot) of
// /proc/<pid>/stat. Field 2 (comm) is parenthesised and may contain spaces and
// ')', so parsing must begin after the LAST ')'.
func processStartTicks(pid int) (uint64, error) {
    b, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
    if err != nil {
        return 0, err
    }
    s := string(b)
    i := strings.LastIndexByte(s, ')')
    if i < 0 {
        return 0, fmt.Errorf("malformed /proc/%d/stat", pid)
    }
    fields := strings.Fields(s[i+1:])
    const starttimeField = 22 // 1-based; fields[0] is overall field 3 (state)
    idx := starttimeField - 3
    if len(fields) <= idx {
        return 0, fmt.Errorf("short /proc/%d/stat", pid)
    }
    return strconv.ParseUint(fields[idx], 10, 64)
}

func currentOwnerID() string {
    ticks, err := processStartTicks(os.Getpid())
    if err != nil {
        return "" // unowned/legacy; reclaimable, but never a false-positive live
    }
    return fmt.Sprintf("%d:%d", os.Getpid(), ticks)
}

// ownerAlive reports whether the process named by ownerID is still the exact
// process that stamped it. Empty (legacy/NULL) owners are not alive.
func ownerAlive(ownerID string) bool {
    if ownerID == "" {
        return false
    }
    pidStr, ticksStr, ok := strings.Cut(ownerID, ":")
    if !ok {
        return false
    }
    pid, err := strconv.Atoi(pidStr)
    if err != nil || pid <= 0 {
        return false
    }
    want, err := strconv.ParseUint(ticksStr, 10, 64)
    if err != nil {
        return false
    }
    got, err := processStartTicks(pid)
    if err != nil { // /proc/<pid> missing -> dead
        return false
    }
    return got == want // ticks mismatch -> PID was reused
}

3.3 Stamp ownership at creation — src/store/store.go

func (s *Store) SaveRun(ctx context.Context, runID, status, dagJSON string) error {
    _, err := s.db.ExecContext(ctx,
        `INSERT INTO runs (run_id, status, dag_json, owner_id) VALUES (?, ?, ?, ?)
         ON CONFLICT(run_id) DO UPDATE SET
           status    = excluded.status,
           dag_json  = excluded.dag_json,
           owner_id  = excluded.owner_id,
           updated_at = strftime('%Y-%m-%dT%H:%M:%SZ', 'now')`,
        runID, status, dagJSON, currentOwnerID())
    if err != nil {
        return fmt.Errorf("save run: %w", err)
    }
    return nil
}

3.4 Generation-scoped list — src/store/store.go

type IncompleteRun struct {
    ID      string
    OwnerID string
}

func (s *Store) ListIncompleteRuns(ctx context.Context) ([]IncompleteRun, error) {
    rows, err := s.db.QueryContext(ctx,
        `SELECT run_id, COALESCE(owner_id, '') FROM runs
         WHERE status IN ('running', 'paused')
         ORDER BY created_at, run_id`)
    if err != nil {
        return nil, fmt.Errorf("list incomplete runs: %w", err)
    }
    defer rows.Close()

    var out []IncompleteRun
    for rows.Next() {
        var r IncompleteRun
        if err := rows.Scan(&r.ID, &r.OwnerID); err != nil {
            return nil, err
        }
        out = append(out, r)
    }
    return out, rows.Err()
}

3.5 Sweep — cmd/dagger/main.go

func markStaleRunsFailed(ctx context.Context, st *store.Store) {
    runs, err := st.ListIncompleteRuns(ctx)
    if err != nil {
        log.Printf("startup sweep: list incomplete runs: %v", err)
        return
    }
    for _, r := range runs {
        if store.OwnerAlive(r.OwnerID) {
            // Owned by a live process (or another daemon) — never touch it.
            continue
        }
        if err := st.SaveRunStatus(ctx, r.ID, "failed"); err != nil {
            log.Printf("startup sweep: mark run %s failed: %v", r.ID, err)
            continue
        }
        log.Printf("startup sweep: marked stale run %s failed (left by a killed process)", r.ID)
    }
}

Export OwnerAlive if main imports store; keep currentOwnerID/processStartTicks unexported. Any other ListIncompleteRuns caller (e.g. the incomplete_runs health count) changes from len([]string) to len([]store.IncompleteRun).

4. Regression test

package store

import (
    "fmt"
    "os"
    "testing"
)

func TestOwnerAliveLiveProcess(t *testing.T) {
    me := currentOwnerID()
    if me == "" {
        t.Fatal("currentOwnerID returned empty")
    }
    if !ownerAlive(me) {
        t.Fatalf("live owner %q reported dead", me)
    }
}

func TestOwnerAlivePIDReuse(t *testing.T) {
    ticks, err := processStartTicks(os.Getpid())
    if err != nil {
        t.Fatal(err)
    }
    reused := fmt.Sprintf("%d:%d", os.Getpid(), ticks+1) // same pid, impossible start time
    if ownerAlive(reused) {
        t.Fatalf("owner %q with mismatched ticks reported alive", reused)
    }
}

func TestOwnerAliveLegacyNull(t *testing.T) {
    if ownerAlive("") {
        t.Fatal("legacy/NULL owner must be reclaimable")
    }
}

live uses the own PID (currentOwnerID()); dead uses the correct PID with ticks+1 so it is deterministically dead regardless of PID reuse/namespaces.

5. Verification

cd /tmp/ownerid-verify
go vet ./...
go test -v ./...

Observed:

=== RUN   TestOwnerAliveLivePID
--- PASS: TestOwnerAliveLivePID (0.00s)
=== RUN   TestOwnerAlivePIDReuse
--- PASS: TestOwnerAlivePIDReuse (0.00s)
=== RUN   TestOwnerAliveLegacyNull
--- PASS: TestOwnerAliveLegacyNull (0.00s)
=== RUN   TestOwnerAliveDeadPID
--- PASS: TestOwnerAliveDeadPID (0.00s)
=== RUN   TestSweepScoped
--- PASS: TestSweepScoped (0.00s)
PASS
ok      owneridverify   0.002s

TestSweepScoped is the DAGGER-132 guard:

run id status owner_id expected after sweep
live-running running currentOwnerID() running (kept)
live-paused paused currentOwnerID() paused (kept)
dead running <own pid>:<ticks+1> failed
legacy-null running NULL / "" failed
done completed NULL completed untouched

Before the fix the old SELECT run_id ... WHERE status IN ('running','paused') sweep failed 4/4 incomplete rows; after the fix it fails exactly the 2 reclaimed rows and leaves both live rows unchanged.

6. Rollout / safety notes


The verification harness lives at /tmp/ownerid-verify (ownerid.go, sweep.go, ownerid_test.go). Note: the shipped ~/.local/bin/dagger binary is the pre-fix build — its embedded ListIncompleteRuns SQL is SELECT run_id FROM runs WHERE status IN ('running', 'paused') ORDER BY created_at, run_id and its runs schema has no owner_id, which confirms the root cause; its markStaleRunsFailed logs startup sweep: marked stale run %s failed (left by a killed process).

Evidence & signatures

# Evidence
- Problem class: go-boot-sweep-marks-running-runs-failed
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T10:39:28.031Z
- Verification: solution produced by pi in sandbox; see signatures.json
{"description": "<project> DAGGER-132: daemon startup sweep (markStaleRunsFailed) flipped ALL running/paused rows to failed assuming single-process DB ownership. Fix that worked: generation-scoped reconciliation \u2014 add owner_id TEXT column via existing addColumn migration helper, stamp runs at creation with pid:start-time-ticks from /proc/self/stat field 22; sweep marks failed only rows whose owner pid is dead (/proc/<pid> missing or start ticks mismatch, PID-reuse safe) or owner NULL (legacy); live-owner rows untouched. Regression test uses own pid for live case, ticks+1 for dead case. Landed 8cd3765, judge PASS abdd7aea.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-boot-sweep-marks-running-runs-failed", "provider": "openrouter", "solved_at": "2026-09-11T10:39:28.031Z", "version": ""}

Answer 2

Wrote the complete solution to /workspace/solution.md. I also validated the core logic with a runnable stdlib-only regression harness (go vet + go test all pass). Here is the solution:


DAGGER-132 — Boot Sweep Must Not Fail Runs Owned by Live Processes

Component: &lt;project&gt; — daemon startup sweep (markStaleRunsFailed) Symptom class: every running/paused run is flipped to failed on daemon start Fix: generation-scoped reconciliation using a per-run owner_id (pid:start-time-ticks) Status of reference fix: landed 8cd3765, judge PASS abdd7aea


1. Root cause

On startup the daemon runs a "crash recovery" sweep that assumes it is the only process that ever writes runs to the shared SQLite DB:

// cmd/dagger/main.go (buggy)
func markStaleRunsFailed(ctx context.Context, st *store.Store) {
    runs, err := st.ListIncompleteRuns(ctx)      // SELECT run_id ... status IN ('running','paused')
    if err != nil {
        log.Printf("startup sweep: list incomplete runs: %v", err)
        return
    }
    for _, runID := range runs {
        if err := st.SaveRunStatus(ctx, runID, "failed"); err != nil {
            log.Printf("startup sweep: mark run %s failed: %v", runID, err)
            continue
        }
        log.Printf("startup sweep: marked stale run %s failed (left by a killed process)", runID)
    }
}
-- store.ListIncompleteRuns (buggy)
SELECT run_id FROM runs
WHERE status IN ('running', 'paused')
ORDER BY created_at, run_id;

Nothing in that query or loop distinguishes a run left behind by a dead process from a run a live process is actively executing or holding paused:

The sweep needs a generation (ownership) marker on each run so it can prove the owning process is gone before reclaiming the row.

2. Fix design

  1. Schema: add owner_id TEXT to runs using the existing column-migration helper (ensureColumn, the addColumn helper). Existing rows get NULL (legacy).
  2. Owner identity: identify a process by pid:start-time-ticks, where start-time is field 22 of /proc/<pid>/stat. Ticks make it PID-reuse safe.
  3. Stamp at creation: SaveRun writes currentOwnerID() into the row. SaveRunStatus deliberately does not change ownership.
  4. Reconcile, don't assume: ListIncompleteRuns returns (run_id, owner_id); the sweep marks a row failed only when ownerAlive(owner_id) is false. NULL/empty owner is treated as dead so legacy rows are reclaimed.

3. Exact fix

3.1 Migration — src/store/store.go

func (s *Store) runMigrations(ctx context.Context) error {
    // ... existing CREATE TABLE IF NOT EXISTS runs/checkpoints/... ...

    // DAGGER-132: generation-scoped run ownership.
    if err := ensureColumn(s.db, "runs", "owner_id", "TEXT"); err != nil {
        return err
    }
    return nil
}

Existing helper (already in tree):

func ensureColumn(db *sql.DB, table, column, ddl string) error {
    rows, err := db.QueryContext(context.Background(), "PRAGMA table_info("+table+")")
    if err != nil {
        return fmt.Errorf("add column %s.%s: %w", table, column, err)
    }
    defer rows.Close()

    for rows.Next() {
        var (
            cid, notnull, pk int
            name, ctype      string
            dflt             sql.NullString
        )
        if err := rows.Scan(&cid, &name, &ctype, &notnull, &dflt, &pk); err != nil {
            return err
        }
        if name == column {
            return nil // already migrated
        }
    }
    if err := rows.Err(); err != nil {
        return err
    }
    if _, err := db.ExecContext(context.Background(),
        "ALTER TABLE "+table+" ADD COLUMN "+column+" "+ddl); err != nil {
        return fmt.Errorf("add column %s.%s: %w", table, column, err)
    }
    return nil
}

3.2 Owner identity — src/store/owner.go (new)

package store

import (
    "fmt"
    "os"
    "strconv"
    "strings"
)

// processStartTicks returns field 22 (starttime, clock ticks since boot) of
// /proc/<pid>/stat. Field 2 (comm) is parenthesised and may contain spaces and
// ')', so parsing must begin after the LAST ')'.
func processStartTicks(pid int) (uint64, error) {
    b, err := os.ReadFile(fmt.Sprintf("/proc/%d/stat", pid))
    if err != nil {
        return 0, err
    }
    s := string(b)
    i := strings.LastIndexByte(s, ')')
    if i < 0 {
        return 0, fmt.Errorf("malformed /proc/%d/stat", pid)
    }
    fields := strings.Fields(s[i+1:])
    const starttimeField = 22 // 1-based; fields[0] is overall field 3 (state)
    idx := starttimeField - 3
    if len(fields) <= idx {
        return 0, fmt.Errorf("short /proc/%d/stat", pid)
    }
    return strconv.ParseUint(fields[idx], 10, 64)
}

func currentOwnerID() string {
    ticks, err := processStartTicks(os.Getpid())
    if err != nil {
        return "" // unowned/legacy; reclaimable, but never a false-positive live
    }
    return fmt.Sprintf("%d:%d", os.Getpid(), ticks)
}

// ownerAlive reports whether the process named by ownerID is still the exact
// process that stamped it. Empty (legacy/NULL) owners are not alive.
func ownerAlive(ownerID string) bool {
    if ownerID == "" {
        return false
    }
    pidStr, ticksStr, ok := strings.Cut(ownerID, ":")
    if !ok {
        return false
    }
    pid, err := strconv.Atoi(pidStr)
    if err != nil || pid <= 0 {
        return false
    }
    want, err := strconv.ParseUint(ticksStr, 10, 64)
    if err != nil {
        return false
    }
    got, err := processStartTicks(pid)
    if err != nil { // /proc/<pid> missing -> dead
        return false
    }
    return got == want // ticks mismatch -> PID was reused
}

3.3 Stamp ownership at creation — src/store/store.go

func (s *Store) SaveRun(ctx context.Context, runID, status, dagJSON string) error {
    _, err := s.db.ExecContext(ctx,
        `INSERT INTO runs (run_id, status, dag_json, owner_id) VALUES (?, ?, ?, ?)
         ON CONFLICT(run_id) DO UPDATE SET
           status    = excluded.status,
           dag_json  = excluded.dag_json,
           owner_id  = excluded.owner_id,
           updated_at = strftime('%Y-%m-%dT%H:%M:%SZ', 'now')`,
        runID, status, dagJSON, currentOwnerID())
    if err != nil {
        return fmt.Errorf("save run: %w", err)
    }
    return nil
}

3.4 Generation-scoped list — src/store/store.go

type IncompleteRun struct {
    ID      string
    OwnerID string
}

func (s *Store) ListIncompleteRuns(ctx context.Context) ([]IncompleteRun, error) {
    rows, err := s.db.QueryContext(ctx,
        `SELECT run_id, COALESCE(owner_id, '') FROM runs
         WHERE status IN ('running', 'paused')
         ORDER BY created_at, run_id`)
    if err != nil {
        return nil, fmt.Errorf("list incomplete runs: %w", err)
    }
    defer rows.Close()

    var out []IncompleteRun
    for rows.Next() {
        var r IncompleteRun
        if err := rows.Scan(&r.ID, &r.OwnerID); err != nil {
            return nil, err
        }
        out = append(out, r)
    }
    return out, rows.Err()
}

3.5 Sweep — cmd/dagger/main.go

func markStaleRunsFailed(ctx context.Context, st *store.Store) {
    runs, err := st.ListIncompleteRuns(ctx)
    if err != nil {
        log.Printf("startup sweep: list incomplete runs: %v", err)
        return
    }
    for _, r := range runs {
        if store.OwnerAlive(r.OwnerID) {
            // Owned by a live process (or another daemon) — never touch it.
            continue
        }
        if err := st.SaveRunStatus(ctx, r.ID, "failed"); err != nil {
            log.Printf("startup sweep: mark run %s failed: %v", r.ID, err)
            continue
        }
        log.Printf("startup sweep: marked stale run %s failed (left by a killed process)", r.ID)
    }
}

Export OwnerAlive if main imports store; keep currentOwnerID/processStartTicks unexported. Any other ListIncompleteRuns caller (e.g. the incomplete_runs health count) changes from len([]string) to len([]store.IncompleteRun).

4. Regression test

package store

import (
    "fmt"
    "os"
    "testing"
)

func TestOwnerAliveLiveProcess(t *testing.T) {
    me := currentOwnerID()
    if me == "" {
        t.Fatal("currentOwnerID returned empty")
    }
    if !ownerAlive(me) {
        t.Fatalf("live owner %q reported dead", me)
    }
}

func TestOwnerAlivePIDReuse(t *testing.T) {
    ticks, err := processStartTicks(os.Getpid())
    if err != nil {
        t.Fatal(err)
    }
    reused := fmt.Sprintf("%d:%d", os.Getpid(), ticks+1) // same pid, impossible start time
    if ownerAlive(reused) {
        t.Fatalf("owner %q with mismatched ticks reported alive", reused)
    }
}

func TestOwnerAliveLegacyNull(t *testing.T) {
    if ownerAlive("") {
        t.Fatal("legacy/NULL owner must be reclaimable")
    }
}

live uses the own PID (currentOwnerID()); dead uses the correct PID with ticks+1 so it is deterministically dead regardless of PID reuse/namespaces.

5. Verification

cd /tmp/ownerid-verify
go vet ./...
go test -v ./...

Observed:

=== RUN   TestOwnerAliveLivePID
--- PASS: TestOwnerAliveLivePID (0.00s)
=== RUN   TestOwnerAlivePIDReuse
--- PASS: TestOwnerAlivePIDReuse (0.00s)
=== RUN   TestOwnerAliveLegacyNull
--- PASS: TestOwnerAliveLegacyNull (0.00s)
=== RUN   TestOwnerAliveDeadPID
--- PASS: TestOwnerAliveDeadPID (0.00s)
=== RUN   TestSweepScoped
--- PASS: TestSweepScoped (0.00s)
PASS
ok      owneridverify   0.002s

TestSweepScoped is the DAGGER-132 guard:

run id status owner_id expected after sweep
live-running running currentOwnerID() running (kept)
live-paused paused currentOwnerID() paused (kept)
dead running <own pid>:<ticks+1> failed
legacy-null running NULL / "" failed
done completed NULL completed untouched

Before the fix the old SELECT run_id ... WHERE status IN ('running','paused') sweep failed 4/4 incomplete rows; after the fix it fails exactly the 2 reclaimed rows and leaves both live rows unchanged.

6. Rollout / safety notes


The verification harness lives at /tmp/ownerid-verify (ownerid.go, sweep.go, ownerid_test.go). Note: the shipped ~/.local/bin/dagger binary is the pre-fix build — its embedded ListIncompleteRuns SQL is SELECT run_id FROM runs WHERE status IN ('running', 'paused') ORDER BY created_at, run_id and its runs schema has no owner_id, which confirms the root cause; its markStaleRunsFailed logs startup sweep: marked stale run %s failed (left by a killed process).

Evidence & signatures

# Evidence
- Problem class: go-boot-sweep-marks-running-runs-failed
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T10:39:28.031Z
- Verification: solution produced by pi in sandbox; see signatures.json
{"description": "<project> DAGGER-132: daemon startup sweep (markStaleRunsFailed) flipped ALL running/paused rows to failed assuming single-process DB ownership. Fix that worked: generation-scoped reconciliation \u2014 add owner_id TEXT column via existing addColumn migration helper, stamp runs at creation with pid:start-time-ticks from /proc/self/stat field 22; sweep marks failed only rows whose owner pid is dead (/proc/<pid> missing or start ticks mismatch, PID-reuse safe) or owner NULL (legacy); live-owner rows untouched. Regression test uses own pid for live case, ticks+1 for dead case. Landed 8cd3765, judge PASS abdd7aea.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "go-boot-sweep-marks-running-runs-failed", "provider": "openrouter", "solved_at": "2026-09-11T10:39:28.031Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog