go-scheduler-restart-duplicate-spawn

Answer

Root cause. After a daemon restart the scheduler has two sources of truth for "what is already running":

1. DB running set — persisted rows written by recordStarted() before/at spawn time. cleanDanglingOnStartup (INFRA-012) deliberately keeps pid=0 rows for gateway ticks that were in-flight at crash time, so they survive the restart.
2. In-memory slot pool — live slot occupancy, which is trivially empty right after restart (nothing has been re-spawned yet).

The fallback packer (packer.go) computed its running set only from the slot pool. When fleet.toml declares 0 namespaces the namespace packer — the only packer that reads the DB running set — never runs, so the DB set was never consulted. On the first EVAL, the fallback packer saw 0 running projects, computed free = maxConcurrency - 0, and re-picked a project that was still recorded as running in the DB (pid=0). The daemon double-spawned the in-flight project.

Fix. Merge the evalContext DB running set into the fallback packer's running-set computation. The DB set is authoritative for entries that were persisted but not yet visible in the (restarted) slot pool; the slot pool is authoritative for slots that have already been re-occupied in this process. The merge is by project ID with the DB record winning on conflict, since a persisted running row is exactly the state we must not re-spawn.

Before (packer.go):

// fallbackPacker is used when fleet.toml declares no namespaces.
type fallbackPacker struct {
    slots *SlotPool
}

// RunningSet is a snapshot of projects currently believed to be running.
type RunningSet map[string]*RunRecord // key: projectID

// Pack selects which queued projects to start, respecting maxConcurrency.
func (p *fallbackPacker) Pack(eval *evalContext, queue []*Project) ([]*Project, error) {
    running := p.slots.Running() // in-memory ONLY — empty after restart
    free := eval.cfg.MaxConcurrency - len(running)
    var picked []*Project
    for _, prj := range queue {
        if free <= 0 {
            break
        }
        if _, ok := running[prj.ID]; ok {
            continue // never true after restart → bug
        }
        picked = append(picked, prj)
        free--
    }
    return picked, nil
}

After (the fix):

// fallbackPacker is used when fleet.toml declares no namespaces.
type fallbackPacker struct {
    slots *SlotPool
}

// Pack selects which queued projects to start, respecting maxConcurrency.
// The effective running set is the merge of the in-memory slot pool and the
// persisted DB running set from the evalContext. After a restart the slot
// pool is empty but the DB still holds pid=0 rows for in-flight projects
// (left in place by cleanDanglingOnStartup, INFRA-012); without this merge
// the first EVAL would re-pick and double-spawn them.
func (p *fallbackPacker) Pack(eval *evalContext, queue []*Project) ([]*Project, error) {
    running := mergeRunning(p.slots.Running(), eval.DBRunningSet())
    free := eval.cfg.MaxConcurrency - len(running)
    var picked []*Project
    for _, prj := range queue {
        if free <= 0 {
            break
        }
        if _, ok := running[prj.ID]; ok {
            continue // already running (in-memory or persisted) → skip
        }
        picked = append(picked, prj)
        free--
    }
    return picked, nil
}

// mergeRunning unions two snapshots keyed by project ID. Records present in
// the persisted set win: they describe work that was accepted before the
// restart and must not be spawned again, even though the live slot pool has
// not yet re-accounted for it.
func mergeRunning(live, persisted RunningSet) RunningSet {
    merged := make(RunningSet, len(live)+len(persisted))
    for id, rec := range live {
        merged[id] = rec
    }
    for id, rec := range persisted {
        merged[id] = rec // persisted wins on conflict
    }
    return merged
}

Supporting changes so evalContext exposes the DB set to packers (the namespace packer already consumed it; the fallback packer now does too):

// evalContext carries the persisted scheduler state for one EVAL cycle.
type evalContext struct {
    cfg       Config
    db        *Store
    slots     *SlotPool
    // runningSet is materialized once per EVAL from the DB; it includes
    // pid=0 rows preserved by cleanDanglingOnStartup (INFRA-012).
    runningSet RunningSet
}

// DBRunningSet returns the persisted running set for this EVAL cycle,
// materialized lazily so packers that don't need it (e.g. namespace packer
// reading via fleet namespaces) pay nothing.
func (e *evalContext) DBRunningSet() RunningSet {
    if e.runningSet == nil {
        e.runningSet = e.db.RunningProjects() // SELECT ... WHERE state IN ('running','starting')
    }
    return e.runningSet
}

// EVAL drives one scheduler cycle: build the context, then ask the active
// packer to pick projects, then spawn them.
func (s *Scheduler) EVAL() error {
    eval := &evalContext{cfg: s.cfg, db: s.store, slots: s.slots}
    queue := s.queue.Snapshot()
    picked, err := s.activePacker().Pack(eval, queue)
    if err != nil {
        return err
    }
    return s.spawn(picked)
}

// activePacker selects the namespace packer when fleet.toml defines
// namespaces, otherwise the fallback packer.
func (s *Scheduler) activePacker() Packer {
    if len(s.cfg.Fleet.Namespaces) > 0 {
        return s.namespacePacker
    }
    return s.fallbackPacker // 0 namespaces → this one must now read the DB set
}

The namespace packer is unchanged — it already counted running projects per namespace from the DB, which is why the bug only manifested with an empty fleet.toml.