go-zombie-cooldown-policy-correction

Answer

Context. The muster zombie daemon wakes stalled nodes after a cooldown. Per the Bane 2026-07-31 matrix, with 0 real pending work the cooldown is 7200s (2h) — the old protocol's 43200s (12h) is obsolete. Three things carried the wrong value and had to be corrected in lockstep: the policy constant (43200), the durable fleet.toml pin (900, a bad override), and a stale CRON_PAUSE_REQUESTED marker that could pin the daemon into a legacy pause.

1. Policy constant corrected 43200 → 7200 (internal/cooldown/policy.go):

// CooldownSeconds is the effective zombie wake cooldown in seconds.
// Bane 2026-07-31 matrix: 0 real pending -> 7200s (NOT 43200 old protocol).
const CooldownSeconds = 7200

// LegacyCooldownSeconds retained only so guards reject regressions.
const LegacyCooldownSeconds = 43200

func Cooldown() time.Duration { return time.Duration(CooldownSeconds) * time.Second } // 2h

2. Surgical fleet.toml pin fix 900 → 7200, durable across daemon restarts (internal/fleetcfg/fleet.go). It rewrites only the value token on the cooldown_seconds line — every comment and unrelated key stays byte-identical, and the value is persisted to the file, so it survives restarts:

func SetPinSurgically(dir string, v int) error {
    data, _ := os.ReadFile(filepath.Join(dir, "fleet.toml"))
    lines := strings.Split(string(data), "\n")
    for i, ln := range lines {
        trimmed := strings.TrimSpace(ln)
        if trimmed == "" || strings.HasPrefix(trimmed, "#") { continue }
        key, val, ok := splitKeyValue(trimmed)
        if !ok || key != PinKey { continue }
        if val == strconv.Itoa(v) { return nil } // already correct
        lines[i] = strings.Replace(ln, val, strconv.Itoa(v), 1) // surgical: this line only
        ...
    }
    return os.WriteFile(path, []byte(strings.Join(lines, "\n")), 0o644)
}

3. Stale marker retired — CRON_PAUSE_REQUESTED removed idempotently (missing file is not an error):

func RetireCRONPauseRequested(dir string) (removed bool, err error) {
    path := filepath.Join(dir, CRONPauseMarker)
    if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) { return false, nil }
    if err := os.Remove(path); err != nil { return false, err }
    return true, nil
}

4. Daemon startup reconciliation (cmd/muster-daemon/main.go) — runs the correction on every boot, making the fix durable and self-healing (drift to 900/43200/overrides is re-pinned to 7200):

func applyStartupCorrection(dir string) error {
    fleetcfg.RetireCRONPauseRequested(dir)             // retire stale marker
    cfg, _ := fleetcfg.Load(dir)
    if cfg.Pin() != cooldown.CooldownSeconds {          // enforce 7200
        fleetcfg.SetPinSurgically(dir, cooldown.CooldownSeconds)
    }
    return nil
}

5. API PUT + GET-verify (GET|PUT /api/v1/fleet/cooldown):

# apply the correction through the API and verify
curl -s -X PUT  http://<ip-address>:38111/api/v1/fleet/cooldown \
     -H 'Content-Type: application/json' -d '{"cooldown_seconds":7200}'
curl -s GET  http://<ip-address>:38111/api/v1/fleet/cooldown
# -> {"cooldown_seconds":7200,"source":"pin","legacy_43200_removed":true}

PUT rejects <=0, missing field, and legacy 43200 (HTTP 400).