python-canary-staleness-probe

Answer

Root cause (tick 65): the board subcommand of smoke_check.py only asserted that the last_tick KEY existed. A stalled scheduler never removes the key — it just stops updating it — so the canary stayed green forever while the scheduler was dead.

Fix: three coordinated changes.

1. Staleness probe — the core change in smoke_check.py:

WARN_FACTOR = 1.5
FAIL_FACTOR = 3.0

def staleness_probe(last_tick, cooldown_s, now=None):
    """Return (level, message). level in {"ok","warn","fail"}."""
    now = time.time() if now is None else now

    if last_tick is None:                      # key-existence check kept
        return "fail", "last_tick KEY missing — scheduler never ticked"

    try:
        age = now - float(last_tick)
    except (TypeError, ValueError):
        return "fail", f"last_tick unparsable: {last_tick!r}"

    if age < 0:
        return "warn", f"last_tick is {abs(age):.0f}s in the future (clock skew?)"

    if cooldown_s is None:                     # cannot judge age -> never FAIL
        return "warn", f"cooldown unknown — cannot age-check (age {age:.0f}s)"

    if age > FAIL_FACTOR * cooldown_s:
        return "fail", (f"stalled scheduler: last_tick age {age:.0f}s "
                        f"> {FAIL_FACTOR}x cooldown ({FAIL_FACTOR*cooldown_s:.0f}s)")
    if age > WARN_FACTOR * cooldown_s:
        return "warn", (f"slow scheduler: last_tick age {age:.0f}s "
                        f"> {WARN_FACTOR}x cooldown ({WARN_FACTOR*cooldown_s:.0f}s)")
    return "ok", f"fresh: last_tick age {age:.0f}s <= {WARN_FACTOR}x cooldown"

2. Cooldown resolution — live first, header as fallback (the pin-proofing):

def resolve_cooldown(live_cooldown_s, header_cooldown_s):
    """LIVE scheduler cooldown (2nd arg from smoke-check.sh §1) wins;
    header cooldown_s is the fallback. Rejects 0/negative/unparseable."""
    for label, raw in (("live", live_cooldown_s), ("header", header_cooldown_s)):
        if raw is None:
            continue
        try:
            cd = float(raw)
        except (TypeError, ValueError):
            continue
        if cd > 0:
            return cd
    return None

cmd_board becomes: cooldown_s = resolve_cooldown(args.cooldown, header.get("cooldown_s")), then level, msg = staleness_probe(data.get("last_tick"), cooldown_s) and return EXIT_FAIL if level == "fail" else EXIT_OK (WARN does not fail the canary; the board subcommand now takes an optional second positional arg).

3. smoke-check.sh §1 plumbing — the live cooldown is discovered and passed as the second arg:

# smoke-check.sh §1: live scheduler cooldown, operator-pin aware
live_cd="$(scheduler_get_cooldown "$board")"     # pin -> 21600, else 7200
python3 smoke_check.py board "$board" "$live_cd"

Why this matters: the operator pin holds the live cooldown at 21600s (6h) while the baked header still says 7200s (2h). Header-only thresholds would FAIL a board that ticked 8.3h ago (30000 > 3×7200 = 21600); live thresholds correctly PASS (30000 < 1.5×21600 = 32400). No false FAIL.