go-scheduler-starvation-alert-throttle

Answer

Root cause. CheckStarvation emits a MEDIUM starvation event on every 60s evaluation cycle for each project past 2x cooldown. Because the escalator is constructed fresh per cycle, no in-memory/struct state survives between cycles — the only durable place to remember "we already told the user" is the events table itself. Observed: 179 events/33min across projects.

Fix. A DB-backed per-project throttle: lastStarvationEvent queries the events table filtering severity, component, and json_extract(details, '$.project'); ShouldEmit suppresses re-emission while a matching event exists within a 30-minute window. The DB is the source of truth, so fresh escalators see prior emissions automatically.

const (
    StarvationSeverity    = "MEDIUM"
    StarvationComponent   = "starvation"
    ThrottleWindow        = 30 * time.Minute
)

// StarvationThrottle carries no mutable state on purpose: the escalator is
// rebuilt every cycle, so throttle state must live in the events table.
type StarvationThrottle struct {
    db     *sql.DB
    now    func() time.Time
    window time.Duration
}

// lastStarvationEvent returns the most recent MEDIUM starvation event for
// projectID, or zero time if none exists. Per-project keying comes from
// json_extract(details,'$.project'); severity+component keep unrelated
// alerts (HIGH, cpu-throttle, ...) from suppressing this one.
func (t *StarvationThrottle) lastStarvationEvent(ctx context.Context, projectID string) (time.Time, error) {
    var created string
    err := t.db.QueryRowContext(ctx, `
        SELECT MAX(created_at)
        FROM events
        WHERE severity = ? AND component = ?
          AND json_extract(details, '$.project') = ?`,
        StarvationSeverity, StarvationComponent, projectID,
    ).Scan(&created)
    if err == sql.ErrNoRows || created == "" {
        return time.Time{}, nil          // never emitted for this project -> emit
    }
    if err != nil {
        return time.Time{}, err
    }
    return time.Parse(time.RFC3339Nano, created)
}

// ShouldEmit suppresses re-emission within the 30-minute window (boundary
// inclusive: now - last >= window re-emits).
func (t *StarvationThrottle) ShouldEmit(ctx context.Context, projectID string) (bool, error) {
    last, err := t.lastStarvationEvent(ctx, projectID)
    if err != nil {
        return false, err
    }
    if last.IsZero() {
        return true, nil
    }
    return t.now().Sub(last) >= t.window, nil
}

Integration in CheckStarvation — the throttle is rebuilt per cycle from the DB handle, so the fresh-escalator constraint is satisfied:

func (e *Escalator) CheckStarvation(ctx context.Context, projects []Project) error {
    throttle := NewStarvationThrottle(e.db, e.now, ThrottleWindow)
    for _, p := range projects {
        if !starvedPast2xCooldown(p) {
            continue
        }
        ok, err := throttle.ShouldEmit(ctx, p.ID)
        if err != nil {
            return fmt.Errorf("starvation throttle: %w", err)
        }
        if !ok {
            continue // MEDIUM event exists within 30-min window: suppress
        }
        e.emitEvent(ctx, StarvationSeverity, StarvationComponent,
            map[string]any{"project": p.ID, "reason": "starvation past 2x cooldown"})
    }
    return nil
}

Emission stores created_at as RFC3339 (UTC) so MAX(created_at) is lexicographically correct; an index on (severity, component, created_at) keeps the lookup cheap.