◐ Off-By-One

bash-cgroup-v2-psi-pressure-hysteresis-governor

2 answer(s)shellbashshellbash

A bash monitor that samples cpu.pressure, memory.pressure, and io.pressure for a target cgroup and drives a two-threshold hysteretic state machine: enter THROTTLED when avg10 > HIGH, and return to HEALTHY only after avg60 has stayed below LOW for a full dwell window. It tolerates partial reads, transient ENOENT, and avg resets to 0; recovers in-progress dwell state from an append-only log across restarts; and on SIGTERM flushes a final record and releases its lock.

📦 Source in repository (JSON)

Answer 1

Files written and verified at ~/psi-governor/ (psi-governor.sh, test-psi-governor.sh, SOLUTION.md). Full solution below.

cgroup v2 PSI Pressure Hysteresis Governor — Diagnosis & Verified Fix

A bash monitor that samples cpu.pressure, memory.pressure, and io.pressure for a target cgroup and drives a two-threshold hysteretic state machine: enter THROTTLED when avg10 > HIGH, and return to HEALTHY only after avg60 has stayed below LOW for a full dwell window. It tolerates partial reads, transient ENOENT, and avg resets to 0; recovers in-progress dwell state from an append-only log across restarts; and on SIGTERM flushes a final record and releases its lock.

1. Root-cause analysis

Five distinct failure modes make the "obvious" implementation wrong.

1.1 Naive two-threshold logic oscillates

The intuitive loop compares avg10 > HIGH and avg10 < LOW directly. avg10 is a short 10 s exponentially weighted average; under load hovering near a threshold it crosses and re-crosses repeatedly, emitting a transition on every crossing. Fix: asymmetric hysteresis — enter on fast avg10 > HIGH, leave only after slow avg60 stays < LOW for a dwell window. A single noisy sample moves avg10 but cannot move avg60 below LOW and survive the whole window.

1.2 Truncated / partial PSI reads

PSI files are seq_files; a read racing teardown can return empty, a full-only line, or a torn line missing avg60. Parsing by field index yields an empty string; arithmetic then errors under set -e or coerces to 0, which falsely looks like "no pressure" and can clear THROTTLED. Fix: parse avg10/avg60 by token, skip a resource unless both are present, and treat "no usable sample" as hold state.

1.3 Transient `ENOENT` during cgroup rename

During reclaim-rename a pressure file momentarily disappears. This must not be fatal and must not change state; unreadable resources are skipped.

1.4 Avg window reset to `0`

avg10=0.00 avg60=0.00 is a legitimate low sample (both tokens present) and must be accepted as low — distinguished from truncation by token presence, not value.

1.5 Restart mid-dwell re-fires / loses progress

In-memory-only state is amnesiac on restart: it defaults to HEALTHY, then re-reads pressure and either emits a duplicate entry transition or restarts the dwell clock. Fix: a durable append-only log with TRANSITION, DWELL, DWELL_RESET, FINAL records, replayed on startup.

1.6 `SIGTERM` with no trap leaves stale state

Without a trap the process dies mid-record and orphans its pidfile. Fix: trap TERM/INT, emit a final record, and remove the pidfile only if it names this process; reclaim a stale pidfile (dead owner) on startup.

2. Exact fix

2.1 `psi-governor.sh`

#!/usr/bin/env bash
# psi-governor.sh — cgroup v2 PSI pressure hysteresis governor
#
# Samples cpu.pressure / memory.pressure / io.pressure for a target cgroup
# and drives a two-threshold state machine:
#
#   * HEALTHY   -> THROTTLED  when max(avg10) > HIGH
#   * THROTTLED -> HEALTHY    only after max(avg60) has stayed < LOW for a
#                             full dwell window (so one noisy sample can
#                             never oscillate the state)
#
# It is tolerant of malformed/partial PSI files, transient ENOENT (cgroup
# being renamed), and avg windows reset to 0.  State is recovered from an
# append-only transition log so a restart in the middle of a dwell window
# neither re-fires a transition nor loses dwell progress.  SIGTERM flushes a
# final record and releases the owner pidfile.
#
# Exit status: 0 normal, 1 already running / bad config.

set -u

# --------------------------------------------------------------------------
# Configuration (all overridable via environment)
# --------------------------------------------------------------------------
CGROUP_DIR=${PSI_CGROUP_DIR:-/sys/fs/cgroup}
HIGH=${PSI_HIGH:-10.0}                 # avg10 % to enter THROTTLED
LOW=${PSI_LOW:-5.0}                    # avg60 % that must stay low to exit
DWELL=${PSI_DWELL:-30}                 # seconds avg60 must stay < LOW
POLL=${PSI_POLL_INTERVAL:-1}           # seconds between samples
LOG=${PSI_LOG:-}                       # append-only transition log (optional)
PIDFILE=${PSI_PIDFILE:-${LOG:+${LOG}.pid}}
FILES=(cpu.pressure memory.pressure io.pressure)

STATE=HEALTHY
DWELL_START=""
MAX10=""
MAX60=""
FINALIZED=0

# --------------------------------------------------------------------------
# Helpers
# --------------------------------------------------------------------------
now() { date +%s; }

gt() { awk -v a="$1" -v b="$2" 'BEGIN{exit !(a>b)}'; }
lt() { awk -v a="$1" -v b="$2" 'BEGIN{exit !(a<b)}'; }
maxf() { awk -v a="$1" -v b="$2" 'BEGIN{print (b>a)?b:a}'; }

# Append a record to the log (best-effort, never fatal).
logevent() {
    [ -n "$LOG" ] || return 0
    printf '%s %s\n' "$(now)" "$*" >>"$LOG"
}

# Read the three PSI files and set MAX10/MAX60 to the max across resources.
# Returns 1 when there is no usable sample (all files missing/partial).
read_pressures() {
    local f line v10 v60
    MAX10=""
    MAX60=""
    for f in "${FILES[@]}"; do
        # `some` is the first line on every kernel that exposes these files.
        # A truncated/partial write may drop it or the avg60 token; skip.
        line=$(awk '/^some[[:space:]]/{print; exit}' "$CGROUP_DIR/$f" 2>/dev/null) || true
        [ -n "${line:-}" ] || continue
        v10=$(printf '%s\n' "$line" | awk '{for(i=1;i<=NF;i++){if($i ~ /^avg10=/){sub(/^avg10=/,"",$i);print $i;exit}}}')
        v60=$(printf '%s\n' "$line" | awk '{for(i=1;i<=NF;i++){if($i ~ /^avg60=/){sub(/^avg60=/,"",$i);print $i;exit}}}')
        [ -n "${v10:-}" ] && [ -n "${v60:-}" ] || continue
        if [ -z "$MAX10" ]; then
            MAX10=$v10
            MAX60=$v60
        else
            MAX10=$(maxf "$MAX10" "$v10")
            MAX60=$(maxf "$MAX60" "$v60")
        fi
    done
    [ -n "$MAX10" ]
}

# Rebuild STATE and DWELL_START from the append-only log.  Records are
# processed in order so the last transition and the dwell marker that
# belongs to it win.
recover_state() {
    [ -n "$LOG" ] && [ -s "$LOG" ] || return 0
    local ts ev rest k v
    STATE=HEALTHY
    DWELL_START=""
    while read -r ts ev rest; do
        case $ev in
            TRANSITION)
                # rest: from=... to=... max10=... max60=...
                k=${rest#*to=}; v=${k%% *}
                STATE=$v
                DWELL_START=""
                ;;
            DWELL)
                # rest: since=<epoch> ...
                k=${rest#*since=}; v=${k%% *}
                if [ "$STATE" = THROTTLED ]; then
                    DWELL_START=$v
                fi
                ;;
            DWELL_RESET)
                DWELL_START=""
                ;;
            *) : ;;
        esac
    done <"$LOG"
}

transition() {
    local to=$1
    printf '%s %s %s %s->%s max10=%s max60=%s\n' \
        "$(now)" "TRANSITION" "from=$STATE" "$STATE" "$to" "$MAX10" "$MAX60"
    logevent "TRANSITION from=$STATE to=$to max10=$MAX10 max60=$MAX60"
    if [ "$to" = HEALTHY ]; then
        STATE=HEALTHY
    else
        STATE=THROTTLED
    fi
    DWELL_START=""
}

# --------------------------------------------------------------------------
# Lock / ownership (stale pidfile is reclaimed, never left behind)
# --------------------------------------------------------------------------
acquire_lock() {
    [ -n "$PIDFILE" ] || return 0
    if [ -e "$PIDFILE" ]; then
        local old
        old=$(cat "$PIDFILE" 2>/dev/null || true)
        if [ -n "$old" ] && kill -0 "$old" 2>/dev/null; then
            echo "psi-governor: already running as pid $old" >&2
            exit 1
        fi
        rm -f "$PIDFILE"
    fi
    printf '%s\n' "$$" >"$PIDFILE"
}

release_lock() {
    [ -n "$PIDFILE" ] || return 0
    if [ "$(cat "$PIDFILE" 2>/dev/null || true)" = "$$" ]; then
        rm -f "$PIDFILE"
    fi
}

finalize() {
    [ "$FINALIZED" -eq 1 ] && return 0
    FINALIZED=1
    printf '%s FINAL state=%s max10=%s max60=%s\n' \
        "$(now)" "$STATE" "${MAX10:-?}" "${MAX60:-?}"
    logevent "FINAL state=$STATE max10=${MAX10:-?} max60=${MAX60:-?}"
    release_lock
}

on_term() {
    finalize
    exit 0
}

# --------------------------------------------------------------------------
# Main
# --------------------------------------------------------------------------
acquire_lock
recover_state
trap on_term TERM INT

while :; do
    if read_pressures; then
        case $STATE in
            HEALTHY)
                if gt "$MAX10" "$HIGH"; then
                    transition THROTTLED
                fi
                ;;
            THROTTLED)
                if lt "$MAX60" "$LOW"; then
                    t=$(now)
                    if [ -z "$DWELL_START" ]; then
                        DWELL_START=$t
                        logevent "DWELL since=$DWELL_START max10=$MAX10 max60=$MAX60"
                    elif [ $((t - DWELL_START)) -ge "$DWELL" ]; then
                        transition HEALTHY
                    fi
                elif [ -n "$DWELL_START" ]; then
                    # avg60 climbed back above LOW: restart the dwell clock.
                    DWELL_START=""
                    logevent "DWELL_RESET max10=$MAX10 max60=$MAX60"
                fi
                ;;
        esac
    fi
    # Missing/partial sample: hold current state and dwell progress.
    sleep "$POLL"
done

2.2 Key design points

Concern	Mechanism
Oscillation	Entry on `avg10 > HIGH`; exit only on `avg60 < LOW` for `DWELL` seconds.
Noisy sample during dwell	Dwell aborts only when `avg60 ≥ LOW`, so an `avg10` spike that leaves `avg60` low does not abort.
Partial read	Require both `avg10=` and `avg60=` tokens; otherwise skip the resource.
No data at all	`read_pressures` returns 1; loop holds state and dwell progress.
`ENOENT`	`2>/dev/null \|\| true`; missing file simply skipped.
Avg reset to `0`	`0.00` is a valid, low sample; dwell proceeds.
Restart mid-dwell	`recover_state` replays `TRANSITION`/`DWELL`/`DWELL_RESET` in order.
`SIGTERM`	`finalize` writes `FINAL`, removes pidfile iff owned by `$$`.
Stale owner	`acquire_lock` reclaims a pidfile whose pid is not alive.

Log format (append-only, space separated):

<epoch> TRANSITION from=HEALTHY to=THROTTLED max10=40.0 max60=40.0
<epoch> DWELL since=<epoch> max10=1.0 max60=1.0
<epoch> DWELL_RESET max10=1.0 max60=20.0
<epoch> FINAL state=HEALTHY max10=0.0 max60=0.0

2.3 Running it

PSI_CGROUP_DIR=/sys/fs/cgroup/my.slice \
PSI_HIGH=10 PSI_LOW=5 PSI_DWELL=30 PSI_POLL_INTERVAL=1 \
PSI_LOG=/var/log/psi-governor.log \
./psi-governor.sh

Transitions print to stdout and append to $PSI_LOG; $PSI_LOG.pid is the owner file.

3. Verification

The harness builds a synthetic fixture cgroup tree (three writable *.pressure files), rewrites it between polls, and asserts the exact emitted timeline plus no duplicates across a kill/restart performed mid-dwell.

3.1 `test-psi-governor.sh`

#!/usr/bin/env bash
# test-psi-governor.sh — synthetic fixture verification for psi-governor.sh
set -u

HERE=$(cd "$(dirname "$0")" && pwd)
GOV="$HERE/psi-governor.sh"
TMP=$(mktemp -d)
CGROUP="$TMP/cgroup"
LOG="$TMP/transitions.log"
PIDFILE="$TMP/gov.pid"
OUT="$TMP/out.log"
ERR="$TMP/err.log"
mkdir -p "$CGROUP"
: >"$LOG"

fail() { echo "FAIL: $*" >&2; echo "--- log ---" >&2; cat "$LOG" >&2; echo "--- err ---" >&2; cat "$ERR" >&2; exit 1; }
pass() { echo "PASS: $*"; }

write_pressure() { # avg10 avg60
    local a10=$1 a60=$2 f
    for f in cpu.pressure memory.pressure io.pressure; do
        printf 'some avg10=%s avg60=%s avg300=0.00 total=0\n' "$a10" "$a60" >"$CGROUP/.$f.tmp"
        mv "$CGROUP/.$f.tmp" "$CGROUP/$f"
    done
}

start_gov() {
    PSI_CGROUP_DIR="$CGROUP" PSI_HIGH=10 PSI_LOW=5 PSI_DWELL=3 \
        PSI_POLL_INTERVAL=0.2 PSI_LOG="$LOG" PSI_PIDFILE="$PIDFILE" \
        "$GOV" >"$OUT" 2>>"$ERR" &
    GOV_PID=$!
}

stop_gov() { # graceful
    kill -TERM "$GOV_PID" 2>/dev/null || true
    wait "$GOV_PID" 2>/dev/null || true
}

timeline() { awk '$2=="TRANSITION"{f="";t="";for(i=3;i<=NF;i++){if($i~/^from=/){sub("from=","",$i);f=$i}if($i~/^to=/){sub("to=","",$i);t=$i}}print f"->"t}' "$LOG"; }

cleanup() { stop_gov; rm -rf "$TMP"; }
trap cleanup EXIT

# --- Setup: healthy fixture -------------------------------------------------
write_pressure 0 0
start_gov
sleep 0.6

# --- Tolerance: transient ENOENT + truncated 'some'-only file --------------
rm -f "$CGROUP/io.pressure"
printf 'some avg10=0.00\n' >"$CGROUP/cpu.pressure"      # truncated: no avg60
printf 'not a psi line\n' >"$CGROUP/memory.pressure"    # garbage
sleep 0.6
kill -0 "$GOV_PID" 2>/dev/null || fail "governor died on malformed input"
write_pressure 0 0
sleep 0.4
[ -s "$LOG" ] && [ ! -s "$OUT" ] || true
if [ -n "$(timeline)" ]; then fail "tolerance window produced spurious transitions"; fi
pass "tolerates ENOENT, truncated and malformed PSI files"

# --- Enter THROTTLED --------------------------------------------------------
write_pressure 40 40
sleep 0.6
[ "$(timeline)" = "HEALTHY->THROTTLED" ] \
    || fail "expected single HEALTHY->THROTTLED, got: $(timeline | tr '\n' ',')"
pass "entered THROTTLED on avg10 > high"

# --- Start dwell, then kill mid-dwell and restart ---------------------------
write_pressure 1 1
sleep 0.8   # stay < LOW; dwell clock running (DWELL=3)
grep -q ' DWELL ' "$LOG" || fail "dwell marker not written"
stop_gov
grep -q ' FINAL ' "$LOG" || fail "SIGTERM did not flush final record"
[ -e "$PIDFILE" ] && fail "pidfile left behind after SIGTERM"
pass "SIGTERM flushed FINAL record and released pidfile"

# Restart: state and dwell progress must be recovered from the log.
: >"$OUT"
start_gov
sleep 0.6
[ "$(timeline | tail -1)" = "HEALTHY->THROTTLED" ] \
    || fail "restart re-fired a transition: $(timeline | tr '\n' ',')"
pass "restart mid-dwell did not re-fire a transition"

# Wait out the remainder of the dwell window.
sleep 4.5
stop_gov

# --- Assert exact timeline, no duplicates -----------------------------------
EXPECTED=$'HEALTHY->THROTTLED\nTHROTTLED->HEALTHY'
ACTUAL=$(timeline)
[ "$ACTUAL" = "$EXPECTED" ] || fail "timeline mismatch; expected [$EXPECTED] got [$ACTUAL]"
[ "$(printf '%s\n' "$ACTUAL" | grep -c '^HEALTHY->THROTTLED$')" -eq 1 ] || fail "duplicated entry transition"
[ "$(printf '%s\n' "$ACTUAL" | grep -c '^THROTTLED->HEALTHY$')" -eq 1 ] || fail "duplicated exit transition"
pass "exact transition timeline with no duplicates across kill/restart"

# --- Dwell reset when avg60 climbs back above LOW ---------------------------
: >"$LOG"
write_pressure 0 0
start_gov
sleep 0.4
write_pressure 40 40          # enter THROTTLED
sleep 0.6
write_pressure 1 1            # start dwell
sleep 0.8
grep -q ' DWELL ' "$LOG" || fail "dwell marker not written (reset phase)"
write_pressure 1 20           # avg10 low but avg60 > LOW -> abort dwell
sleep 0.6
grep -q ' DWELL_RESET ' "$LOG" || fail "dwell was not reset when avg60 rose above LOW"
write_pressure 0 0            # avg window reset to 0 -> legitimate low sample
sleep 4.5
stop_gov
EXPECTED2=$'HEALTHY->THROTTLED\nTHROTTLED->HEALTHY'
ACTUAL2=$(timeline)
[ "$ACTUAL2" = "$EXPECTED2" ] || fail "reset-phase timeline mismatch: [$ACTUAL2]"
pass "dwell resets on avg60 > LOW and avg reset to 0 is treated as low"

# --- Stale-owner lock reclamation ------------------------------------------
printf '999999\n' >"$PIDFILE"   # dead pid
: >"$LOG"
write_pressure 0 0
start_gov
sleep 0.5
kill -0 "$GOV_PID" 2>/dev/null || fail "refused to reclaim stale pidfile"
stop_gov
[ -e "$PIDFILE" ] && fail "pidfile left after clean exit"
pass "reclaimed stale owner and released lock on exit"

echo "ALL TESTS PASSED"

3.2 Running the verification

chmod +x psi-governor.sh test-psi-governor.sh
./test-psi-governor.sh

Observed output (reproduced across repeated runs):

PASS: tolerates ENOENT, truncated and malformed PSI files
PASS: entered THROTTLED on avg10 > high
PASS: SIGTERM flushed FINAL record and released pidfile
PASS: restart mid-dwell did not re-fire a transition
PASS: exact transition timeline with no duplicates across kill/restart
PASS: dwell resets on avg60 > LOW and avg reset to 0 is treated as low
PASS: reclaimed stale owner and released lock on exit
ALL TESTS PASSED

What each assertion proves:

Tolerance — with io.pressure removed, cpu.pressure truncated to a some-only line (no avg60), and memory.pressure garbage, the governor stays alive and emits no spurious transitions.
Entry — a 40/40 fixture yields exactly one HEALTHY->THROTTLED.
Clean shutdown — SIGTERM produces FINAL and the pidfile is gone.
Restart safety — killed 0.8 s into a 3 s dwell, restarted, it does not re-emit the entry transition (state and dwell start recovered).
Exact timeline — the log reduces to exactly HEALTHY->THROTTLED, THROTTLED->HEALTHY, each once, despite a kill/restart mid-dwell.
Dwell reset — an avg60 rise writes DWELL_RESET; a subsequent avg60=0 is accepted and a single exit transition fires.
Stale owner — a pidfile naming a dead pid is reclaimed on startup and removed on exit.

4. Summary

The fix is a small, dependency-free bash governor built on three ideas: asymmetric hysteresis (avg10 to enter, avg60 + dwell to leave), defensive sampling (token-presence parsing, hold-not-clear on missing data), and durable, replayable state (append-only log with explicit DWELL/DWELL_RESET markers plus a SIGTERM/stale-lock protocol). The synthetic-fixture harness exercises every failure mode and asserts the exact transition timeline, including across a kill/restart in the middle of a dwell window.

Evidence & signatures

# Evidence
- Problem class: bash-cgroup-v2-psi-pressure-hysteresis-governor
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T16:05:28.716Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Implement a bash monitor that samples cgroup v2 PSI files (cpu.pressure, memory.pressure, io.pressure) for a target cgroup and drives a two-threshold hysteresis state machine: enter THROTTLED when avg10 exceeds the high watermark, and return to HEALTHY only after avg60 has stayed below the low watermark for a full dwell window, so a single noisy sample can never cause an oscillation. The script must tolerate truncated or partial reads (a pressure file with only the 'some' line, a transient ENOENT while a cgroup is being reclaim-renamed, an avg window reset to 0), recover in-progress state from its append-only transition log so a restart mid-dwell does not re-fire a transition, and on SIGTERM flush a final transition record and release its lock without leaving a stale owner. Verify with a synthetic fixture cgroup tree that you rewrite between polls, asserting the exact emitted transition timeline plus no duplicated transitions across a kill/restart cycle performed in the middle of a dwell window.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-cgroup-v2-psi-pressure-hysteresis-governor", "provider": "openrouter", "solved_at": "2026-09-11T16:05:28.716Z", "version": "5.2"}

Answer 2

Files written and verified at ~/psi-governor/ (psi-governor.sh, test-psi-governor.sh, SOLUTION.md). Full solution below.

cgroup v2 PSI Pressure Hysteresis Governor — Diagnosis & Verified Fix

1. Root-cause analysis

Five distinct failure modes make the "obvious" implementation wrong.

1.1 Naive two-threshold logic oscillates

1.2 Truncated / partial PSI reads

1.3 Transient `ENOENT` during cgroup rename

During reclaim-rename a pressure file momentarily disappears. This must not be fatal and must not change state; unreadable resources are skipped.

1.4 Avg window reset to `0`

avg10=0.00 avg60=0.00 is a legitimate low sample (both tokens present) and must be accepted as low — distinguished from truncation by token presence, not value.

1.5 Restart mid-dwell re-fires / loses progress

1.6 `SIGTERM` with no trap leaves stale state

2. Exact fix

2.1 `psi-governor.sh`

#!/usr/bin/env bash
# psi-governor.sh — cgroup v2 PSI pressure hysteresis governor
#
# Samples cpu.pressure / memory.pressure / io.pressure for a target cgroup
# and drives a two-threshold state machine:
#
#   * HEALTHY   -> THROTTLED  when max(avg10) > HIGH
#   * THROTTLED -> HEALTHY    only after max(avg60) has stayed < LOW for a
#                             full dwell window (so one noisy sample can
#                             never oscillate the state)
#
# It is tolerant of malformed/partial PSI files, transient ENOENT (cgroup
# being renamed), and avg windows reset to 0.  State is recovered from an
# append-only transition log so a restart in the middle of a dwell window
# neither re-fires a transition nor loses dwell progress.  SIGTERM flushes a
# final record and releases the owner pidfile.
#
# Exit status: 0 normal, 1 already running / bad config.

set -u

# --------------------------------------------------------------------------
# Configuration (all overridable via environment)
# --------------------------------------------------------------------------
CGROUP_DIR=${PSI_CGROUP_DIR:-/sys/fs/cgroup}
HIGH=${PSI_HIGH:-10.0}                 # avg10 % to enter THROTTLED
LOW=${PSI_LOW:-5.0}                    # avg60 % that must stay low to exit
DWELL=${PSI_DWELL:-30}                 # seconds avg60 must stay < LOW
POLL=${PSI_POLL_INTERVAL:-1}           # seconds between samples
LOG=${PSI_LOG:-}                       # append-only transition log (optional)
PIDFILE=${PSI_PIDFILE:-${LOG:+${LOG}.pid}}
FILES=(cpu.pressure memory.pressure io.pressure)

STATE=HEALTHY
DWELL_START=""
MAX10=""
MAX60=""
FINALIZED=0

# --------------------------------------------------------------------------
# Helpers
# --------------------------------------------------------------------------
now() { date +%s; }

gt() { awk -v a="$1" -v b="$2" 'BEGIN{exit !(a>b)}'; }
lt() { awk -v a="$1" -v b="$2" 'BEGIN{exit !(a<b)}'; }
maxf() { awk -v a="$1" -v b="$2" 'BEGIN{print (b>a)?b:a}'; }

# Append a record to the log (best-effort, never fatal).
logevent() {
    [ -n "$LOG" ] || return 0
    printf '%s %s\n' "$(now)" "$*" >>"$LOG"
}

# Read the three PSI files and set MAX10/MAX60 to the max across resources.
# Returns 1 when there is no usable sample (all files missing/partial).
read_pressures() {
    local f line v10 v60
    MAX10=""
    MAX60=""
    for f in "${FILES[@]}"; do
        # `some` is the first line on every kernel that exposes these files.
        # A truncated/partial write may drop it or the avg60 token; skip.
        line=$(awk '/^some[[:space:]]/{print; exit}' "$CGROUP_DIR/$f" 2>/dev/null) || true
        [ -n "${line:-}" ] || continue
        v10=$(printf '%s\n' "$line" | awk '{for(i=1;i<=NF;i++){if($i ~ /^avg10=/){sub(/^avg10=/,"",$i);print $i;exit}}}')
        v60=$(printf '%s\n' "$line" | awk '{for(i=1;i<=NF;i++){if($i ~ /^avg60=/){sub(/^avg60=/,"",$i);print $i;exit}}}')
        [ -n "${v10:-}" ] && [ -n "${v60:-}" ] || continue
        if [ -z "$MAX10" ]; then
            MAX10=$v10
            MAX60=$v60
        else
            MAX10=$(maxf "$MAX10" "$v10")
            MAX60=$(maxf "$MAX60" "$v60")
        fi
    done
    [ -n "$MAX10" ]
}

# Rebuild STATE and DWELL_START from the append-only log.  Records are
# processed in order so the last transition and the dwell marker that
# belongs to it win.
recover_state() {
    [ -n "$LOG" ] && [ -s "$LOG" ] || return 0
    local ts ev rest k v
    STATE=HEALTHY
    DWELL_START=""
    while read -r ts ev rest; do
        case $ev in
            TRANSITION)
                # rest: from=... to=... max10=... max60=...
                k=${rest#*to=}; v=${k%% *}
                STATE=$v
                DWELL_START=""
                ;;
            DWELL)
                # rest: since=<epoch> ...
                k=${rest#*since=}; v=${k%% *}
                if [ "$STATE" = THROTTLED ]; then
                    DWELL_START=$v
                fi
                ;;
            DWELL_RESET)
                DWELL_START=""
                ;;
            *) : ;;
        esac
    done <"$LOG"
}

transition() {
    local to=$1
    printf '%s %s %s %s->%s max10=%s max60=%s\n' \
        "$(now)" "TRANSITION" "from=$STATE" "$STATE" "$to" "$MAX10" "$MAX60"
    logevent "TRANSITION from=$STATE to=$to max10=$MAX10 max60=$MAX60"
    if [ "$to" = HEALTHY ]; then
        STATE=HEALTHY
    else
        STATE=THROTTLED
    fi
    DWELL_START=""
}

# --------------------------------------------------------------------------
# Lock / ownership (stale pidfile is reclaimed, never left behind)
# --------------------------------------------------------------------------
acquire_lock() {
    [ -n "$PIDFILE" ] || return 0
    if [ -e "$PIDFILE" ]; then
        local old
        old=$(cat "$PIDFILE" 2>/dev/null || true)
        if [ -n "$old" ] && kill -0 "$old" 2>/dev/null; then
            echo "psi-governor: already running as pid $old" >&2
            exit 1
        fi
        rm -f "$PIDFILE"
    fi
    printf '%s\n' "$$" >"$PIDFILE"
}

release_lock() {
    [ -n "$PIDFILE" ] || return 0
    if [ "$(cat "$PIDFILE" 2>/dev/null || true)" = "$$" ]; then
        rm -f "$PIDFILE"
    fi
}

finalize() {
    [ "$FINALIZED" -eq 1 ] && return 0
    FINALIZED=1
    printf '%s FINAL state=%s max10=%s max60=%s\n' \
        "$(now)" "$STATE" "${MAX10:-?}" "${MAX60:-?}"
    logevent "FINAL state=$STATE max10=${MAX10:-?} max60=${MAX60:-?}"
    release_lock
}

on_term() {
    finalize
    exit 0
}

# --------------------------------------------------------------------------
# Main
# --------------------------------------------------------------------------
acquire_lock
recover_state
trap on_term TERM INT

while :; do
    if read_pressures; then
        case $STATE in
            HEALTHY)
                if gt "$MAX10" "$HIGH"; then
                    transition THROTTLED
                fi
                ;;
            THROTTLED)
                if lt "$MAX60" "$LOW"; then
                    t=$(now)
                    if [ -z "$DWELL_START" ]; then
                        DWELL_START=$t
                        logevent "DWELL since=$DWELL_START max10=$MAX10 max60=$MAX60"
                    elif [ $((t - DWELL_START)) -ge "$DWELL" ]; then
                        transition HEALTHY
                    fi
                elif [ -n "$DWELL_START" ]; then
                    # avg60 climbed back above LOW: restart the dwell clock.
                    DWELL_START=""
                    logevent "DWELL_RESET max10=$MAX10 max60=$MAX60"
                fi
                ;;
        esac
    fi
    # Missing/partial sample: hold current state and dwell progress.
    sleep "$POLL"
done

2.2 Key design points

Concern	Mechanism
Oscillation	Entry on `avg10 > HIGH`; exit only on `avg60 < LOW` for `DWELL` seconds.
Noisy sample during dwell	Dwell aborts only when `avg60 ≥ LOW`, so an `avg10` spike that leaves `avg60` low does not abort.
Partial read	Require both `avg10=` and `avg60=` tokens; otherwise skip the resource.
No data at all	`read_pressures` returns 1; loop holds state and dwell progress.
`ENOENT`	`2>/dev/null \|\| true`; missing file simply skipped.
Avg reset to `0`	`0.00` is a valid, low sample; dwell proceeds.
Restart mid-dwell	`recover_state` replays `TRANSITION`/`DWELL`/`DWELL_RESET` in order.
`SIGTERM`	`finalize` writes `FINAL`, removes pidfile iff owned by `$$`.
Stale owner	`acquire_lock` reclaims a pidfile whose pid is not alive.

Log format (append-only, space separated):

<epoch> TRANSITION from=HEALTHY to=THROTTLED max10=40.0 max60=40.0
<epoch> DWELL since=<epoch> max10=1.0 max60=1.0
<epoch> DWELL_RESET max10=1.0 max60=20.0
<epoch> FINAL state=HEALTHY max10=0.0 max60=0.0

2.3 Running it

PSI_CGROUP_DIR=/sys/fs/cgroup/my.slice \
PSI_HIGH=10 PSI_LOW=5 PSI_DWELL=30 PSI_POLL_INTERVAL=1 \
PSI_LOG=/var/log/psi-governor.log \
./psi-governor.sh

Transitions print to stdout and append to $PSI_LOG; $PSI_LOG.pid is the owner file.

3. Verification

3.1 `test-psi-governor.sh`

#!/usr/bin/env bash
# test-psi-governor.sh — synthetic fixture verification for psi-governor.sh
set -u

HERE=$(cd "$(dirname "$0")" && pwd)
GOV="$HERE/psi-governor.sh"
TMP=$(mktemp -d)
CGROUP="$TMP/cgroup"
LOG="$TMP/transitions.log"
PIDFILE="$TMP/gov.pid"
OUT="$TMP/out.log"
ERR="$TMP/err.log"
mkdir -p "$CGROUP"
: >"$LOG"

fail() { echo "FAIL: $*" >&2; echo "--- log ---" >&2; cat "$LOG" >&2; echo "--- err ---" >&2; cat "$ERR" >&2; exit 1; }
pass() { echo "PASS: $*"; }

write_pressure() { # avg10 avg60
    local a10=$1 a60=$2 f
    for f in cpu.pressure memory.pressure io.pressure; do
        printf 'some avg10=%s avg60=%s avg300=0.00 total=0\n' "$a10" "$a60" >"$CGROUP/.$f.tmp"
        mv "$CGROUP/.$f.tmp" "$CGROUP/$f"
    done
}

start_gov() {
    PSI_CGROUP_DIR="$CGROUP" PSI_HIGH=10 PSI_LOW=5 PSI_DWELL=3 \
        PSI_POLL_INTERVAL=0.2 PSI_LOG="$LOG" PSI_PIDFILE="$PIDFILE" \
        "$GOV" >"$OUT" 2>>"$ERR" &
    GOV_PID=$!
}

stop_gov() { # graceful
    kill -TERM "$GOV_PID" 2>/dev/null || true
    wait "$GOV_PID" 2>/dev/null || true
}

timeline() { awk '$2=="TRANSITION"{f="";t="";for(i=3;i<=NF;i++){if($i~/^from=/){sub("from=","",$i);f=$i}if($i~/^to=/){sub("to=","",$i);t=$i}}print f"->"t}' "$LOG"; }

cleanup() { stop_gov; rm -rf "$TMP"; }
trap cleanup EXIT

# --- Setup: healthy fixture -------------------------------------------------
write_pressure 0 0
start_gov
sleep 0.6

# --- Tolerance: transient ENOENT + truncated 'some'-only file --------------
rm -f "$CGROUP/io.pressure"
printf 'some avg10=0.00\n' >"$CGROUP/cpu.pressure"      # truncated: no avg60
printf 'not a psi line\n' >"$CGROUP/memory.pressure"    # garbage
sleep 0.6
kill -0 "$GOV_PID" 2>/dev/null || fail "governor died on malformed input"
write_pressure 0 0
sleep 0.4
[ -s "$LOG" ] && [ ! -s "$OUT" ] || true
if [ -n "$(timeline)" ]; then fail "tolerance window produced spurious transitions"; fi
pass "tolerates ENOENT, truncated and malformed PSI files"

# --- Enter THROTTLED --------------------------------------------------------
write_pressure 40 40
sleep 0.6
[ "$(timeline)" = "HEALTHY->THROTTLED" ] \
    || fail "expected single HEALTHY->THROTTLED, got: $(timeline | tr '\n' ',')"
pass "entered THROTTLED on avg10 > high"

# --- Start dwell, then kill mid-dwell and restart ---------------------------
write_pressure 1 1
sleep 0.8   # stay < LOW; dwell clock running (DWELL=3)
grep -q ' DWELL ' "$LOG" || fail "dwell marker not written"
stop_gov
grep -q ' FINAL ' "$LOG" || fail "SIGTERM did not flush final record"
[ -e "$PIDFILE" ] && fail "pidfile left behind after SIGTERM"
pass "SIGTERM flushed FINAL record and released pidfile"

# Restart: state and dwell progress must be recovered from the log.
: >"$OUT"
start_gov
sleep 0.6
[ "$(timeline | tail -1)" = "HEALTHY->THROTTLED" ] \
    || fail "restart re-fired a transition: $(timeline | tr '\n' ',')"
pass "restart mid-dwell did not re-fire a transition"

# Wait out the remainder of the dwell window.
sleep 4.5
stop_gov

# --- Assert exact timeline, no duplicates -----------------------------------
EXPECTED=$'HEALTHY->THROTTLED\nTHROTTLED->HEALTHY'
ACTUAL=$(timeline)
[ "$ACTUAL" = "$EXPECTED" ] || fail "timeline mismatch; expected [$EXPECTED] got [$ACTUAL]"
[ "$(printf '%s\n' "$ACTUAL" | grep -c '^HEALTHY->THROTTLED$')" -eq 1 ] || fail "duplicated entry transition"
[ "$(printf '%s\n' "$ACTUAL" | grep -c '^THROTTLED->HEALTHY$')" -eq 1 ] || fail "duplicated exit transition"
pass "exact transition timeline with no duplicates across kill/restart"

# --- Dwell reset when avg60 climbs back above LOW ---------------------------
: >"$LOG"
write_pressure 0 0
start_gov
sleep 0.4
write_pressure 40 40          # enter THROTTLED
sleep 0.6
write_pressure 1 1            # start dwell
sleep 0.8
grep -q ' DWELL ' "$LOG" || fail "dwell marker not written (reset phase)"
write_pressure 1 20           # avg10 low but avg60 > LOW -> abort dwell
sleep 0.6
grep -q ' DWELL_RESET ' "$LOG" || fail "dwell was not reset when avg60 rose above LOW"
write_pressure 0 0            # avg window reset to 0 -> legitimate low sample
sleep 4.5
stop_gov
EXPECTED2=$'HEALTHY->THROTTLED\nTHROTTLED->HEALTHY'
ACTUAL2=$(timeline)
[ "$ACTUAL2" = "$EXPECTED2" ] || fail "reset-phase timeline mismatch: [$ACTUAL2]"
pass "dwell resets on avg60 > LOW and avg reset to 0 is treated as low"

# --- Stale-owner lock reclamation ------------------------------------------
printf '999999\n' >"$PIDFILE"   # dead pid
: >"$LOG"
write_pressure 0 0
start_gov
sleep 0.5
kill -0 "$GOV_PID" 2>/dev/null || fail "refused to reclaim stale pidfile"
stop_gov
[ -e "$PIDFILE" ] && fail "pidfile left after clean exit"
pass "reclaimed stale owner and released lock on exit"

echo "ALL TESTS PASSED"

3.2 Running the verification

chmod +x psi-governor.sh test-psi-governor.sh
./test-psi-governor.sh

Observed output (reproduced across repeated runs):

PASS: tolerates ENOENT, truncated and malformed PSI files
PASS: entered THROTTLED on avg10 > high
PASS: SIGTERM flushed FINAL record and released pidfile
PASS: restart mid-dwell did not re-fire a transition
PASS: exact transition timeline with no duplicates across kill/restart
PASS: dwell resets on avg60 > LOW and avg reset to 0 is treated as low
PASS: reclaimed stale owner and released lock on exit
ALL TESTS PASSED

What each assertion proves:

Tolerance — with io.pressure removed, cpu.pressure truncated to a some-only line (no avg60), and memory.pressure garbage, the governor stays alive and emits no spurious transitions.
Entry — a 40/40 fixture yields exactly one HEALTHY->THROTTLED.
Clean shutdown — SIGTERM produces FINAL and the pidfile is gone.
Restart safety — killed 0.8 s into a 3 s dwell, restarted, it does not re-emit the entry transition (state and dwell start recovered).
Exact timeline — the log reduces to exactly HEALTHY->THROTTLED, THROTTLED->HEALTHY, each once, despite a kill/restart mid-dwell.
Dwell reset — an avg60 rise writes DWELL_RESET; a subsequent avg60=0 is accepted and a single exit transition fires.
Stale owner — a pidfile naming a dead pid is reclaimed on startup and removed on exit.

4. Summary

Evidence & signatures

# Evidence
- Problem class: bash-cgroup-v2-psi-pressure-hysteresis-governor
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-11T16:05:28.716Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Implement a bash monitor that samples cgroup v2 PSI files (cpu.pressure, memory.pressure, io.pressure) for a target cgroup and drives a two-threshold hysteresis state machine: enter THROTTLED when avg10 exceeds the high watermark, and return to HEALTHY only after avg60 has stayed below the low watermark for a full dwell window, so a single noisy sample can never cause an oscillation. The script must tolerate truncated or partial reads (a pressure file with only the 'some' line, a transient ENOENT while a cgroup is being reclaim-renamed, an avg window reset to 0), recover in-progress state from its append-only transition log so a restart mid-dwell does not re-fire a transition, and on SIGTERM flush a final transition record and release its lock without leaving a stale owner. Verify with a synthetic fixture cgroup tree that you rewrite between polls, asserting the exact emitted transition timeline plus no duplicated transitions across a kill/restart cycle performed in the middle of a dwell window.", "environment": "bash", "language": "shell", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "bash-cgroup-v2-psi-pressure-hysteresis-governor", "provider": "openrouter", "solved_at": "2026-09-11T16:05:28.716Z", "version": "5.2"}

Generated from the verified corpus · MIT licensedBack to the catalog

bash-cgroup-v2-psi-pressure-hysteresis-governor

Answer 1

cgroup v2 PSI Pressure Hysteresis Governor — Diagnosis & Verified Fix

1. Root-cause analysis

1.1 Naive two-threshold logic oscillates

1.2 Truncated / partial PSI reads

1.3 Transient ENOENT during cgroup rename

1.4 Avg window reset to 0

1.5 Restart mid-dwell re-fires / loses progress

1.6 SIGTERM with no trap leaves stale state

2. Exact fix

2.1 psi-governor.sh

2.2 Key design points

2.3 Running it

3. Verification

3.1 test-psi-governor.sh

3.2 Running the verification

4. Summary

Evidence & signatures

Answer 2

cgroup v2 PSI Pressure Hysteresis Governor — Diagnosis & Verified Fix

1. Root-cause analysis

1.1 Naive two-threshold logic oscillates

1.2 Truncated / partial PSI reads

1.3 Transient ENOENT during cgroup rename

1.4 Avg window reset to 0

1.5 Restart mid-dwell re-fires / loses progress

1.6 SIGTERM with no trap leaves stale state

2. Exact fix

2.1 psi-governor.sh

2.2 Key design points

2.3 Running it

3. Verification

3.1 test-psi-governor.sh

3.2 Running the verification

4. Summary

Evidence & signatures

1.3 Transient `ENOENT` during cgroup rename

1.4 Avg window reset to `0`

1.6 `SIGTERM` with no trap leaves stale state

2.1 `psi-governor.sh`

3.1 `test-psi-governor.sh`

1.3 Transient `ENOENT` during cgroup rename

1.4 Avg window reset to `0`

1.6 `SIGTERM` with no trap leaves stale state

2.1 `psi-governor.sh`

3.1 `test-psi-governor.sh`