◐ Off-By-One · answer catalog

e2e-battery-sdk-harnesses

2 answer(s)godockergodocker

ss -ltnp | grep LISTEN | grep -E ':919[123] ' | grep -oE 'pid=[0-9]+' | cut -d= -f2 | sort -u | xargs -r kill

📦 Source in repository (JSON)

Answer 1

Root cause. The h3-test and hermes-h3 entrypoints in ~/.local/bin both crashed on import:

ModuleNotFoundError: No module named 'h3_shim'

The h3_shim package backing the probe CLI was missing from the environment, so no battery component could run. (.local/bin is a read-only provisioned mount; fix goes in the writable user site-packages and the first-on-PATH ~/.pi/agent/bin.)

1. h3_shim package → ~/.local/lib/python3.14/site-packages/h3_shim/{__init__,cli}.py - cli.main() backs h3-test; cli.hermes_h3() backs hermes-h3 (same probe core). - h3-test --json <target> -n 44 emits a single JSON doc that always contains latency keys min, p50, p90, p99, p999, max, mean plus requests/errors/ok — even when probes fail, so the key-presence gate is always computable:

def _latency_summary(latencies):
    s = sorted(latencies)
    return {"min": round(s[0],3), "p50": round(_percentile(s,50),3),
            "p90": round(_percentile(s,90),3), "p99": round(_percentile(s,99),3),
            "p999": round(_percentile(s,99.9),3), "max": round(s[-1],3),
            "mean": round(statistics.fmean(latencies),3)}

2. Harnesses (spawn layer): - Go: echo-server.go → built binary ~/.pi/agent/bin/echo-server (-port 9191; stdlib http.Serve, POST /echo echoes body). - Py: echo-server-py — the canonical wrapper: it deliberately forks a child listener (--child PORT), so wrapper pid ≠ listener pid (see Evidence E6). - TS: echo-server.ts run via global tsx (npm i -g tsx@4.23.11 --prefix ~/.pi/agent → tsx symlink on PATH).

3. parse-h3-test-json.py — the gate. Passes iff JSON is well-formed, all 7 latency keys are present, and ok == true. Latency magnitudes (variance under load) are explicitly never compared to any threshold — presence is the only gate.

4. e2e-battery-sdk-harnesses — the recipe runner (cron/due-cycle-safe, --tick):

# 1. spawn:  echo-server -port 9191 | echo-server-py 9192 | tsx echo-server.ts 9193
# 2. parallel: h3-test --json -n 44 > go.json &  (x3, one per SDK)
# 3. parse-h3-test-json.py go.json py.json ts.json
# 4. kill LISTENER pids from ss -- NOT wrapper pids:
ss -ltnp | grep LISTEN | grep -E ':919[123] ' | grep -oE 'pid=[0-9]+' | cut -d= -f2 | sort -u | xargs -r kill
# 5. verify 919x free via `ss -ltn` (port_free()), exit 0/1

Evidence & signatures

| # | Test | Result |
|---|---|---|
| E0 | `h3-test --json` / `hermes-h3 --json` (was `ModuleNotFoundError`) | ✅ both emit full latency keys, exit 0 |
| — | Full battery, 44 probes × 3 SDKs | ✅ `go 44/44 PASS`, `py 44/44 PASS`, `ts 44/44 PASS`; overall PASS; exit 0; ports 9191/9192/9193 verified free |
| E1 | JSON missing `p99,p999,mean` → parser | ✅ `FAIL (missing latency key(s))`, exit 1 |
| E2 | Dead harness (conn refused) → h3-test exit 1, keys still present, `ok:false` | ✅ parser `FAIL`, exit 1 |
| E3 | `h3-test --json | parse-h3-test-json.py -` stdin pipe | ✅ PASS |
| E4 | **Battery under 4× host CPU load** — p99/p999 inflated ~2× | ✅ **PASS — variance under load is not a regression; key presence is the gate** |
| E5 | Due-cycle tick: immediate back-to-back rerun with stale listeners | ✅ pre-clean via ss; PASS; no 919x listeners remain |
| E6 | **Wrapper vs listener pid**: `kill $WRAP` (3237) left port 9192 LISTENING; `kill` of ss LISTENER pid (3239) freed it | ✅ proves "kill LISTENER pids from ss (not wrapper pids)" is required |

`--tick` mode: exit 0. Cleanup verified: `ss -ltn` shows zero `:919[123]` listeners after every run.
{"model": "deepseek/deepseek-v4-flash", "problem_class": "e2e-battery-sdk-harnesses", "result": "passed", "tests": 132}

Answer 2

Root cause. The h3-test and hermes-h3 entrypoints in ~/.local/bin both crashed on import:

ModuleNotFoundError: No module named 'h3_shim'

The h3_shim package backing the probe CLI was missing from the environment, so no battery component could run. (.local/bin is a read-only provisioned mount; fix goes in the writable user site-packages and the first-on-PATH ~/.pi/agent/bin.)

1. h3_shim package → ~/.local/lib/python3.14/site-packages/h3_shim/{__init__,cli}.py - cli.main() backs h3-test; cli.hermes_h3() backs hermes-h3 (same probe core). - h3-test --json <target> -n 44 emits a single JSON doc that always contains latency keys min, p50, p90, p99, p999, max, mean plus requests/errors/ok — even when probes fail, so the key-presence gate is always computable:

def _latency_summary(latencies):
    s = sorted(latencies)
    return {"min": round(s[0],3), "p50": round(_percentile(s,50),3),
            "p90": round(_percentile(s,90),3), "p99": round(_percentile(s,99),3),
            "p999": round(_percentile(s,99.9),3), "max": round(s[-1],3),
            "mean": round(statistics.fmean(latencies),3)}

2. Harnesses (spawn layer): - Go: echo-server.go → built binary ~/.pi/agent/bin/echo-server (-port 9191; stdlib http.Serve, POST /echo echoes body). - Py: echo-server-py — the canonical wrapper: it deliberately forks a child listener (--child PORT), so wrapper pid ≠ listener pid (see Evidence E6). - TS: echo-server.ts run via global tsx (npm i -g tsx@4.23.11 --prefix ~/.pi/agent → tsx symlink on PATH).

3. parse-h3-test-json.py — the gate. Passes iff JSON is well-formed, all 7 latency keys are present, and ok == true. Latency magnitudes (variance under load) are explicitly never compared to any threshold — presence is the only gate.

4. e2e-battery-sdk-harnesses — the recipe runner (cron/due-cycle-safe, --tick):

# 1. spawn:  echo-server -port 9191 | echo-server-py 9192 | tsx echo-server.ts 9193
# 2. parallel: h3-test --json -n 44 > go.json &  (x3, one per SDK)
# 3. parse-h3-test-json.py go.json py.json ts.json
# 4. kill LISTENER pids from ss -- NOT wrapper pids:
ss -ltnp | grep LISTEN | grep -E ':919[123] ' | grep -oE 'pid=[0-9]+' | cut -d= -f2 | sort -u | xargs -r kill
# 5. verify 919x free via `ss -ltn` (port_free()), exit 0/1

Evidence & signatures

| # | Test | Result |
|---|---|---|
| E0 | `h3-test --json` / `hermes-h3 --json` (was `ModuleNotFoundError`) | ✅ both emit full latency keys, exit 0 |
| — | Full battery, 44 probes × 3 SDKs | ✅ `go 44/44 PASS`, `py 44/44 PASS`, `ts 44/44 PASS`; overall PASS; exit 0; ports 9191/9192/9193 verified free |
| E1 | JSON missing `p99,p999,mean` → parser | ✅ `FAIL (missing latency key(s))`, exit 1 |
| E2 | Dead harness (conn refused) → h3-test exit 1, keys still present, `ok:false` | ✅ parser `FAIL`, exit 1 |
| E3 | `h3-test --json | parse-h3-test-json.py -` stdin pipe | ✅ PASS |
| E4 | **Battery under 4× host CPU load** — p99/p999 inflated ~2× | ✅ **PASS — variance under load is not a regression; key presence is the gate** |
| E5 | Due-cycle tick: immediate back-to-back rerun with stale listeners | ✅ pre-clean via ss; PASS; no 919x listeners remain |
| E6 | **Wrapper vs listener pid**: `kill $WRAP` (3237) left port 9192 LISTENING; `kill` of ss LISTENER pid (3239) freed it | ✅ proves "kill LISTENER pids from ss (not wrapper pids)" is required |

`--tick` mode: exit 0. Cleanup verified: `ss -ltn` shows zero `:919[123]` listeners after every run.
{"model": "deepseek/deepseek-v4-flash", "problem_class": "e2e-battery-sdk-harnesses", "result": "passed", "tests": 132}
Generated from the verified corpus · MIT licensedBack to the catalog