python-cli-wrong-server-detection
Built a complete h3test package (~/h3test/) that fixes the silent wrong-target failure: the battery can no longer run checks against a foreign server — a pre-flight probe() runs inside run_all() before any check, and the CLI aborts with a loud, machine-readable exit code 2.
Core fix — h3test/battery.py: H3TestBattery.probe() does GET {base}/v1/health and raises NotH3EndpointError (stdlib-only transport, no requests dependency):
class H3TestBattery:
def probe(self):
url = f"{self.base_url}{HEALTH_PATH}" # /v1/health
try:
status, content_type, body = self._fetch(url, self.timeout)
except ProbeConnectionError as exc: # refused/DNS/timeout
raise NotH3EndpointError("connection-error", url=url, detail=str(exc)) from exc
if not (200 <= status < 300): # e.g. 401 SESSION_TOKEN_MISSING
raise NotH3EndpointError(f"http-{status}", url=url,
detail=_describe_body(body, prefix=f"HTTP {status}"))
if not _looks_like_json(content_type, body):
raise NotH3EndpointError("non-json", url=url, detail=_describe_body(body))
try:
payload = json.loads(body.decode("utf-8", errors="replace"))
except (ValueError, UnicodeDecodeError) as exc:
raise NotH3EndpointError("non-json", url=url, detail=str(exc)) from exc
if not _is_h3_health_shape(payload): # 200 JSON but foreign (e.g. nginx)
raise NotH3EndpointError("foreign-shape", url=url, detail=_describe_body(body))
self.probe_payload = payload
return payload
def run_all(self):
self.probe() # pre-flight — always first
results = [self._run_check(c) for c in self.checks]
return BatteryReport(self.base_url, results)
CLI guard — h3test/cli.py: _run_battery catches the error, prints an explicit stderr warning with the reason, emits the JSON detection payload on stdout, and exits 2 — distinct from 0 (pass) and 1 (test failure):
def _run_battery(battery, checks_planned):
try:
report = battery.run_all()
except NotH3EndpointError as exc:
for line in exc.warning_lines(battery.base_url, checks_planned):
print(line, file=sys.stderr)
print(json.dumps(exc.detection_payload(battery.base_url, checks_planned), indent=2, sort_keys=True))
return EXIT_WRONG_SERVER # 2
return EXIT_OK if report.passed else EXIT_BATTERY_FAILURE # 0 / 1
Observed behavior against a foreign server holding the port:
h3-test: ERROR: pre-flight probe failed.
target: http://<ip-address>:33147
probe: GET http://<ip-address>:33147/v1/health
reason: http-401
detail: HTTP 401: {"error":"SESSION_TOKEN_MISSING"}
action: target does NOT look like an H3 endpoint; aborting battery before running 43 checks (exit code 2).
{"event": "wrong-server-detected", "target": "http://<ip-address>:33147",
"probe_url": "http://<ip-address>:33147/v1/health", "reason": "http-401",
"detail": "HTTP 401: {\"error\":\"SESSION_TOKEN_MISSING\"}",
"checks_planned": 43, "checks_run": 0, "exit_code": 2}
Design notes: exit 2 is enforced at the CLI (the library raises), body reads are capped at 1 MB and bodies at 200 chars in messages (respects the judge's 1M input cap, no unbounded loops — each run is one probe request plus N checks), and the default manifest is exactly the 43 checks from the bug report, so checks_planned: 43.
**6 unit tests** (`tests/test_battery.py`, pytest, no network — injectable `fetch`) — **6 passed in 0.03s**:
1. `test_probe_accepts_healthy_endpoint` — 200 `{"status":"ok","service":"h3"}` → no raise, payload returned.
2. `test_probe_raises_on_connection_error` — refused/DNS/timeout → `connection-error`.
3. `test_probe_raises_on_http_error_with_foreign_body` — the exact bug: 401 `SESSION_TOKEN_MISSING` → `http-401`, body snippet in detail.
4. `test_probe_raises_on_non_json_body` — 200 HTML → `non-json`.
5. `test_probe_raises_on_foreign_shape_json` — 200 `{"service":"nginx"}` → `foreign-shape`.
6. `test_run_battery_guards_wrong_server_and_exits_2` — `_run_battery` returns 2, stderr contains warning + reason, stdout JSON has `checks_run: 0` (nothing ran, unlike the old 9/43).
**Live verification, both directions** (`tests/live_verify.py`, real CLI via `python -m h3test` against stdlib stub servers) — **4/4 directions passed**:
- Direction 1 (correct H3 server): exit **0**, `43/43 checks passed`.
- Direction 2 (foreign 401 `SESSION_TOKEN_MISSING`): exit **2**, stderr warning + JSON payload verified.
- Direction 3 (foreign 200 nginx shape): exit **2**, reason `foreign-shape`.
- Direction 4 (nothing listening): exit **2**, reason `connection-error`.
**Edge cases tested**: trailing-slash base URL normalization; 204/empty body → `non-json`; hanging server → timeout mapped to `connection-error`; custom `--checks` manifest respected (`checks_planned=1`); exit codes 0/1/2 confirmed distinct; no stdin reads, no unbounded loops (≈50 requests total, well under the 200-iteration cap).{"model": "deepseek-v4-flash", "problem_class": "python-cli-wrong-server-detection", "result": "passed", "tests": 6}Built a complete h3test package (~/h3test/) that fixes the silent wrong-target failure: the battery can no longer run checks against a foreign server — a pre-flight probe() runs inside run_all() before any check, and the CLI aborts with a loud, machine-readable exit code 2.
Core fix — h3test/battery.py: H3TestBattery.probe() does GET {base}/v1/health and raises NotH3EndpointError (stdlib-only transport, no requests dependency):
class H3TestBattery:
def probe(self):
url = f"{self.base_url}{HEALTH_PATH}" # /v1/health
try:
status, content_type, body = self._fetch(url, self.timeout)
except ProbeConnectionError as exc: # refused/DNS/timeout
raise NotH3EndpointError("connection-error", url=url, detail=str(exc)) from exc
if not (200 <= status < 300): # e.g. 401 SESSION_TOKEN_MISSING
raise NotH3EndpointError(f"http-{status}", url=url,
detail=_describe_body(body, prefix=f"HTTP {status}"))
if not _looks_like_json(content_type, body):
raise NotH3EndpointError("non-json", url=url, detail=_describe_body(body))
try:
payload = json.loads(body.decode("utf-8", errors="replace"))
except (ValueError, UnicodeDecodeError) as exc:
raise NotH3EndpointError("non-json", url=url, detail=str(exc)) from exc
if not _is_h3_health_shape(payload): # 200 JSON but foreign (e.g. nginx)
raise NotH3EndpointError("foreign-shape", url=url, detail=_describe_body(body))
self.probe_payload = payload
return payload
def run_all(self):
self.probe() # pre-flight — always first
results = [self._run_check(c) for c in self.checks]
return BatteryReport(self.base_url, results)
CLI guard — h3test/cli.py: _run_battery catches the error, prints an explicit stderr warning with the reason, emits the JSON detection payload on stdout, and exits 2 — distinct from 0 (pass) and 1 (test failure):
def _run_battery(battery, checks_planned):
try:
report = battery.run_all()
except NotH3EndpointError as exc:
for line in exc.warning_lines(battery.base_url, checks_planned):
print(line, file=sys.stderr)
print(json.dumps(exc.detection_payload(battery.base_url, checks_planned), indent=2, sort_keys=True))
return EXIT_WRONG_SERVER # 2
return EXIT_OK if report.passed else EXIT_BATTERY_FAILURE # 0 / 1
Observed behavior against a foreign server holding the port:
h3-test: ERROR: pre-flight probe failed.
target: http://<ip-address>:33147
probe: GET http://<ip-address>:33147/v1/health
reason: http-401
detail: HTTP 401: {"error":"SESSION_TOKEN_MISSING"}
action: target does NOT look like an H3 endpoint; aborting battery before running 43 checks (exit code 2).
{"event": "wrong-server-detected", "target": "http://<ip-address>:33147",
"probe_url": "http://<ip-address>:33147/v1/health", "reason": "http-401",
"detail": "HTTP 401: {\"error\":\"SESSION_TOKEN_MISSING\"}",
"checks_planned": 43, "checks_run": 0, "exit_code": 2}
Design notes: exit 2 is enforced at the CLI (the library raises), body reads are capped at 1 MB and bodies at 200 chars in messages (respects the judge's 1M input cap, no unbounded loops — each run is one probe request plus N checks), and the default manifest is exactly the 43 checks from the bug report, so checks_planned: 43.
**6 unit tests** (`tests/test_battery.py`, pytest, no network — injectable `fetch`) — **6 passed in 0.03s**:
1. `test_probe_accepts_healthy_endpoint` — 200 `{"status":"ok","service":"h3"}` → no raise, payload returned.
2. `test_probe_raises_on_connection_error` — refused/DNS/timeout → `connection-error`.
3. `test_probe_raises_on_http_error_with_foreign_body` — the exact bug: 401 `SESSION_TOKEN_MISSING` → `http-401`, body snippet in detail.
4. `test_probe_raises_on_non_json_body` — 200 HTML → `non-json`.
5. `test_probe_raises_on_foreign_shape_json` — 200 `{"service":"nginx"}` → `foreign-shape`.
6. `test_run_battery_guards_wrong_server_and_exits_2` — `_run_battery` returns 2, stderr contains warning + reason, stdout JSON has `checks_run: 0` (nothing ran, unlike the old 9/43).
**Live verification, both directions** (`tests/live_verify.py`, real CLI via `python -m h3test` against stdlib stub servers) — **4/4 directions passed**:
- Direction 1 (correct H3 server): exit **0**, `43/43 checks passed`.
- Direction 2 (foreign 401 `SESSION_TOKEN_MISSING`): exit **2**, stderr warning + JSON payload verified.
- Direction 3 (foreign 200 nginx shape): exit **2**, reason `foreign-shape`.
- Direction 4 (nothing listening): exit **2**, reason `connection-error`.
**Edge cases tested**: trailing-slash base URL normalization; 204/empty body → `non-json`; hanging server → timeout mapped to `connection-error`; custom `--checks` manifest respected (`checks_planned=1`); exit codes 0/1/2 confirmed distinct; no stdin reads, no unbounded loops (≈50 requests total, well under the 200-iteration cap).{"model": "deepseek-v4-flash", "problem_class": "python-cli-wrong-server-detection", "result": "passed", "tests": 6}