A long-lived health probe printed count=3 for a list endpoint that actually returned 5 rows. The number was treated as a real data point for dozens of runs and produced a false narrative ("the API serves 3 records, the seed file holds 5, so the served set differs from the seed"). The probe looked like this:
Diagnosed and verified. Artifacts in ~/work/: count_rows.py (fixed derivation), probe_fixed.py, probe_buggy.py (reproduction), test_probe.py (live-endpoint test), and SOLUTION.md.
Live verification result:
before (buggy): 3
after (fixed): 5
raw len(body['items']): 5
5 tests OK
A long-lived health probe printed count=3 for a list endpoint that actually returned 5 rows. The number was treated as a real data point for dozens of runs and produced a false narrative ("the API serves 3 records, the seed file holds 5, so the served set differs from the seed"). The probe looked like this:
n = len(json.loads(body))
The endpoint answers a JSON object envelope:
{"items": [<5 rows>], "count": 5, "query": ""}
len() of that object is 3 — the number of keys (items, count, query) — not the number of rows. The except branch that read the count field only ran if the expression above raised, and len(json.loads(body)) never raises for a well-formed dict, so the fallback was unreachable.
len() is polymorphic. On a wrapped payload it silently means "number of envelope keys", not "number of rows". Two compounding defects:
len(dict), which is well-defined and never errors, so nothing signalled the mistake.parsed["count"] logic inside except made it dead code: a successful parse of the wrong shape bypassed it entirely. A shape bug that never raises is worse than one that does, because there is no exception to notice.The general rule: a probe that parses a wrapped payload must count the rows, never the envelope. len() of a parsed dict is a shape bug that never raises and therefore never falls back.
Derive every count from a named row collection, branching on the parsed type. Prefer an integer count field, otherwise len(items), and only use len(parsed) for a genuine bare array. Fail loudly on unknown shapes instead of returning a plausible-looking number.
# count_rows.py
import json
class ShapeError(ValueError):
"""The parsed response is not a recognized list envelope."""
def _is_int(value) -> bool:
# bool is an int subclass; a count of True is not a real count.
return isinstance(value, int) and not isinstance(value, bool)
def row_count(parsed) -> int:
"""Return the number of rows in a parsed JSON payload."""
if isinstance(parsed, dict):
count = parsed.get("count")
if _is_int(count):
return count
items = parsed.get("items")
if isinstance(items, list):
return len(items)
raise ShapeError(
f"dict envelope has no integer 'count' and no list 'items': "
f"keys={sorted(parsed)}"
)
if isinstance(parsed, list):
return len(parsed)
raise ShapeError(f"unsupported payload type: {type(parsed).__name__}")
def served_count_from_body(body: str) -> int:
"""Derive the row count directly from a raw response body string."""
return row_count(json.loads(body))
Wire the probe to it, with one assertion against the raw shape before deriving:
# scripts/<probe>.py
import json
import urllib.request
from count_rows import row_count
def served_count(url: str) -> int:
with urllib.request.urlopen(url, timeout=5) as resp:
body = resp.read().decode("utf-8")
parsed = json.loads(body)
# One assertion against the raw shape, then derive from named rows.
if not isinstance(parsed, (dict, list)):
raise ValueError(f"unexpected top-level JSON type: {type(parsed).__name__}")
return row_count(parsed)
Diff of the count derivation:
- try:
- n = len(json.loads(body))
- except Exception:
- parsed = json.loads(body)
- n = int(parsed.get("count", 0))
- return n
+ parsed = json.loads(body)
+ if not isinstance(parsed, (dict, list)):
+ raise ValueError(f"unexpected top-level JSON type: {type(parsed).__name__}")
+ return row_count(parsed)
Diagnostic first step when a legacy number looks surprising: dump the raw response shape before building anything on it.
python3 -c 'import json,sys; d=json.load(open("resp.json")); print(type(d).__name__, len(d)); print(d.keys() if isinstance(d,dict) else "")'
test_probe.py starts a live http.server that serves the envelope, runs the identical buggy and fixed probes against the same URL, and checks all shape branches. Run:
cd ~/work
python3 test_probe.py
Observed output:
test_body_helper ... ok
test_buggy_probe_counts_envelope_keys ... ok
test_fixed_probe_counts_rows ... ok
test_row_count_rejects_unknown_shape ... ok
test_row_count_variants ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.518s
OK
Before/after on the same live endpoint:
before (buggy): 3
after (fixed): 5
raw len(body['items']): 5
count=3 -> count=5, matching the raw len(body["items"]) dump — the same verification pattern described in the report. Additional covered cases:
| Input | Result |
|---|---|
{"items": [...5...], "count": 5} |
5 |
{"items": [...5...], "count": 99} |
99 (integer count wins) |
{"items": [...5...]} (no count) |
5 |
[...5...] (bare array) |
5 |
{"items": [...5...], "count": true} |
5 (bool is not a count) |
{"items": [], "count": 0} |
0 |
{"query": "", "meta": {}} |
raises ShapeError |
len() the envelope object.# Evidence - Problem class: json-probe-counts-envelope-keys-instead-of-payload-rows - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T17:49:47.281Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a long-lived health probe printed 'count=3' for a list endpoint while the endpoint actually returned 5 rows. Worse, the flag had been read as a real data point for dozens of runs and had generated a whole narrative ('the API serves 3 records, the seed file holds 5, so the served set differs from the seed') that was entirely an artifact. ROOT CAUSE: the probe did `n = len(json.loads(body))`. The endpoint answers a JSON OBJECT envelope, so len() counted the envelope's three KEYS (items/count/query) - not the rows - and the exception branch that would have read the `count` field only ran if the first expression raised, which it never did. Classic len(dict) on a wrapped payload, plus a fallback that a successful-but-wrong parse makes unreachable. FIX: branch on the parsed type - if it is a dict, prefer an integer `count` field and otherwise len(items); only fall back to len(parsed) for a bare array - and assert against the raw shape once. VERIFICATION: identical probe before/after on the same live endpoint (count=3 -> count=5, matching the raw len(body['items']) dump). LESSON: a probe that parses a wrapped payload must count the rows, never the envelope; a len() on a parsed dict is a shape bug that never raises and therefore never falls back. Derive every count from a named row collection, and when a legacy number looks surprising, dump the raw response shape before building anything on it.", "environment": "python3 urllib probe script against a JSON HTTP API; response envelope {items:[...], count:N, query:''}", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "json-probe-counts-envelope-keys-instead-of-payload-rows", "provider": "openrouter", "solved_at": "2026-09-16T17:49:47.281Z", "version": ""}Diagnosed and verified. Artifacts in ~/work/: count_rows.py (fixed derivation), probe_fixed.py, probe_buggy.py (reproduction), test_probe.py (live-endpoint test), and SOLUTION.md.
Live verification result:
before (buggy): 3
after (fixed): 5
raw len(body['items']): 5
5 tests OK
A long-lived health probe printed count=3 for a list endpoint that actually returned 5 rows. The number was treated as a real data point for dozens of runs and produced a false narrative ("the API serves 3 records, the seed file holds 5, so the served set differs from the seed"). The probe looked like this:
n = len(json.loads(body))
The endpoint answers a JSON object envelope:
{"items": [<5 rows>], "count": 5, "query": ""}
len() of that object is 3 — the number of keys (items, count, query) — not the number of rows. The except branch that read the count field only ran if the expression above raised, and len(json.loads(body)) never raises for a well-formed dict, so the fallback was unreachable.
len() is polymorphic. On a wrapped payload it silently means "number of envelope keys", not "number of rows". Two compounding defects:
len(dict), which is well-defined and never errors, so nothing signalled the mistake.parsed["count"] logic inside except made it dead code: a successful parse of the wrong shape bypassed it entirely. A shape bug that never raises is worse than one that does, because there is no exception to notice.The general rule: a probe that parses a wrapped payload must count the rows, never the envelope. len() of a parsed dict is a shape bug that never raises and therefore never falls back.
Derive every count from a named row collection, branching on the parsed type. Prefer an integer count field, otherwise len(items), and only use len(parsed) for a genuine bare array. Fail loudly on unknown shapes instead of returning a plausible-looking number.
# count_rows.py
import json
class ShapeError(ValueError):
"""The parsed response is not a recognized list envelope."""
def _is_int(value) -> bool:
# bool is an int subclass; a count of True is not a real count.
return isinstance(value, int) and not isinstance(value, bool)
def row_count(parsed) -> int:
"""Return the number of rows in a parsed JSON payload."""
if isinstance(parsed, dict):
count = parsed.get("count")
if _is_int(count):
return count
items = parsed.get("items")
if isinstance(items, list):
return len(items)
raise ShapeError(
f"dict envelope has no integer 'count' and no list 'items': "
f"keys={sorted(parsed)}"
)
if isinstance(parsed, list):
return len(parsed)
raise ShapeError(f"unsupported payload type: {type(parsed).__name__}")
def served_count_from_body(body: str) -> int:
"""Derive the row count directly from a raw response body string."""
return row_count(json.loads(body))
Wire the probe to it, with one assertion against the raw shape before deriving:
# scripts/<probe>.py
import json
import urllib.request
from count_rows import row_count
def served_count(url: str) -> int:
with urllib.request.urlopen(url, timeout=5) as resp:
body = resp.read().decode("utf-8")
parsed = json.loads(body)
# One assertion against the raw shape, then derive from named rows.
if not isinstance(parsed, (dict, list)):
raise ValueError(f"unexpected top-level JSON type: {type(parsed).__name__}")
return row_count(parsed)
Diff of the count derivation:
- try:
- n = len(json.loads(body))
- except Exception:
- parsed = json.loads(body)
- n = int(parsed.get("count", 0))
- return n
+ parsed = json.loads(body)
+ if not isinstance(parsed, (dict, list)):
+ raise ValueError(f"unexpected top-level JSON type: {type(parsed).__name__}")
+ return row_count(parsed)
Diagnostic first step when a legacy number looks surprising: dump the raw response shape before building anything on it.
python3 -c 'import json,sys; d=json.load(open("resp.json")); print(type(d).__name__, len(d)); print(d.keys() if isinstance(d,dict) else "")'
test_probe.py starts a live http.server that serves the envelope, runs the identical buggy and fixed probes against the same URL, and checks all shape branches. Run:
cd ~/work
python3 test_probe.py
Observed output:
test_body_helper ... ok
test_buggy_probe_counts_envelope_keys ... ok
test_fixed_probe_counts_rows ... ok
test_row_count_rejects_unknown_shape ... ok
test_row_count_variants ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.518s
OK
Before/after on the same live endpoint:
before (buggy): 3
after (fixed): 5
raw len(body['items']): 5
count=3 -> count=5, matching the raw len(body["items"]) dump — the same verification pattern described in the report. Additional covered cases:
| Input | Result |
|---|---|
{"items": [...5...], "count": 5} |
5 |
{"items": [...5...], "count": 99} |
99 (integer count wins) |
{"items": [...5...]} (no count) |
5 |
[...5...] (bare array) |
5 |
{"items": [...5...], "count": true} |
5 (bool is not a count) |
{"items": [], "count": 0} |
0 |
{"query": "", "meta": {}} |
raises ShapeError |
len() the envelope object.# Evidence - Problem class: json-probe-counts-envelope-keys-instead-of-payload-rows - Model: openrouter/deepseek/deepseek-v4.1-flash - Solved: 2026-09-16T17:49:47.281Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "SYMPTOM: a long-lived health probe printed 'count=3' for a list endpoint while the endpoint actually returned 5 rows. Worse, the flag had been read as a real data point for dozens of runs and had generated a whole narrative ('the API serves 3 records, the seed file holds 5, so the served set differs from the seed') that was entirely an artifact. ROOT CAUSE: the probe did `n = len(json.loads(body))`. The endpoint answers a JSON OBJECT envelope, so len() counted the envelope's three KEYS (items/count/query) - not the rows - and the exception branch that would have read the `count` field only ran if the first expression raised, which it never did. Classic len(dict) on a wrapped payload, plus a fallback that a successful-but-wrong parse makes unreachable. FIX: branch on the parsed type - if it is a dict, prefer an integer `count` field and otherwise len(items); only fall back to len(parsed) for a bare array - and assert against the raw shape once. VERIFICATION: identical probe before/after on the same live endpoint (count=3 -> count=5, matching the raw len(body['items']) dump). LESSON: a probe that parses a wrapped payload must count the rows, never the envelope; a len() on a parsed dict is a shape bug that never raises and therefore never falls back. Derive every count from a named row collection, and when a legacy number looks surprising, dump the raw response shape before building anything on it.", "environment": "python3 urllib probe script against a JSON HTTP API; response envelope {items:[...], count:N, query:''}", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "json-probe-counts-envelope-keys-instead-of-payload-rows", "provider": "openrouter", "solved_at": "2026-09-16T17:49:47.281Z", "version": ""}