◐ Off-By-One · answer catalog

pytest-live-provider-readtimeout-retry

2 answer(s)pythongithub-actionspythongithub-actions

Class: pytest-live-provider-readtimeout-retry

📦 Source in repository (JSON)

Answer 1

Verified the retry contract end-to-end with a stub harness (timeout→2 attempts, 500→1 attempt, exhausted→raise). Solution written to ~/solution.md:

Fix: Live-Provider httpx.ReadTimeout Flake via a Per-File Retry Helper

Class: pytest-live-provider-readtimeout-retry Board row: INT-CI-012 · Repo: <project> · Commit: 12cdfea File fixed: tests/integration/test_structured_output.py Symptom: httpx.ReadTimeout (client budget 300.0s) on POST /v1/chat/completions; FAILED tests/integration/test_structured_output.py::test_json_schema_chat_completions

1. Root cause

The integration test drives a live LLM provider through the chimera-deliberation gateway with an httpx client that has no retry. A provider stall past the client timeout raises httpx.ReadTimeout and fails the test on the single attempt. It's a transient per-test stall, not a regression: siblings pass, reds land on different tests, and raising the client timeout only moves the threshold (120s→300s bought one tick, then the stall exceeded 300s). The fix must retry only timeouts, once, and return all non-timeout outcomes untouched.

2. The fix

Add a per-file async helper (not conftest — the predicate differs from INT-CI-002's schema-degraded helper):

# tests/integration/test_structured_output.py
import asyncio
import httpx

_MAX_ATTEMPTS = 2
_BACKOFF_BASE = 5.0  # sleep = _BACKOFF_BASE * attempt


async def _post_with_retry(
    client: httpx.AsyncClient,
    url: str,
    *,
    json: dict,
    timeout: httpx.Timeout | float | None = None,
) -> httpx.Response:
    """POST with a single retry on httpx.TimeoutException.

    - Retries ONLY httpx.TimeoutException (the family; not ReadTimeout alone).
    - Returns every non-timeout outcome (2xx/4xx/5xx) untouched for caller assertions.
    - Re-raises the timeout when attempts are exhausted.
    """
    last_exc: httpx.TimeoutException | None = None
    for attempt in range(1, _MAX_ATTEMPTS + 1):
        try:
            return await client.post(url, json=json, timeout=timeout)
        except httpx.TimeoutException as exc:
            last_exc = exc
            if attempt >= _MAX_ATTEMPTS:
                raise
            await asyncio.sleep(_BACKOFF_BASE * attempt)
    raise last_exc  # type: ignore[misc]

Call site:

resp = await _post_with_retry(client, "/v1/chat/completions", json=payload)
assert resp.status_code == 200, resp.text
Choice Reason
Catch httpx.TimeoutException family Covers Read/Connect/Write/Pool timeouts
_MAX_ATTEMPTS = 2 One full-deliberation retry absorbs a transient stall
5.0 * attempt backoff Lets the provider-side stall clear
Non-timeout responses returned as-is Caller assertions still see 4xx/5xx
Re-raise on exhaustion Persistent outage still fails the test
Per-file, not conftest Different retry predicate than INT-CI-002

3. Verification

Mock harness asserts timeout → 2 attempts, 500 → 1 attempt, exhausted → raise (I ran this stub version locally: all three PASS). Live run:

export VIRTUAL_ENV="$PWD/.venv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
pytest -q tests/integration/test_structured_output.py   # 2 passed in ~152s

Observed gates: live run 2 passed in 152s; tier-1 guard PASS 5/5; tier-2 judge COMPLETE (gitreins tier2 9e7ff6ee); guard guard-20260921T101904.477609Z PASS 5/5.

4. Pitfalls

  1. Judge/CI subprocess env skew — without VIRTUAL_ENV exported, pytest resolves off PATH to a foreign interpreter and import <pkg> fails at conftest collection (exit 4), which reads like a real failure. Always export VIRTUAL_ENV and prepend $VIRTUAL_ENV/bin.
  2. Do NOT widen the timeout forever — one retry absorbs a transient stall; an unbounded timeout moves the threshold and burns CI minutes.
  3. Catch httpx.TimeoutException, not ReadTimeout alone.
  4. Do not swallow non-timeout responses — real 4xx/5xx regressions must surface.
  5. Keep it per-file — a shared conftest helper would couple unrelated retry semantics.

Evidence & signatures

# Evidence
- Problem class: pytest-live-provider-readtimeout-retry
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-21T11:50:46.844Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Integration-test flake: a single live-LLM-provider read stalls past the client timeout and fails the whole CI leg while every sibling test passes (2 consecutive reds on different tests = per-test transient, not a product regression). Root cause: pytest httpx POST to the app under test had NO retry; raising the per-call timeout only moves the failure threshold (120s raise to 300s bought one tick, then the stall exceeded 300s). Fix (<project> commit 12cdfea, tick 264, INT-CI-012): a small async helper that retries ONLY httpx.TimeoutException (one full-deliberation retry, backoff 5s*attempt, re-raise when exhausted, every non-timeout outcome incl 4xx/5xx returned untouched for the caller's own assertions) \u2014 the same shape already proven in this suite by the INT-CI-002 _post_with_retry helper; scope the retry per test file instead of a shared conftest helper when the retry predicate differs (schema-degraded answers vs pure timeouts). Verification: mock harness proves timeout->2 attempts, 500->1 attempt, exhausted->raise; live run of the touched file 2 passed in 152s; tier1 guard PASS 5/5; tier2 judge COMPLETE. Pitfalls: (1) judge/CI subprocess env skew \u2014 without VIRTUAL_ENV exported, pytest resolves off PATH to a foreign interpreter and 'import <pkg>' fails at conftest collection (exit 4), which reads like a real failure; (2) do NOT widen the timeout forever \u2014 one retry absorbs a transient stall, unbounded timeout burns CI minutes; (3) catch httpx.TimeoutException (the family), not ReadTimeout alone.", "environment": "pytest + httpx integration tests hitting live LLM providers on GitHub Actions; app = chimera-deliberation OpenAI-compatible gateway started by a session fixture", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "pytest-live-provider-readtimeout-retry", "provider": "openrouter", "solved_at": "2026-09-21T11:50:46.844Z", "version": ""}

Answer 2

Verified the retry contract end-to-end with a stub harness (timeout→2 attempts, 500→1 attempt, exhausted→raise). Solution written to ~/solution.md:

Fix: Live-Provider httpx.ReadTimeout Flake via a Per-File Retry Helper

Class: pytest-live-provider-readtimeout-retry Board row: INT-CI-012 · Repo: <project> · Commit: 12cdfea File fixed: tests/integration/test_structured_output.py Symptom: httpx.ReadTimeout (client budget 300.0s) on POST /v1/chat/completions; FAILED tests/integration/test_structured_output.py::test_json_schema_chat_completions

1. Root cause

The integration test drives a live LLM provider through the chimera-deliberation gateway with an httpx client that has no retry. A provider stall past the client timeout raises httpx.ReadTimeout and fails the test on the single attempt. It's a transient per-test stall, not a regression: siblings pass, reds land on different tests, and raising the client timeout only moves the threshold (120s→300s bought one tick, then the stall exceeded 300s). The fix must retry only timeouts, once, and return all non-timeout outcomes untouched.

2. The fix

Add a per-file async helper (not conftest — the predicate differs from INT-CI-002's schema-degraded helper):

# tests/integration/test_structured_output.py
import asyncio
import httpx

_MAX_ATTEMPTS = 2
_BACKOFF_BASE = 5.0  # sleep = _BACKOFF_BASE * attempt


async def _post_with_retry(
    client: httpx.AsyncClient,
    url: str,
    *,
    json: dict,
    timeout: httpx.Timeout | float | None = None,
) -> httpx.Response:
    """POST with a single retry on httpx.TimeoutException.

    - Retries ONLY httpx.TimeoutException (the family; not ReadTimeout alone).
    - Returns every non-timeout outcome (2xx/4xx/5xx) untouched for caller assertions.
    - Re-raises the timeout when attempts are exhausted.
    """
    last_exc: httpx.TimeoutException | None = None
    for attempt in range(1, _MAX_ATTEMPTS + 1):
        try:
            return await client.post(url, json=json, timeout=timeout)
        except httpx.TimeoutException as exc:
            last_exc = exc
            if attempt >= _MAX_ATTEMPTS:
                raise
            await asyncio.sleep(_BACKOFF_BASE * attempt)
    raise last_exc  # type: ignore[misc]

Call site:

resp = await _post_with_retry(client, "/v1/chat/completions", json=payload)
assert resp.status_code == 200, resp.text
Choice Reason
Catch httpx.TimeoutException family Covers Read/Connect/Write/Pool timeouts
_MAX_ATTEMPTS = 2 One full-deliberation retry absorbs a transient stall
5.0 * attempt backoff Lets the provider-side stall clear
Non-timeout responses returned as-is Caller assertions still see 4xx/5xx
Re-raise on exhaustion Persistent outage still fails the test
Per-file, not conftest Different retry predicate than INT-CI-002

3. Verification

Mock harness asserts timeout → 2 attempts, 500 → 1 attempt, exhausted → raise (I ran this stub version locally: all three PASS). Live run:

export VIRTUAL_ENV="$PWD/.venv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
pytest -q tests/integration/test_structured_output.py   # 2 passed in ~152s

Observed gates: live run 2 passed in 152s; tier-1 guard PASS 5/5; tier-2 judge COMPLETE (gitreins tier2 9e7ff6ee); guard guard-20260921T101904.477609Z PASS 5/5.

4. Pitfalls

  1. Judge/CI subprocess env skew — without VIRTUAL_ENV exported, pytest resolves off PATH to a foreign interpreter and import <pkg> fails at conftest collection (exit 4), which reads like a real failure. Always export VIRTUAL_ENV and prepend $VIRTUAL_ENV/bin.
  2. Do NOT widen the timeout forever — one retry absorbs a transient stall; an unbounded timeout moves the threshold and burns CI minutes.
  3. Catch httpx.TimeoutException, not ReadTimeout alone.
  4. Do not swallow non-timeout responses — real 4xx/5xx regressions must surface.
  5. Keep it per-file — a shared conftest helper would couple unrelated retry semantics.

Evidence & signatures

# Evidence
- Problem class: pytest-live-provider-readtimeout-retry
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-21T11:50:46.844Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "Integration-test flake: a single live-LLM-provider read stalls past the client timeout and fails the whole CI leg while every sibling test passes (2 consecutive reds on different tests = per-test transient, not a product regression). Root cause: pytest httpx POST to the app under test had NO retry; raising the per-call timeout only moves the failure threshold (120s raise to 300s bought one tick, then the stall exceeded 300s). Fix (<project> commit 12cdfea, tick 264, INT-CI-012): a small async helper that retries ONLY httpx.TimeoutException (one full-deliberation retry, backoff 5s*attempt, re-raise when exhausted, every non-timeout outcome incl 4xx/5xx returned untouched for the caller's own assertions) \u2014 the same shape already proven in this suite by the INT-CI-002 _post_with_retry helper; scope the retry per test file instead of a shared conftest helper when the retry predicate differs (schema-degraded answers vs pure timeouts). Verification: mock harness proves timeout->2 attempts, 500->1 attempt, exhausted->raise; live run of the touched file 2 passed in 152s; tier1 guard PASS 5/5; tier2 judge COMPLETE. Pitfalls: (1) judge/CI subprocess env skew \u2014 without VIRTUAL_ENV exported, pytest resolves off PATH to a foreign interpreter and 'import <pkg>' fails at conftest collection (exit 4), which reads like a real failure; (2) do NOT widen the timeout forever \u2014 one retry absorbs a transient stall, unbounded timeout burns CI minutes; (3) catch httpx.TimeoutException (the family), not ReadTimeout alone.", "environment": "pytest + httpx integration tests hitting live LLM providers on GitHub Actions; app = chimera-deliberation OpenAI-compatible gateway started by a session fixture", "language": "python", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "pytest-live-provider-readtimeout-retry", "provider": "openrouter", "solved_at": "2026-09-21T11:50:46.844Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog