flaky-integration-test-retry
The solution is written to ~/flaky-integration-test-retry.md.
Root cause: Retry logic was gated entirely on output_schema, and that gate wrapped a single detector. So the aggregator's validation-failure envelope {"passed": false, "errors": [...]} arriving on a non-schema deliberation was never retried, flowed into the HTML check, and raised AssertionError: No html tag or doctype found ….
Fix: Split the detector in tests/integration/test_collaborative_evals.py:
- _is_validation_failure_object(text) — matches {passed: false, errors: [...]} → retry unconditionally (never a legitimate answer).
- _is_degraded_schema_answer(text) — the no-JSON-no-HTML heuristic; stays gated on has_output_schema so prose answers (math proofs) don't burn all retries.
- _should_retry(text, has_output_schema) composes the two, keeping behavior consistent everywhere.
Verification: Includes targeted pytest -k collaborative_static_website, a self-test snippet asserting both detector paths, and a note that a separately-flaking sibling SSE test gets its own row while this issue stays closed.
One caveat: this sandbox doesn't have the chimera checkout, so I couldn't diff the actual file — the two new functions plus the gating change are complete, and the doc flags the two placeholders (_retry_payload(...) and the has_output_schema flag name) to match your exact utility signatures.
# Evidence - Problem class: flaky-integration-test-retry - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-27T21:14:19.085Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "CI flake fix: retry logic gated to output_schema payloads missed the aggregator validation-failure object {\"passed\": false, \"errors\": [...]} arriving on a NON-schema deliberation (test_collaborative_static_website). The validation-failure object is never a legitimate answer \u2014 retry it unconditionally; only the no-JSON-no-HTML heuristic should stay schema-gated (prose answers like math proofs would burn all retries).", "environment": "<project> integration tests, pytest + httpx, CI on GitHub Actions", "language": "python", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "flaky-integration-test-retry", "provider": "openrouter", "solved_at": "2026-08-27T21:14:19.086Z", "version": ""}The solution is written to ~/flaky-integration-test-retry.md.
Root cause: Retry logic was gated entirely on output_schema, and that gate wrapped a single detector. So the aggregator's validation-failure envelope {"passed": false, "errors": [...]} arriving on a non-schema deliberation was never retried, flowed into the HTML check, and raised AssertionError: No html tag or doctype found ….
Fix: Split the detector in tests/integration/test_collaborative_evals.py:
- _is_validation_failure_object(text) — matches {passed: false, errors: [...]} → retry unconditionally (never a legitimate answer).
- _is_degraded_schema_answer(text) — the no-JSON-no-HTML heuristic; stays gated on has_output_schema so prose answers (math proofs) don't burn all retries.
- _should_retry(text, has_output_schema) composes the two, keeping behavior consistent everywhere.
Verification: Includes targeted pytest -k collaborative_static_website, a self-test snippet asserting both detector paths, and a note that a separately-flaking sibling SSE test gets its own row while this issue stays closed.
One caveat: this sandbox doesn't have the chimera checkout, so I couldn't diff the actual file — the two new functions plus the gating change are complete, and the doc flags the two placeholders (_retry_payload(...) and the has_output_schema flag name) to match your exact utility signatures.
# Evidence - Problem class: flaky-integration-test-retry - Model: openrouter/deepseek/deepseek-v4-flash-0731 - Solved: 2026-08-27T21:14:19.085Z - Verification: solution produced by pi in sandbox; see signatures.json
{"description": "CI flake fix: retry logic gated to output_schema payloads missed the aggregator validation-failure object {\"passed\": false, \"errors\": [...]} arriving on a NON-schema deliberation (test_collaborative_static_website). The validation-failure object is never a legitimate answer \u2014 retry it unconditionally; only the no-JSON-no-HTML heuristic should stay schema-gated (prose answers like math proofs would burn all retries).", "environment": "<project> integration tests, pytest + httpx, CI on GitHub Actions", "language": "python", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "flaky-integration-test-retry", "provider": "openrouter", "solved_at": "2026-08-27T21:14:19.086Z", "version": ""}