uv lock && uv lock --check # re-resolve; uv.lock now binds to extra == 'dev'
The repo was /tmp/bench-demo (hermes-sdk, src-layout, uv 0.11.17, py3.14, pytest 9.1.1, pytest-benchmark 5.2.3). Four changes:
1. Move the dep from [dependency-groups] into the existing dev extra, re-lock
uv add --dev wrote a PEP 735 group that pip cannot see. Fix by editing pyproject.toml, then re-locking:
[project.optional-dependencies]
dev = [
"pytest==9.1.1",
"pytest-benchmark==5.2.3", # moved here
]
# [dependency-groups] <- delete (or keep only non-dev groups)
# dev = [
# "pytest-benchmark==5.2.3",
# ]
uv lock && uv lock --check # re-resolve; uv.lock now binds to extra == 'dev'
pip install -e ".[dev]" # classic workflow works again
(Alternative that never writes the group: uv add --optional dev pytest-benchmark.)
2. Replace bare ini keys with addopts — pytest-benchmark 5.2.3 registers no benchmark_* ini keys (only --benchmark-* CLI options), so bare keys trigger PytestConfigWarning and are ignored:
[tool.pytest.ini_options]
addopts = ["--benchmark-min-rounds=3", "--benchmark-max-time=0.5"]
# (remove benchmark_min_rounds / benchmark_max_time bare keys)
3. Sync test driving a dedicated event loop — pytest-benchmark never awaits coroutines; it times construction (~1 µs). The benchmark must be a plain sync def wrapping asyncio.run:
# tests/test_bench_hermes.py
import asyncio
from hermes_sdk import MockHermes
def test_mock_hermes_benchmark(benchmark):
mock = MockHermes()
async def _run() -> str:
return await mock.generate("hello")
def _sync_run() -> str:
return asyncio.run(_run()) # dedicated event loop per round
result = benchmark(_sync_run)
assert result == "reply-to:hello" # assert on the awaited value
4. Gitignore the artifact dir:
.benchmarks/
Bonus gotcha found during verification: pytest's default collection pattern is test_*.py/*_test.py — a file named bench_hermes_buggy.py was silently not collected in suite runs (it only ran when passed explicitly). Rename benchmark modules to test_bench_*.py.
| # | Before (broken) | After (fixed) |
|---|---|---|
| 1 | `uv add --dev` → `[dependency-groups] dev = ["pytest-benchmark==5.2.3"]`. `pip install --dry-run -e '.[dev]'` → *Would install ... pytest-9.1.1 ...* — **no pytest-benchmark** | `uv.lock` shows `{ name = "pytest-benchmark", marker = "extra == 'dev'", specifier = "==5.2.3" }`; `uv lock --check` OK. `pip install --dry-run -e '.[dev]'` → *Collecting pytest-benchmark==5.2.3 (from hermes-sdk==0.1.0)* and *Would install ... pytest-benchmark-5.2.3 ...* |
| 2 | `grep addini` in 5.2.3 → only the `markers` line; all tuning via `addoption`. Runtime: `PytestConfigWarning: Unknown config option: benchmark_max_time / benchmark_min_rounds`; with `-W error::pytest.PytestConfigWarning` → **INTERNALERROR at config time** (breaks strict CI) | Session header: `benchmark: 5.2.3 (defaults: ... min_rounds=3 ... max_time=0.5 ...)`; `uv run pytest -W error::pytest.PytestConfigWarning` → **2 passed, 0 warnings** |
| 3 | `benchmark(mock.generate, "hello")` → mean **932 ns** (coroutine construction), `AssertionError` (result is a coroutine), `RuntimeWarning: coroutine 'MockHermes.generate' was never awaited` | `test_mock_hermes_benchmark` → mean **16.59 ms**, 40 rounds, 2 passed. With `--benchmark-disable`: 2 passed in 0.03 s — wrapper still executes once, assertion still validates correctness |
| 4 | Empty `.benchmarks/` dir created on every run | `.gitignore:2:.benchmarks/` matches via `git check-ignore -v .benchmarks/`; `git add -A -n` stages 0 benchmark files |
Edge cases tested: (a) `asyncio.run()` from a running loop → `RuntimeError: asyncio.run() cannot be called from a running event loop` — proving the benchmark test must remain a plain sync `def` (would break under pytest-asyncio-style async tests); (b) per-round `asyncio.run` loop-creation overhead (~0.1 ms) is negligible against the 5 ms+ mock latency, and `--benchmark-disable` still exercises the assertion path; (c) `pip -q` dry-run output is suppressed, so verification must use verbose pip.{"model": "deepseek-v4-flash", "problem_class": "python-benchmark-pytest-benchmark-setup", "result": "passed", "tests": 2}