static-html-docs-formula-replication-drift
Root cause. docs/example-calc.js replicated the tool's recalculate() but omitted the IE-GAP-011 TP×PP floor, so gpusNeeded = max(rMemGpus, rThrGpus) instead of max(…, tp×pp). For the 70B QUICKSTART (TP=2, PP=4): memory said 1 GPU, throughput said 1 GPU → docs printed 1 and $2/hr, while the tool floored to 8 and $16/hr.
The fix (4 parts):
docs/example-calc.js now mirrors tool/calc.js exactly (parity is enforced by tests, so it can't silently drift again):// IE-GAP-011 TP×PP floor — replicated from tool/calc.js (was missing here).
var rTpPpFloor = c.tp * c.pp;
var gpusNeeded = Math.max(rMemGpus, rThrGpus, rTpPpFloor);
docs/generate-quickstart.js computes every number in quickstart.html from the calculator and substitutes {{placeholders}}. It also has a parity gate that refuses to publish if the replication disagrees with the tool:// 1) Parity gate: abort if the docs replication disagrees with the tool.
for (const f of fields) {
if (Math.abs(a[f] - b[f]) > 1e-9) {
throw new Error(`PARITY FAIL: ${f} tool=${a[f]} docs=${b[f]}. ` +
'Fix docs/example-calc.js before regenerating the docs.');
}
}
test.js — literals derived from the tool (not from the docs), tight enough to be mutation-proof:const KA = {
rModelMem: 65.1925802230835, // GiB: 70B × 2B FP16 ÷ TP=2
rKvCache: 0.625, // GiB: K+V × 80L × 8H × 128d × 2B × 4k ctx ÷ TP=2
rGpusNeeded: 8, // IE-GAP-011 floor: TP(2) × PP(4)
rDecodeThroughput: 30, // tok/s/GPU: 60 base ÷ TP=2
costPerHour: 16, costPerDay: 384,
};
t('KA: rKvCache = 0.625 GiB (K+V ×2 included)', () => {
approx(docs.recalculate(QUICKSTART).rKvCache, KA.rKvCache); // fails if ×2 dropped
});
t('KA: gpusNeeded = 8 (TP×PP floor …)', () => {
assert.equal(docs.recalculate(QUICKSTART).gpusNeeded, KA.rGpusNeeded); // fails if floor dropped
});
docs must deep-equal tool on 4 scenarios, and gpusNeeded ≥ tp×pp always holds.All runs against the final tree (`~`, Node v22): - **Suite (fixed):** `14 passed, 0 failed` — known answers, 4 parity checks, 3 edge scenarios, floor invariant, unknown-field guard. - **Regeneration:** `quickstart.html regenerated: gpusNeeded=8, costPerHour=$16, costPerDay=$384`; HTML contains `GPUs Needed = 8` and the corrected cost table. - **Mutation A — drop K+V ×2 (default `kvTensors: 2→1`):** `10 passed, 4 failed` (exit 1). Caught by parity (`expected 160, got 80`) and the memory-bound edge. - **Mutation B — remove TP×PP floor (the original drift):** `11 passed, 3 failed` (exit 1). Caught by `gpusNeeded: 8 ≠ 1`, `costPerHour: 16 ≠ 2`, QUICKSTART parity — exactly the incident signature. - **Mutation C — drop ×2 in the formula (docs only):** `4 passed, 10 failed` — KA `rKvCache`, KA `gpusNeeded`, KA costs, and all parity tests fail. - **Generator parity gate:** aborts with `PARITY FAIL: rTpPpFloor tool=8 docs=1` (exit 1) and writes no HTML — a drifted replication can never be published. - **Edge cases:** single-GPU model not inflated by floor (`gpusNeeded=1`); memory-bound 256k ctx (KV = 160 GiB → 5 GPUs); throughput-bound 1000 tok/s (17 GPUs); `gpusNeeded ≥ tp×pp` on all 4 scenarios.
{"model": "deepseek-v4-flash", "problem_class": "static-html-docs-formula-replication-drift", "result": "passed", "tests": 14}