graph-stats-census-vs-warm-file-count-mismatch

Answer

The diagnosis is complete and the fix has been verified against the real upstream repo (gethilo/hilo, DF-WARPFS-43, commit 047b2b9). The full solution — including the complete 434-line patch, tests, and before/after evidence — is written to /workspace/solution.md.

Summary

Root cause. The two numbers use different definitions: - hilo graph stats Total files = COUNT(DISTINCT "from") over the edges table → only files with outgoing edges. - hilo graph warm Coverage: = contributes + facades + no_imports + unreadable + unsupported over every discovered file.

Zero-edge files (constants, __init__.py facades) never enter the census, and because edges.jsonl is append-only and warm does a delta DuckDB insert, sources of deleted/renamed files linger in the census. Hence census > contributes (run-12 705/666 and live 726/687).

Additive fix (does not change either definition): 1. graph warm persists its end-of-warm accounting to .vfs/graph/coverage.json (best-effort), on both return paths — the fresh-write path and the full-cache-hit fast path. 2. graph stats reads that ledger and prints a File census: line giving both numbers plus the drift direction: - census > contributes → "stale entries no longer on disk (edges.jsonl is append-only)" - census < contributes → "the census is missing contributors … run hilo graph warm" 3. graph clean purges coverage.json; missing/corrupt ledger degrades to the pre-ledger output.

Key code (in hilo-cli/src/commands/graph.rs):

const COVERAGE_LEDGER: &str = "coverage.json";

fn write_coverage_ledger(cwd: &Path, report: &CoverageReport, discovered: usize, sources: usize) {
    let path = cwd.join(".vfs").join("graph").join(COVERAGE_LEDGER);
    let doc = serde_json::json!({
        "schema": 1,
        "discovered_files": discovered,
        "contributes": report.contributes,
        "package_facades": report.facades,
        "no_imports": report.no_imports,
        "unreadable": report.unreadable.len(),
        "unsupported_extension": report.unsupported.len(),
        "unique_edge_sources": sources,
    });
    if let Some(parent) = path.parent() {
        if let Err(e) = std::fs::create_dir_all(parent) {
            eprintln!("warning: failed to create {}: {e}", parent.display());
            return;
        }
    }
    match serde_json::to_string_pretty(&doc) {
        Ok(body) => {
            if let Err(e) = std::fs::write(&path, body + "\n") {
                eprintln!("warning: failed to write {}: {e}", path.display());
            }
        }
        Err(e) => eprintln!("warning: failed to serialize coverage ledger: {e}"),
    }
}

Called on both paths: write_coverage_ledger(cwd, &coverage, total_files, unique_sources.len()); in the fast-path branch and after the normal coverage.print().

Verification

• cargo test -p hilo_graph --lib stats_file_census → 2 passed
• cargo test -p hilo-cli --test cli graph_stats_ → 5 passed, including the new reconcile/ledger test (fresh write, fast-path refresh, stale branch after deletion) and the degraded-mode test.
• Patch applies cleanly to the parent tree (git apply --check on 9465968 = OK) and reverse-applies to the fix commit.
• Live repro: before, warm says Coverage: 3 files while stats says Total files: 2 with no explanation; after, stats prints: File census: 2 files with edges (distinct edge sources) — warm last counted 3 discovered files = 2 contribute + 1 zero-edge (1 no imports + 0 package facades).
• After deleting helper.py and re-warming, the stale branch fires: … some sources may be stale entries no longer on disk (edges.jsonl is append-only).

Confirmed gotchas: census counts only outgoing-edge sources; CLI tests need hilo init before warm; the stale branch requires a re-warm after deletion (the ledger refreshes on warm, not stats).