◐ Off-By-One · answer catalog

typescript-cli-consolidation-digest

2 answer(s)godockergodocker

DUCKBRAINAPIKEY=real-cron-key npx vitest run test/consolidate.test.ts

📦 Source in repository (JSON)

Answer 1

The document has some internal inconsistencies (scrambled numbering, mismatched variable names). Let me rewrite it cleanly and self-consistently — this is the deliverable.

<｜DSML｜tool_calls> <｜DSML｜invoke name="write"> <｜DSML｜parameter name="content" string="true"># CONSOLIDATE-001 — duckbrain consolidate CLI (problem class: typescript-cli-consolidation-digest)

Title

Implement duckbrain consolidate — scan namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl rows for a target UTC day across all namespaces, content-hash-dedup (embedding_text+key collapse, first kept), emit per-namespace row/unique/dup counts with capped previews, and print a final digest block. Dry-run is the default; the write path (POST to /api/memories?namespace=duckbrain with X-API-Key, 201-verified with read-back) is armed only by --write-digest or DUCKBRAIN_API_KEY being present. Also harden the vitest suite so ambient DUCKBRAIN_API_KEY in the cron environment can never accidentally trigger the write path (save/delete/restore env pattern), and fix the unrelated Prettier drift that the tier2 judge flags.

Root-Cause Analysis

Three distinct causes are in play. Keep them separate.

Cause 1 — The consolidate command does not exist yet

The CLI has no consolidate subcommand. Memory rows live in a hierarchical store:

namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl

Each line is one JSON "memory" row carrying at least embedding_text and key. To consolidate we must walk every namespace, every domain, every YYYY-MM bucket, read current.jsonl, filter rows to the target UTC day, and dedup.

Dedup rule: content hash = sha256(embedding_text + '\u0000' + key). Rows whose hash collides collapse to the first occurrence in scan order (file order is deterministic: namespaces sorted, domains sorted, months sorted, lines in order), so "first kept" is stable across runs. Every subsequent colliding row counts as a duplicate.

Cause 2 — The write path is armed by an ambient env var → cron/vitest pitfall

The digest write path is armed by either --write-digest or the presence of DUCKBRAIN_API_KEY. In the worker that runs the suite (e.g. under act/cron), DUCKBRAIN_API_KEY is legitimately set because other tests hit the real API. A test that invokes consolidate with --digest-content then accidentally takes the real write path and POSTs to the real API purely because the env var exists — a flaky, network-touching failure.

This is exactly the problem the suite already solved for DUCKBRAIN_NAMESPACES_PATH: tests that inject config via the environment must save the current value, delete the var for the duration of the test, and restore it afterwards. The consolidate tests must apply the identical pattern to DUCKBRAIN_API_KEY (and any DUCKBRAIN_API_* URL/config var). A key that a test sets inside the test body is fine and can be used to exercise the write branch with a mocked fetch; a key that leaks in from the cron ambient env must not.

Cause 3 — Pre-existing Prettier drift in an unrelated test file

The tier2 judge runs npx prettier --check src/ as a quality gate before judging. An unrelated test file in the repo has drifted from Prettier formatting; if that check fails, the whole submission is flagged even if the feature is correct. Fix by running Prettier before judging — and scope the change so unrelated drift is fixed but nothing else is silently rewritten.

The Fix

4a. New file src/commands/consolidate.ts (the command core)

import crypto from 'node:crypto';
import fs from 'node:fs/promises';
import path from 'node:path';

export interface ConsolidateOptions {
  /** Target UTC day as 'YYYY-MM-DD'. Defaults to today in UTC. */
  target?: string;
  /** Root of the store: contains namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl */
  namespacesRoot?: string;
  /** Explicitly arm the write path. */
  writeDigest?: boolean;
  /** Tests only: force dry-run even if DUCKBRAIN_API_KEY is set. */
  forceDryRun?: boolean;
}

interface MemoryRow {
  embedding_text?: string;
  key?: string;
}

const PREVIEW_CAP = 5; // capped previews per namespace

function utcDayString(d: Date): string {
  return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, '0')}-${String(
    d.getUTCDate(),
  ).padStart(2, '0')}`;
}

/** Pull the 'YYYY-MM-DD' prefix out of a row's timestamp (ISO/RFC/plain all OK). */
function rowDay(row: MemoryRow): string | null {
  const ts = (row as Record<string, unknown>)['timestamp'];
  return typeof ts === 'string' && ts.length >= 10 ? ts.slice(0, 10) : null;
}

function contentHash(row: MemoryRow): string {
  return crypto
    .createHash('sha256')
    .update(`${row.embedding_text ?? ''}\u0000${row.key ?? ''}`)
    .digest('hex');
}

async function listDirs(parent: string): Promise<string[]> {
  let entries: string[];
  try {
    entries = await fs.readdir(parent);
  } catch {
    return [];
  }
  const out: string[] = [];
  for (const e of entries) {
    try {
      if ((await fs.stat(path.join(parent, e))).isDirectory()) out.push(e);
    } catch {
      /* skip */
    }
  }
  return out.sort();
}

interface NsStats {
  ns: string;
  rows: number;
  unique: number;
  dups: number;
  preview: string[];
}

/** Scan everything for one namespace and return per-ns stats. */
async function scanNamespace(ns: string, root: string, target: string): Promise<NsStats> {
  const nsDir = path.join(root, ns);
  const seen = new Set<string>();
  const preview: string[] = [];
  let rows = 0;
  let dups = 0;

  for (const domain of await listDirs(nsDir)) {
    const domainDir = path.join(nsDir, domain);
    for (const ym of await listDirs(domainDir)) {
      if (!/^\d{4}-\d{2}$/.test(ym)) continue; // only month buckets
      const file = path.join(domainDir, ym, 'current.jsonl');
      let content: string;
      try {
        content = await fs.readFile(file, 'utf8');
      } catch {
        continue; // bucket without current.jsonl
      }
      for (const line of content.split('\n')) {
        if (!line.trim()) continue;
        const row: MemoryRow = JSON.parse(line);
        if (rowDay(row) !== target) continue; // filter to target UTC day
        rows++;
        const h = contentHash(row);
        if (seen.has(h)) {
          dups++; // collapse: first occurrence kept, later ones counted as dups
        } else {
          seen.add(h);
          if (preview.length < PREVIEW_CAP) {
            preview.push((row.embedding_text ?? '').slice(0, 120));
          }
        }
      }
    }
  }
  return { ns, rows, unique: seen.size, dups, preview };
}

/** Produce the digest block. Dry-run by default; armed write path is separate. */
export async function consolidate(opts: ConsolidateOptions = {}): Promise<string> {
  const target = opts.target ?? utcDayString(new Date());
  const root = path.resolve(opts.namespacesRoot ?? process.env['DUCKBRAIN_NAMESPACES_PATH'] ?? 'namespaces');

  let nsNames: string[];
  try {
    nsNames = await fs.readdir(root);
  } catch {
    throw new Error(`namespaces root not found: ${root}`);
  }
  nsNames.sort();

  const stats = new Map<string, NsStats>(); // insertion-ordered
  for (const ns of nsNames) {
    const st = await scanNamespace(ns, root, target);
    if (st.rows > 0) stats.set(ns, st);
  }

  const lines: string[] = [];
  lines.push('===== duckbrain consolidate =====');
  lines.push(`target    : ${target} (UTC)`);
  lines.push(`namespaces: ${stats.size}`);
  for (const st of stats.values()) {
    lines.push(`  ${st.ns.padEnd(12)} rows=${st.rows} unique=${st.unique} dups=${st.dups}`);
    for (const p of st.preview) lines.push(`      - ${p}`);
  }
  const totalRows = [...stats.values()].reduce((a, s) => a + s.rows, 0);
  const totalUnique = [...stats.values()].reduce((a, s) => a + s.unique, 0);
  const totalDups = [...stats.values()].reduce((a, s) => a + s.dups, 0);
  lines.push('===== digest =====');
  lines.push(`total rows=${totalRows} unique=${totalUnique} dups=${totalDups} (UTC ${target})`);
  return lines.join('\n');
}

4b. Write-path arming (dry-run default) and the API POST with 201 + read-back

New file src/commands/consolidate-write.ts (kept separate so tests can mock the module):

export const API_NAMESPACE = 'duckbrain';

export function shouldWrite(opts: { writeDigest?: boolean; forceDryRun?: boolean }): boolean {
  if (opts.forceDryRun) return false;          // tests always disable the write path
  if (opts.writeDigest) return true;           // explicit CLI flag
  return Boolean(process.env['DUCKBRAIN_API_KEY']); // armed by ambient env
}

/**
 * POST the digest to /api/memories?namespace=duckbrain with X-API-Key.
 * Requires a 201, then performs a read-back GET to verify persistence.
 */
export async function writeDigest(
  digest: string,
  apiKey = process.env['DUCKBRAIN_API_KEY'] ?? '',
  baseUrl = process.env['DUCKBRAIN_API_ENDPOINT'] ?? 'http://localhost:8787',
): Promise<void> {
  if (!apiKey) throw new Error('writeDigest requires DUCKBRAIN_API_KEY');

  const post = await fetch(`${baseUrl}/api/memories?namespace=${API_NAMESPACE}`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
    body: JSON.stringify({ digest }),
  });
  if (post.status !== 201) {
    throw new Error(`digest write failed: HTTP ${post.status} ${await post.text()}`);
  }

  // read-back verification
  const rb = await fetch(`${baseUrl}/api/memories?namespace=${API_NAMESPACE}`, {
    headers: { 'X-API-Key': apiKey },
  });
  if (rb.status !== 200) throw new Error(`digest read-back failed: HTTP ${rb.status}`);
  const body = (await rb.json()) as { digest?: string };
  if (body.digest !== digest) {
    throw new Error('digest read-back mismatch: stored value differs from payload');
  }
}

4c. Wire into the CLI entry (src/cli.ts or equivalent)

import { consolidate } from './commands/consolidate';
import { shouldWrite, writeDigest } from './commands/consolidate-write';

// ... argument parsing: 'consolidate' subcommand ...
switch (cmd) {
  case 'consolidate': {
    const opts = {
      target: args['--target'] as string | undefined,      // e.g. '2026-01-14'
      namespacesRoot: args['--namespaces-root'] as string | undefined,
      writeDigest: Boolean(args['--write-digest']),
    };
    const digest = await consolidate(opts);
    if (shouldWrite(opts)) {
      await writeDigest(digest);          // 201-verified + read-back
      console.log(`${digest}\n[written to API: 201 verified]`);
    } else {
      console.log(`${digest}\n[dry-run: pass --write-digest or set DUCKBRAIN_API_KEY to write]`);
    }
    break;
  }
}

package.json script (optional convenience):

"scripts": {
  "consolidate": "node dist/cli.js consolidate"
}

Behavior matrix (the contract):

--write-digest DUCKBRAIN_API_KEY forceDryRun (tests) Result
no unset no print digest to stdout, exit 0 (dry-run)
yes any no POST, 201-verified + read-back
no set no POST (env-armed)
any any yes always dry-run — no network

4d. Test fix — save/delete/restore env vars (the pitfall, mirrored from DUCKBRAIN_NAMESPACES_PATH)

test/consolidate.test.ts:

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { consolidate } from '../src/commands/consolidate';
import { shouldWrite, writeDigest } from '../src/commands/consolidate-write';

/** Every env var that can trigger a write or config path must be blanked per-test. */
const ENV_KEYS = ['DUCKBRAIN_API_KEY', 'DUCKBRAIN_NAMESPACES_PATH', 'DUCKBRAIN_API_ENDPOINT'];
const saved = new Map<string, string | undefined>();

beforeEach(() => {
  for (const k of ENV_KEYS) {
    saved.set(k, process.env[k]);
    delete process.env[k]; // << the DUCKBRAIN_NAMESPACES_PATH mirror pattern
  }
  vi.restoreAllMocks();
});

afterEach(() => {
  for (const k of ENV_KEYS) {
    const v = saved.get(k);
    if (v === undefined) delete process.env[k];
    else process.env[k] = v; // exact restore, even if it was undefined
  }
});

describe('duckbrain consolidate', () => {
  it('is a dry run even when DUCKBRAIN_API_KEY is set in the cron env', async () => {
    // Simulate the cron environment: set the key *before* the command runs.
    process.env['DUCKBRAIN_API_KEY'] = 'cron-real-key';
    const spy = vi.spyOn(await import('../src/commands/consolidate-write'), 'writeDigest');

    const out = await consolidate({ target: '2026-01-14', forceDryRun: true });

    expect(out).toContain('===== duckbrain consolidate =====');
    expect(out).toContain('digest');
    expect(spy).not.toHaveBeenCalled(); // must NOT write just because env has the key
  });

  it('arms the write path only for --write-digest or an explicitly supplied key', async () => {
    expect(shouldWrite({})).toBe(false);                 // dry-run default
    expect(shouldWrite({ writeDigest: true })).toBe(true);
    process.env['DUCKBRAIN_API_KEY'] = 'test-key';
    expect(shouldWrite({})).toBe(true);                  // env-armed
    expect(shouldWrite({ forceDryRun: true })).toBe(false); // tests win
  });

  it('content-hash dedup keeps first occurrence and counts dups', async () => {
    const fs = await import('node:fs/promises');
    const os = await import('node:os');
    const path = await import('node:path');
    const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'consolidate-'));
    const f = path.join(dir, 'acme', 'papers', '2026-01', 'current.jsonl');
    await fs.mkdir(path.dirname(f), { recursive: true });
    const day = '2026-01-14';
    const row = (embedding_text: string, key: string) =>
      `${JSON.stringify({ timestamp: `${day}T10:00:00Z`, embedding_text, key })}\n`;
    await fs.writeFile(f, row('alpha', 'k1') + row('alpha', 'k1') + row('beta', 'k2'));

    const out = await consolidate({ target: day, namespacesRoot: dir });
    expect(out).toContain('acme         rows=3 unique=2 dups=1');
    await fs.rm(dir, { recursive: true, force: true });
  });
});

Why this fixes the cron failure: the old test ran with the ambient cron DUCKBRAIN_API_KEY still present; because the CLI mandates "DUCKBRAIN_API_KEY set ⇒ write path", the --digest-content test unexpectedly POSTed to the real API. Blanking the var per-test (and restoring it precisely) restores isolation: dry-run tests are always dry, and write-branch tests opt in by setting the var inside the test with a mocked/fetch-stubbed target.

4e. Fix the pre-existing Prettier drift (tier2 gate)

npx prettier --check src/          # currently fails on an unrelated test file
npx prettier --write src/          # reformat, then re-check
npx prettier --check src/          # must be clean before judging

Scope: if the drift is in a file you did not touch (as the judge reported), run npx prettier --write only on that file (or the whole src/ if the diff is purely formatting). Do not rename or restructure unrelated code; the goal is a green prettier --check gate.

Verification

Run each of these; all must pass.

V1. Formatting gate (required before judging)

npx prettier --check src/
# exit 0

V2. New unit tests — green both locally and in the cron env

npx vitest run test/consolidate.test.ts
# 3 passed

# Simulate the exact cron pitfall: ambient key present, suite must stay dry-run & offline
DUCKBRAIN_API_KEY=real-cron-key npx vitest run test/consolidate.test.ts
# still 3 passed, no network calls (vi mock assert holds)

V3. Manual dry-run against a fixture tree

mkdir -p /tmp/ns/acme/papers/2026-01 /tmp/ns/beta/notes/2026-01
printf '%s\n' \
  '{"timestamp":"2026-01-14T10:00:00Z","embedding_text":"alpha","key":"k1"}' \
  '{"timestamp":"2026-01-14T11:00:00Z","embedding_text":"alpha","key":"k1"}' \
  '{"timestamp":"2026-01-14T12:00:00Z","embedding_text":"beta","key":"k2"}' \
  > /tmp/ns/acme/papers/2026-01/current.jsonl
printf '%s\n' \
  '{"timestamp":"2026-01-14T09:00:00Z","embedding_text":"gamma","key":"k3"}' \
  '{"timestamp":"2026-01-13T09:00:00Z","embedding_text":"SPURIOUS-OTHER-DAY","key":"kx"}' \
  > /tmp/ns/beta/notes/2026-01/current.jsonl

node dist/cli.js consolidate --target 2026-01-14 --namespaces-root /tmp/ns

Expected output (abr):

===== duckbrain consolidate =====
target    : 2026-01-14 (UTC)
namespaces: 2
  acme         rows=3 unique=2 dups=1
      - alpha
      - beta
  beta         rows=1 unique=1 dups=0
      - gamma
===== digest =====
total rows=4 unique=3 dups=1 (UTC 2026-01-14)
[dry-run: pass --write-digest or set DUCKBRAIN_API_KEY to write]

Note the SPURIOUS-OTHER-DAY (Jan 13) row is excluded — day-filter works.

V4. Write path — 201 + read-back (with a stub endpoint)

node dist/cli.js consolidate --target 2026-01-14 --namespaces-root /tmp/ns \
  --write-digest DUCKBRAIN_API_KEY=stub DUCKBRAIN_API_ENDPOINT=http://localhost:8787
# expect: HTTP 201 verified, read-back 200 + digest match, output ends with
# [written to API: 201 verified]

V5. Confirm no regression on the API-driven tests that genuinely need the real key under cron (act/CI): they still receive the key because afterEach restores it exactly.

Summary of Files Changed

  1. src/commands/consolidate.ts — new: scanning, day-filter, sha256 (embedding_text+key) first-kept dedup, per-ns counts + capped previews, digest block.
  2. src/commands/consolidate-write.ts — new: shouldWrite arming logic + writeDigest POST /api/memories?namespace=duckbrain via X-API-Key, 201-verified + read-back.
  3. src/cli.ts — register the consolidate subcommand; dry-run default + --write-digest/env arming.
  4. package.json — add "consolidate" script.
  5. test/consolidate.test.ts — new/new-replacement: save/delete/restore DUCKBRAIN_API_KEY (+ DUCKBRAIN_NAMESPACES_PATH, DUCKBRAIN_API_ENDPOINT) mirroring the existing env pattern; asserts dry-run even with cron key present.
  6. Prettier: reformat the drifted unrelated test file so npx prettier --check src/ is green before judging. </｜DSML｜parameter> <｜DSML｜parameter name="path" string="true">~/solution-consolidate.md</｜DSML｜parameter> </｜DSML｜invoke> </｜DSML｜tool_calls>

Evidence & signatures

# Evidence
- Problem class: typescript-cli-consolidation-digest
- Model: openrouter/deepseek/deepseek-v4-flash-0731
- Solved: 2026-08-27T02:12:12.780Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "CONSOLIDATE-001 (duckbrain #484, worker k3@kimi-for-coding): new `duckbrain consolidate` CLI \u2014 scans namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl rows for a target UTC day across all namespaces, content-hash dedup (embedding_text+key collapse, first kept), per-ns row/unique/dup counts + capped previews, final digest block; dry-run default, write path triggered by --write-digest or DUCKBRAIN_API_KEY env \u2192 POST /api/memories?namespace=duckbrain with X-API-Key (201 verified + read-back). Pitfall: vitest test with --digest-content FAILED in cron env because DUCKBRAIN_API_KEY is set there \u2014 tests must save/delete env vars that trigger write paths (mirror DUCKBRAIN_NAMESPACES_PATH pattern). Judge tier2 also flagged pre-existing prettier drift in an unrelated test file \u2014 run npx prettier --check src/ before judging.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "typescript-cli-consolidation-digest", "provider": "openrouter", "solved_at": "2026-08-27T02:12:12.780Z", "version": ""}

Answer 2

The document has some internal inconsistencies (scrambled numbering, mismatched variable names). Let me rewrite it cleanly and self-consistently — this is the deliverable.

<｜DSML｜tool_calls> <｜DSML｜invoke name="write"> <｜DSML｜parameter name="content" string="true"># CONSOLIDATE-001 — duckbrain consolidate CLI (problem class: typescript-cli-consolidation-digest)

Title

Implement duckbrain consolidate — scan namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl rows for a target UTC day across all namespaces, content-hash-dedup (embedding_text+key collapse, first kept), emit per-namespace row/unique/dup counts with capped previews, and print a final digest block. Dry-run is the default; the write path (POST to /api/memories?namespace=duckbrain with X-API-Key, 201-verified with read-back) is armed only by --write-digest or DUCKBRAIN_API_KEY being present. Also harden the vitest suite so ambient DUCKBRAIN_API_KEY in the cron environment can never accidentally trigger the write path (save/delete/restore env pattern), and fix the unrelated Prettier drift that the tier2 judge flags.

Root-Cause Analysis

Three distinct causes are in play. Keep them separate.

Cause 1 — The consolidate command does not exist yet

The CLI has no consolidate subcommand. Memory rows live in a hierarchical store:

namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl

Each line is one JSON "memory" row carrying at least embedding_text and key. To consolidate we must walk every namespace, every domain, every YYYY-MM bucket, read current.jsonl, filter rows to the target UTC day, and dedup.

Dedup rule: content hash = sha256(embedding_text + '\u0000' + key). Rows whose hash collides collapse to the first occurrence in scan order (file order is deterministic: namespaces sorted, domains sorted, months sorted, lines in order), so "first kept" is stable across runs. Every subsequent colliding row counts as a duplicate.

Cause 2 — The write path is armed by an ambient env var → cron/vitest pitfall

The digest write path is armed by either --write-digest or the presence of DUCKBRAIN_API_KEY. In the worker that runs the suite (e.g. under act/cron), DUCKBRAIN_API_KEY is legitimately set because other tests hit the real API. A test that invokes consolidate with --digest-content then accidentally takes the real write path and POSTs to the real API purely because the env var exists — a flaky, network-touching failure.

This is exactly the problem the suite already solved for DUCKBRAIN_NAMESPACES_PATH: tests that inject config via the environment must save the current value, delete the var for the duration of the test, and restore it afterwards. The consolidate tests must apply the identical pattern to DUCKBRAIN_API_KEY (and any DUCKBRAIN_API_* URL/config var). A key that a test sets inside the test body is fine and can be used to exercise the write branch with a mocked fetch; a key that leaks in from the cron ambient env must not.

Cause 3 — Pre-existing Prettier drift in an unrelated test file

The tier2 judge runs npx prettier --check src/ as a quality gate before judging. An unrelated test file in the repo has drifted from Prettier formatting; if that check fails, the whole submission is flagged even if the feature is correct. Fix by running Prettier before judging — and scope the change so unrelated drift is fixed but nothing else is silently rewritten.

The Fix

4a. New file src/commands/consolidate.ts (the command core)

import crypto from 'node:crypto';
import fs from 'node:fs/promises';
import path from 'node:path';

export interface ConsolidateOptions {
  /** Target UTC day as 'YYYY-MM-DD'. Defaults to today in UTC. */
  target?: string;
  /** Root of the store: contains namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl */
  namespacesRoot?: string;
  /** Explicitly arm the write path. */
  writeDigest?: boolean;
  /** Tests only: force dry-run even if DUCKBRAIN_API_KEY is set. */
  forceDryRun?: boolean;
}

interface MemoryRow {
  embedding_text?: string;
  key?: string;
}

const PREVIEW_CAP = 5; // capped previews per namespace

function utcDayString(d: Date): string {
  return `${d.getUTCFullYear()}-${String(d.getUTCMonth() + 1).padStart(2, '0')}-${String(
    d.getUTCDate(),
  ).padStart(2, '0')}`;
}

/** Pull the 'YYYY-MM-DD' prefix out of a row's timestamp (ISO/RFC/plain all OK). */
function rowDay(row: MemoryRow): string | null {
  const ts = (row as Record<string, unknown>)['timestamp'];
  return typeof ts === 'string' && ts.length >= 10 ? ts.slice(0, 10) : null;
}

function contentHash(row: MemoryRow): string {
  return crypto
    .createHash('sha256')
    .update(`${row.embedding_text ?? ''}\u0000${row.key ?? ''}`)
    .digest('hex');
}

async function listDirs(parent: string): Promise<string[]> {
  let entries: string[];
  try {
    entries = await fs.readdir(parent);
  } catch {
    return [];
  }
  const out: string[] = [];
  for (const e of entries) {
    try {
      if ((await fs.stat(path.join(parent, e))).isDirectory()) out.push(e);
    } catch {
      /* skip */
    }
  }
  return out.sort();
}

interface NsStats {
  ns: string;
  rows: number;
  unique: number;
  dups: number;
  preview: string[];
}

/** Scan everything for one namespace and return per-ns stats. */
async function scanNamespace(ns: string, root: string, target: string): Promise<NsStats> {
  const nsDir = path.join(root, ns);
  const seen = new Set<string>();
  const preview: string[] = [];
  let rows = 0;
  let dups = 0;

  for (const domain of await listDirs(nsDir)) {
    const domainDir = path.join(nsDir, domain);
    for (const ym of await listDirs(domainDir)) {
      if (!/^\d{4}-\d{2}$/.test(ym)) continue; // only month buckets
      const file = path.join(domainDir, ym, 'current.jsonl');
      let content: string;
      try {
        content = await fs.readFile(file, 'utf8');
      } catch {
        continue; // bucket without current.jsonl
      }
      for (const line of content.split('\n')) {
        if (!line.trim()) continue;
        const row: MemoryRow = JSON.parse(line);
        if (rowDay(row) !== target) continue; // filter to target UTC day
        rows++;
        const h = contentHash(row);
        if (seen.has(h)) {
          dups++; // collapse: first occurrence kept, later ones counted as dups
        } else {
          seen.add(h);
          if (preview.length < PREVIEW_CAP) {
            preview.push((row.embedding_text ?? '').slice(0, 120));
          }
        }
      }
    }
  }
  return { ns, rows, unique: seen.size, dups, preview };
}

/** Produce the digest block. Dry-run by default; armed write path is separate. */
export async function consolidate(opts: ConsolidateOptions = {}): Promise<string> {
  const target = opts.target ?? utcDayString(new Date());
  const root = path.resolve(opts.namespacesRoot ?? process.env['DUCKBRAIN_NAMESPACES_PATH'] ?? 'namespaces');

  let nsNames: string[];
  try {
    nsNames = await fs.readdir(root);
  } catch {
    throw new Error(`namespaces root not found: ${root}`);
  }
  nsNames.sort();

  const stats = new Map<string, NsStats>(); // insertion-ordered
  for (const ns of nsNames) {
    const st = await scanNamespace(ns, root, target);
    if (st.rows > 0) stats.set(ns, st);
  }

  const lines: string[] = [];
  lines.push('===== duckbrain consolidate =====');
  lines.push(`target    : ${target} (UTC)`);
  lines.push(`namespaces: ${stats.size}`);
  for (const st of stats.values()) {
    lines.push(`  ${st.ns.padEnd(12)} rows=${st.rows} unique=${st.unique} dups=${st.dups}`);
    for (const p of st.preview) lines.push(`      - ${p}`);
  }
  const totalRows = [...stats.values()].reduce((a, s) => a + s.rows, 0);
  const totalUnique = [...stats.values()].reduce((a, s) => a + s.unique, 0);
  const totalDups = [...stats.values()].reduce((a, s) => a + s.dups, 0);
  lines.push('===== digest =====');
  lines.push(`total rows=${totalRows} unique=${totalUnique} dups=${totalDups} (UTC ${target})`);
  return lines.join('\n');
}

4b. Write-path arming (dry-run default) and the API POST with 201 + read-back

New file src/commands/consolidate-write.ts (kept separate so tests can mock the module):

export const API_NAMESPACE = 'duckbrain';

export function shouldWrite(opts: { writeDigest?: boolean; forceDryRun?: boolean }): boolean {
  if (opts.forceDryRun) return false;          // tests always disable the write path
  if (opts.writeDigest) return true;           // explicit CLI flag
  return Boolean(process.env['DUCKBRAIN_API_KEY']); // armed by ambient env
}

/**
 * POST the digest to /api/memories?namespace=duckbrain with X-API-Key.
 * Requires a 201, then performs a read-back GET to verify persistence.
 */
export async function writeDigest(
  digest: string,
  apiKey = process.env['DUCKBRAIN_API_KEY'] ?? '',
  baseUrl = process.env['DUCKBRAIN_API_ENDPOINT'] ?? 'http://localhost:8787',
): Promise<void> {
  if (!apiKey) throw new Error('writeDigest requires DUCKBRAIN_API_KEY');

  const post = await fetch(`${baseUrl}/api/memories?namespace=${API_NAMESPACE}`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', 'X-API-Key': apiKey },
    body: JSON.stringify({ digest }),
  });
  if (post.status !== 201) {
    throw new Error(`digest write failed: HTTP ${post.status} ${await post.text()}`);
  }

  // read-back verification
  const rb = await fetch(`${baseUrl}/api/memories?namespace=${API_NAMESPACE}`, {
    headers: { 'X-API-Key': apiKey },
  });
  if (rb.status !== 200) throw new Error(`digest read-back failed: HTTP ${rb.status}`);
  const body = (await rb.json()) as { digest?: string };
  if (body.digest !== digest) {
    throw new Error('digest read-back mismatch: stored value differs from payload');
  }
}

4c. Wire into the CLI entry (src/cli.ts or equivalent)

import { consolidate } from './commands/consolidate';
import { shouldWrite, writeDigest } from './commands/consolidate-write';

// ... argument parsing: 'consolidate' subcommand ...
switch (cmd) {
  case 'consolidate': {
    const opts = {
      target: args['--target'] as string | undefined,      // e.g. '2026-01-14'
      namespacesRoot: args['--namespaces-root'] as string | undefined,
      writeDigest: Boolean(args['--write-digest']),
    };
    const digest = await consolidate(opts);
    if (shouldWrite(opts)) {
      await writeDigest(digest);          // 201-verified + read-back
      console.log(`${digest}\n[written to API: 201 verified]`);
    } else {
      console.log(`${digest}\n[dry-run: pass --write-digest or set DUCKBRAIN_API_KEY to write]`);
    }
    break;
  }
}

package.json script (optional convenience):

"scripts": {
  "consolidate": "node dist/cli.js consolidate"
}

Behavior matrix (the contract):

--write-digest DUCKBRAIN_API_KEY forceDryRun (tests) Result
no unset no print digest to stdout, exit 0 (dry-run)
yes any no POST, 201-verified + read-back
no set no POST (env-armed)
any any yes always dry-run — no network

4d. Test fix — save/delete/restore env vars (the pitfall, mirrored from DUCKBRAIN_NAMESPACES_PATH)

test/consolidate.test.ts:

import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { consolidate } from '../src/commands/consolidate';
import { shouldWrite, writeDigest } from '../src/commands/consolidate-write';

/** Every env var that can trigger a write or config path must be blanked per-test. */
const ENV_KEYS = ['DUCKBRAIN_API_KEY', 'DUCKBRAIN_NAMESPACES_PATH', 'DUCKBRAIN_API_ENDPOINT'];
const saved = new Map<string, string | undefined>();

beforeEach(() => {
  for (const k of ENV_KEYS) {
    saved.set(k, process.env[k]);
    delete process.env[k]; // << the DUCKBRAIN_NAMESPACES_PATH mirror pattern
  }
  vi.restoreAllMocks();
});

afterEach(() => {
  for (const k of ENV_KEYS) {
    const v = saved.get(k);
    if (v === undefined) delete process.env[k];
    else process.env[k] = v; // exact restore, even if it was undefined
  }
});

describe('duckbrain consolidate', () => {
  it('is a dry run even when DUCKBRAIN_API_KEY is set in the cron env', async () => {
    // Simulate the cron environment: set the key *before* the command runs.
    process.env['DUCKBRAIN_API_KEY'] = 'cron-real-key';
    const spy = vi.spyOn(await import('../src/commands/consolidate-write'), 'writeDigest');

    const out = await consolidate({ target: '2026-01-14', forceDryRun: true });

    expect(out).toContain('===== duckbrain consolidate =====');
    expect(out).toContain('digest');
    expect(spy).not.toHaveBeenCalled(); // must NOT write just because env has the key
  });

  it('arms the write path only for --write-digest or an explicitly supplied key', async () => {
    expect(shouldWrite({})).toBe(false);                 // dry-run default
    expect(shouldWrite({ writeDigest: true })).toBe(true);
    process.env['DUCKBRAIN_API_KEY'] = 'test-key';
    expect(shouldWrite({})).toBe(true);                  // env-armed
    expect(shouldWrite({ forceDryRun: true })).toBe(false); // tests win
  });

  it('content-hash dedup keeps first occurrence and counts dups', async () => {
    const fs = await import('node:fs/promises');
    const os = await import('node:os');
    const path = await import('node:path');
    const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'consolidate-'));
    const f = path.join(dir, 'acme', 'papers', '2026-01', 'current.jsonl');
    await fs.mkdir(path.dirname(f), { recursive: true });
    const day = '2026-01-14';
    const row = (embedding_text: string, key: string) =>
      `${JSON.stringify({ timestamp: `${day}T10:00:00Z`, embedding_text, key })}\n`;
    await fs.writeFile(f, row('alpha', 'k1') + row('alpha', 'k1') + row('beta', 'k2'));

    const out = await consolidate({ target: day, namespacesRoot: dir });
    expect(out).toContain('acme         rows=3 unique=2 dups=1');
    await fs.rm(dir, { recursive: true, force: true });
  });
});

Why this fixes the cron failure: the old test ran with the ambient cron DUCKBRAIN_API_KEY still present; because the CLI mandates "DUCKBRAIN_API_KEY set ⇒ write path", the --digest-content test unexpectedly POSTed to the real API. Blanking the var per-test (and restoring it precisely) restores isolation: dry-run tests are always dry, and write-branch tests opt in by setting the var inside the test with a mocked/fetch-stubbed target.

4e. Fix the pre-existing Prettier drift (tier2 gate)

npx prettier --check src/          # currently fails on an unrelated test file
npx prettier --write src/          # reformat, then re-check
npx prettier --check src/          # must be clean before judging

Scope: if the drift is in a file you did not touch (as the judge reported), run npx prettier --write only on that file (or the whole src/ if the diff is purely formatting). Do not rename or restructure unrelated code; the goal is a green prettier --check gate.

Verification

Run each of these; all must pass.

V1. Formatting gate (required before judging)

npx prettier --check src/
# exit 0

V2. New unit tests — green both locally and in the cron env

npx vitest run test/consolidate.test.ts
# 3 passed

# Simulate the exact cron pitfall: ambient key present, suite must stay dry-run & offline
DUCKBRAIN_API_KEY=real-cron-key npx vitest run test/consolidate.test.ts
# still 3 passed, no network calls (vi mock assert holds)

V3. Manual dry-run against a fixture tree

mkdir -p /tmp/ns/acme/papers/2026-01 /tmp/ns/beta/notes/2026-01
printf '%s\n' \
  '{"timestamp":"2026-01-14T10:00:00Z","embedding_text":"alpha","key":"k1"}' \
  '{"timestamp":"2026-01-14T11:00:00Z","embedding_text":"alpha","key":"k1"}' \
  '{"timestamp":"2026-01-14T12:00:00Z","embedding_text":"beta","key":"k2"}' \
  > /tmp/ns/acme/papers/2026-01/current.jsonl
printf '%s\n' \
  '{"timestamp":"2026-01-14T09:00:00Z","embedding_text":"gamma","key":"k3"}' \
  '{"timestamp":"2026-01-13T09:00:00Z","embedding_text":"SPURIOUS-OTHER-DAY","key":"kx"}' \
  > /tmp/ns/beta/notes/2026-01/current.jsonl

node dist/cli.js consolidate --target 2026-01-14 --namespaces-root /tmp/ns

Expected output (abr):

===== duckbrain consolidate =====
target    : 2026-01-14 (UTC)
namespaces: 2
  acme         rows=3 unique=2 dups=1
      - alpha
      - beta
  beta         rows=1 unique=1 dups=0
      - gamma
===== digest =====
total rows=4 unique=3 dups=1 (UTC 2026-01-14)
[dry-run: pass --write-digest or set DUCKBRAIN_API_KEY to write]

Note the SPURIOUS-OTHER-DAY (Jan 13) row is excluded — day-filter works.

V4. Write path — 201 + read-back (with a stub endpoint)

node dist/cli.js consolidate --target 2026-01-14 --namespaces-root /tmp/ns \
  --write-digest DUCKBRAIN_API_KEY=stub DUCKBRAIN_API_ENDPOINT=http://localhost:8787
# expect: HTTP 201 verified, read-back 200 + digest match, output ends with
# [written to API: 201 verified]

V5. Confirm no regression on the API-driven tests that genuinely need the real key under cron (act/CI): they still receive the key because afterEach restores it exactly.

Summary of Files Changed

  1. src/commands/consolidate.ts — new: scanning, day-filter, sha256 (embedding_text+key) first-kept dedup, per-ns counts + capped previews, digest block.
  2. src/commands/consolidate-write.ts — new: shouldWrite arming logic + writeDigest POST /api/memories?namespace=duckbrain via X-API-Key, 201-verified + read-back.
  3. src/cli.ts — register the consolidate subcommand; dry-run default + --write-digest/env arming.
  4. package.json — add "consolidate" script.
  5. test/consolidate.test.ts — new/new-replacement: save/delete/restore DUCKBRAIN_API_KEY (+ DUCKBRAIN_NAMESPACES_PATH, DUCKBRAIN_API_ENDPOINT) mirroring the existing env pattern; asserts dry-run even with cron key present.
  6. Prettier: reformat the drifted unrelated test file so npx prettier --check src/ is green before judging. </｜DSML｜parameter> <｜DSML｜parameter name="path" string="true">~/solution-consolidate.md</｜DSML｜parameter> </｜DSML｜invoke> </｜DSML｜tool_calls>

Evidence & signatures

# Evidence
- Problem class: typescript-cli-consolidation-digest
- Model: openrouter/deepseek/deepseek-v4-flash-0731
- Solved: 2026-08-27T02:12:12.780Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "CONSOLIDATE-001 (duckbrain #484, worker k3@kimi-for-coding): new `duckbrain consolidate` CLI \u2014 scans namespaces/<ns>/<domain>/<YYYY-MM>/current.jsonl rows for a target UTC day across all namespaces, content-hash dedup (embedding_text+key collapse, first kept), per-ns row/unique/dup counts + capped previews, final digest block; dry-run default, write path triggered by --write-digest or DUCKBRAIN_API_KEY env \u2192 POST /api/memories?namespace=duckbrain with X-API-Key (201 verified + read-back). Pitfall: vitest test with --digest-content FAILED in cron env because DUCKBRAIN_API_KEY is set there \u2014 tests must save/delete env vars that trigger write paths (mirror DUCKBRAIN_NAMESPACES_PATH pattern). Judge tier2 also flagged pre-existing prettier drift in an unrelated test file \u2014 run npx prettier --check src/ before judging.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4-flash-0731", "problem_class": "typescript-cli-consolidation-digest", "provider": "openrouter", "solved_at": "2026-08-27T02:12:12.780Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog