hygienic-incremental-macro-expansion

Answer

I built, debugged, and verified a complete solution. The full self-contained markdown is at ~/hygienic-incremental-macro-expander/SOLUTION.md (1032 lines). I verified it by extracting its embedded code blocks into a clean directory and re-running them: 12/12 checks pass.

What the diagnosis found

The failure class has five independent root causes; four are standard, one is the subtle killer:

1. Global-gensym renaming is neither hygienic (a macro temp can capture a user argument) nor stable (a global counter renumbers everything after an unrelated edit, invalidating caches).
2. Caching by spelling instead of resolved binding identity lets an unrelated edit capture a reused expansion.
3. Over-broad invalidation (re-expanding the whole module) defeats incrementality and perturbs generated scopes.
4. Lost source locations because templates are built without loc.
5. Lexical scopes applied after macro expansion — the bug that survives fixes 1–4. If you expand first and then assign lexical scopes, a macro-introduced free identifier receives the use-site scope, so subset resolution lets a local binder capture it: scheme (define-syntax use-helper (syntax-rules () ((_ x) (helper x)))) (let ((helper 42)) (use-helper 5)) ; naive post-pass: helper binds to the local!

The exact fix

A scope-set model (Flatt's Binding as Sets of Scopes) with strict ordering:

• Binding forms (lambda/let) add a fresh lexical scope to binder + body.
• Each macro expansion adds a fresh macro scope only to template-origin syntax; arguments are cloned untouched.
• A reference binds to the largest scope set that is a subset of the reference's scopes.
• Scopes come from a counter reset per top-level form, making expansion reproducible and cache-stable.
• Expansion and resolution run in one pass (process), so lexical scopes are applied to the raw use before the macro fires. This is what makes use-helper stay free.
• Cache validity = source hash + hashes of every macro/top-level value actually used; reverse dependency edges give the smallest sound recomputation region.
• Introduced nodes inherit the macro-use loc; arguments keep theirs.

Key excerpt (full file in SOLUTION.md):

function process(node, env, macroEnv, ctx) {
  if (node.type === 'symbol') { /* resolve by largest subset */ }
  const head = node.items[0];
  if (head && head.type === 'symbol') {
    if (macroEnv.has(head.name)) {
      const macro = macroEnv.get(head.name);
      for (const rule of macro.rules) {
        const binds = match(rule.pattern, node, macro.literals, {});
        if (binds) {
          ctx.usedMacros.add(macro.name);
          const scope = ctx.freshScope();
          // template gets `scope`; substituted args do not
          const inst = instantiate(rule.template, binds, scope, node.loc);
          return process(inst, env, macroEnv, ctx);   // reassign result!
        }
      }
    }
    if (head.name === 'let') {
      const lex = ctx.freshLex();
      // ...add lex to binders, then to raw body BEFORE recursing...
      for (let i = 2; i < node.items.length; i++)
        node.items[i] = process(node.items[i], env.concat(added), macroEnv, ctx);
    }
  }
}

Verification

cd ~/hygienic-incremental-macro-expander
node test.js

12 checks passed

Covers: hygiene/no-capture of a macro temp, definition-site free identifiers, nested-scope shadowing, mutually recursive macros (my-even?/my-odd?), smallest sound recomputation region, unrelated-edit reuse, cache-capture safety, determinism, source-location preservation, cross-expansion distinct temp bindings, and transitive B → A → C → user invalidation with independent forms reused.

Deliverables: SOLUTION.md (title, root-cause analysis, full code, verification), plus the runnable macro-expander.js and test.js.