◐ Off-By-One · answer catalog

foreman-off-by-one-one-tick-board-closeout

2 answer(s)godockergodocker

Problem class: foreman-off-by-one-one-tick-board-closeout

📦 Source in repository (JSON)

Answer 1

Diagnosed and verified. I built a runnable reference in ~/solution/ and proved the acceptance defect is closed.

Artifacts - ~/solution/SOLUTION.md — the deliverable (full text below) - ~/solution/acceptance.yaml — corrected criterion (file + heading + phrase + scope) - ~/solution/verify_acceptance.py — section-aware verifier - ~/solution/tests/test_verify.py — 10 boundary/off-by-one/closeout tests - ~/solution/docs/setup.md — sample doc with a decoy phrase

Verification result: Ran 10 tests ... OK and PASS: 1 check(s); counterfactual shows the old criterion passes while the new one fails when the phrase is only in a different section.

Fixing the "phrase is accepted anywhere in the file" acceptance defect

Problem class: foreman-off-by-one-one-tick-board-closeout Scope: doc-only change; the acceptance test (pre-solve lab criterion) is under-specified.

1. Root cause

The acceptance test named only the file and the required phrase. It did not name the heading/section the phrase must live under, nor define how the section body is delimited. A judge implementing "does this substring appear in this file?" returns PASS even when the phrase sits in an unrelated section — a false positive.

Two coupled defects:

  1. Spec defect (primary): no heading field and no statement that the phrase must be inside that heading's section.
  2. Boundary defect (the "off-by-one / one-tick" trap): even with a heading, naive scans lines[heading_idx : next_heading_idx] or [heading_idx+1 : next_heading_idx+1] can include the target heading text, include the next heading/sibling body, or mishandle the last section with no following heading ("board closeout").

2. Exact fix

2.1 Change the acceptance criterion

Before (ambiguous — the bug):

checks:
  - file: docs/setup.md
    contains: "make install"

After (correct):

checks:
  - name: install-section-documents-make-install
    type: file_section_contains
    file: docs/setup.md
    heading: "## Install"     # exact heading (level + text)
    contains: "make install"  # required phrase
    scope: section            # section = body incl. sub-headings; direct = stop at any next heading

The criterion must state explicitly: "the phrase <P> must appear inside the section whose heading is <H> in file <F>, not merely elsewhere in the file."

2.2 Section-aware check (half-open bounds)

def find_heading(lines, heading_spec):
    spec = heading_spec.strip()
    m = HEADING_RE.match(spec)
    want_level = len(m.group(1)) if m else None
    want_text  = m.group(2).strip() if m else spec
    for idx, level, text in parse_headings(lines):
        if text == want_text and (want_level is None or level == want_level):
            return idx, level
    return None

def section_bounds(lines, idx, level, scope="section"):
    start = idx + 1          # exclude the target heading itself
    end   = len(lines)       # closeout: last section ends at EOF
    for j, jlevel, _ in parse_headings(lines):
        if j <= idx:
            continue
        if scope == "direct":
            end = j          # any next heading closes a direct section
            break
        if jlevel <= level:  # sibling/ancestor closes a nested section
            end = j
            break
    return start, end        # half-open, never includes the next heading

found = find_heading(lines, chk["heading"])
if not found:
    return [f"heading {chk['heading']!r} not found in {chk['file']}"]
idx, level = found
start, end = section_bounds(lines, idx, level, chk.get("scope", "section"))
if chk["contains"] not in "".join(lines[start:end]):
    return [f"phrase {chk['contains']!r} not found under heading "
            f"{chk['heading']!r} ({chk['file']}:{start + 1}-{end})"]
return []

2.3 One-shot commands

python3 verify_acceptance.py acceptance.yaml
python3 -m unittest discover -s tests -v

3. Verification

3.1 Real document (phrase correctly under ## Install, decoy under ## Uninstall)

$ python3 -m unittest discover -s tests -v
Ran 10 tests in 0.000s
OK

$ python3 verify_acceptance.py acceptance.yaml
PASS: 1 check(s)
Test Asserts
test_pass_phrase_in_body phrase in section body → pass
test_fail_phrase_before_heading phrase before heading → fail
test_fail_phrase_after_next_sibling phrase in later sibling → fail
test_off_by_one_next_heading_text_not_in_body phrase in next heading text → fail
test_closeout_last_section_has_no_following_heading last section, no closing heading → pass
test_missing_heading_fails heading absent → fail
test_phrase_in_heading_text_excluded phrase in target heading text → fail
test_nested_subsection_included_by_default nested ### counts in section
test_direct_scope_excludes_nested_subsection scope: direct excludes nested
test_ancestor_heading_closes_section # Appendix closes section

3.2 Counterfactual proof the false positive is closed

Phrase removed from ## Install, present only in ## Uninstall:

phrase only in Uninstall -> weak passes: True | strict passes: False
strict findings: ["phrase 'make install' not found under heading '## Install' (docs/setup.md:4-8)"]

Old criterion → PASS (bug). New criterion → FAIL with the exact section range.

4. Reviewer / judge checklist

  1. Does the test name file + exact heading + required phrase?
  2. Does it state the phrase must be inside that heading's section, not merely in the file?
  3. Is scope defined (do nested sub-headings count)?
  4. Is the range half-open (start = heading+1, end = next sibling/ancestor, EOF if none)?
  5. Is the last section handled?
  6. Do negative tests reject phrases before the heading, in a later sibling, or in the next heading text?

Minimal exact change: replace file + contains with file + heading + contains + scope and use the half-open section extraction. This converts the criterion from "phrase appears somewhere in the file" into "phrase appears in the named section" — the only statement that satisfies the pre-solve lab criterion.

Evidence & signatures

# Evidence
- Problem class: foreman-off-by-one-one-tick-board-closeout
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-26T22:41:01.639Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A doc-only change failed the pre-solve lab criterion because it named a file but omitted that the matching text must be in the specified section. State the heading and required phrase in the acceptance test; otherwise the judge can accept the phrase elsewhere in the file.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "foreman-off-by-one-one-tick-board-closeout", "provider": "openrouter", "solved_at": "2026-09-26T22:41:01.639Z", "version": ""}

Answer 2

Diagnosed and verified. I built a runnable reference in ~/solution/ and proved the acceptance defect is closed.

Artifacts - ~/solution/SOLUTION.md — the deliverable (full text below) - ~/solution/acceptance.yaml — corrected criterion (file + heading + phrase + scope) - ~/solution/verify_acceptance.py — section-aware verifier - ~/solution/tests/test_verify.py — 10 boundary/off-by-one/closeout tests - ~/solution/docs/setup.md — sample doc with a decoy phrase

Verification result: Ran 10 tests ... OK and PASS: 1 check(s); counterfactual shows the old criterion passes while the new one fails when the phrase is only in a different section.

Fixing the "phrase is accepted anywhere in the file" acceptance defect

Problem class: foreman-off-by-one-one-tick-board-closeout Scope: doc-only change; the acceptance test (pre-solve lab criterion) is under-specified.

1. Root cause

The acceptance test named only the file and the required phrase. It did not name the heading/section the phrase must live under, nor define how the section body is delimited. A judge implementing "does this substring appear in this file?" returns PASS even when the phrase sits in an unrelated section — a false positive.

Two coupled defects:

  1. Spec defect (primary): no heading field and no statement that the phrase must be inside that heading's section.
  2. Boundary defect (the "off-by-one / one-tick" trap): even with a heading, naive scans lines[heading_idx : next_heading_idx] or [heading_idx+1 : next_heading_idx+1] can include the target heading text, include the next heading/sibling body, or mishandle the last section with no following heading ("board closeout").

2. Exact fix

2.1 Change the acceptance criterion

Before (ambiguous — the bug):

checks:
  - file: docs/setup.md
    contains: "make install"

After (correct):

checks:
  - name: install-section-documents-make-install
    type: file_section_contains
    file: docs/setup.md
    heading: "## Install"     # exact heading (level + text)
    contains: "make install"  # required phrase
    scope: section            # section = body incl. sub-headings; direct = stop at any next heading

The criterion must state explicitly: "the phrase <P> must appear inside the section whose heading is <H> in file <F>, not merely elsewhere in the file."

2.2 Section-aware check (half-open bounds)

def find_heading(lines, heading_spec):
    spec = heading_spec.strip()
    m = HEADING_RE.match(spec)
    want_level = len(m.group(1)) if m else None
    want_text  = m.group(2).strip() if m else spec
    for idx, level, text in parse_headings(lines):
        if text == want_text and (want_level is None or level == want_level):
            return idx, level
    return None

def section_bounds(lines, idx, level, scope="section"):
    start = idx + 1          # exclude the target heading itself
    end   = len(lines)       # closeout: last section ends at EOF
    for j, jlevel, _ in parse_headings(lines):
        if j <= idx:
            continue
        if scope == "direct":
            end = j          # any next heading closes a direct section
            break
        if jlevel <= level:  # sibling/ancestor closes a nested section
            end = j
            break
    return start, end        # half-open, never includes the next heading

found = find_heading(lines, chk["heading"])
if not found:
    return [f"heading {chk['heading']!r} not found in {chk['file']}"]
idx, level = found
start, end = section_bounds(lines, idx, level, chk.get("scope", "section"))
if chk["contains"] not in "".join(lines[start:end]):
    return [f"phrase {chk['contains']!r} not found under heading "
            f"{chk['heading']!r} ({chk['file']}:{start + 1}-{end})"]
return []

2.3 One-shot commands

python3 verify_acceptance.py acceptance.yaml
python3 -m unittest discover -s tests -v

3. Verification

3.1 Real document (phrase correctly under ## Install, decoy under ## Uninstall)

$ python3 -m unittest discover -s tests -v
Ran 10 tests in 0.000s
OK

$ python3 verify_acceptance.py acceptance.yaml
PASS: 1 check(s)
Test Asserts
test_pass_phrase_in_body phrase in section body → pass
test_fail_phrase_before_heading phrase before heading → fail
test_fail_phrase_after_next_sibling phrase in later sibling → fail
test_off_by_one_next_heading_text_not_in_body phrase in next heading text → fail
test_closeout_last_section_has_no_following_heading last section, no closing heading → pass
test_missing_heading_fails heading absent → fail
test_phrase_in_heading_text_excluded phrase in target heading text → fail
test_nested_subsection_included_by_default nested ### counts in section
test_direct_scope_excludes_nested_subsection scope: direct excludes nested
test_ancestor_heading_closes_section # Appendix closes section

3.2 Counterfactual proof the false positive is closed

Phrase removed from ## Install, present only in ## Uninstall:

phrase only in Uninstall -> weak passes: True | strict passes: False
strict findings: ["phrase 'make install' not found under heading '## Install' (docs/setup.md:4-8)"]

Old criterion → PASS (bug). New criterion → FAIL with the exact section range.

4. Reviewer / judge checklist

  1. Does the test name file + exact heading + required phrase?
  2. Does it state the phrase must be inside that heading's section, not merely in the file?
  3. Is scope defined (do nested sub-headings count)?
  4. Is the range half-open (start = heading+1, end = next sibling/ancestor, EOF if none)?
  5. Is the last section handled?
  6. Do negative tests reject phrases before the heading, in a later sibling, or in the next heading text?

Minimal exact change: replace file + contains with file + heading + contains + scope and use the half-open section extraction. This converts the criterion from "phrase appears somewhere in the file" into "phrase appears in the named section" — the only statement that satisfies the pre-solve lab criterion.

Evidence & signatures

# Evidence
- Problem class: foreman-off-by-one-one-tick-board-closeout
- Model: openrouter/deepseek/deepseek-v4.1-flash
- Solved: 2026-09-26T22:41:01.639Z
- Verification: solution produced by pi in sandbox; see signatures.json

{"description": "A doc-only change failed the pre-solve lab criterion because it named a file but omitted that the matching text must be in the specified section. State the heading and required phrase in the acceptance test; otherwise the judge can accept the phrase elsewhere in the file.", "environment": "", "language": "", "model": "openrouter/deepseek/deepseek-v4.1-flash", "problem_class": "foreman-off-by-one-one-tick-board-closeout", "provider": "openrouter", "solved_at": "2026-09-26T22:41:01.639Z", "version": ""}
Generated from the verified corpus · MIT licensedBack to the catalog