Garm Claude Opus 4.7 (1M context) commited on
Commit
dfbf6cc
·
1 Parent(s): 46dc048

fix(learn): show prior patterns block to LLM to prevent dangling refs

Browse files

When `headroom learn` re-surfaced a section heading that already existed
in CLAUDE.md / MEMORY.md, the writer replaced that section wholesale —
but the LLM never saw the prior block, so it emitted condensed bullets
like "X is *also* large — same rule as Y, Z" assuming Y and Z would
remain siblings. After replacement, Y and Z were gone and the "also"
dangled.

This threads the project's current `<!-- headroom:learn -->` block (from
both CLAUDE.md and MEMORY.md) into the digest as a "Prior Learned
Patterns" section, and extends the system prompt to make the re-emission
contract explicit: re-stating a section replaces it wholesale, so the
LLM must copy forward prior bullets it still agrees with. Prior sections
the LLM omits entirely are still carried forward by the writer (#231
behavior preserved as a safety net).

Changes:
- New `extract_marker_block(file_content)` helper in `learn.writer` that
returns the raw marker block (delimiters included) or None.
- New `_build_prior_patterns_section(project)` in `learn.analyzer` reads
`project.context_file` and `project.memory_file` via the new helper
and formats a labeled section ahead of the per-session event stream.
- `_build_digest` emits the prior-patterns section when present; char
budget accounting unchanged (prior blocks are small).
- `_SYSTEM_PROMPT` gains a "Prior Learned Patterns" rule block telling
the LLM how to integrate prior bullets (preserve / revise / drop-only-
if-contradicted) and warning against unresolved cross-references.
- Tests: 6 new `TestPriorPatternsInjection` cases (present/absent files,
no-marker-block, both-files, end-to-end via mocked `_call_llm`); 4 new
`TestExtractMarkerBlock` cases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CHANGELOG.md CHANGED
@@ -14,6 +14,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
14
  new run win; sections not re-surfaced are carried forward so learnings
15
  accumulate across runs instead of disappearing. To fully rebuild the
16
  block, delete it manually and re-run. (#231)
 
 
 
 
 
 
 
 
 
 
 
17
 
18
  ### Added
19
  - **`turn_id` linking agent-loop API calls to a single user prompt** — a new
 
14
  new run win; sections not re-surfaced are carried forward so learnings
15
  accumulate across runs instead of disappearing. To fully rebuild the
16
  block, delete it manually and re-run. (#231)
17
+ - **`headroom learn` no longer emits dangling cross-references when a
18
+ section is re-surfaced** — the analyzer now includes the project's
19
+ current `<!-- headroom:learn -->` block (from `CLAUDE.md` and
20
+ `MEMORY.md`) in the LLM digest as a "Prior Learned Patterns" section,
21
+ and the system prompt instructs the LLM that re-emitting a section
22
+ replaces the prior one wholesale. Prevents bullets like "`X` is *also*
23
+ large — same rule as `Y`, `Z`" from appearing after `Y` and `Z` got
24
+ dropped during per-section replacement. The writer's section-level
25
+ carry-forward from #231 remains in place as a safety net for sections
26
+ the LLM omits entirely. New helper `extract_marker_block` added to
27
+ `headroom.learn.writer`.
28
 
29
  ### Added
30
  - **`turn_id` linking agent-loop API calls to a single user prompt** — a new
headroom/learn/analyzer.py CHANGED
@@ -29,6 +29,7 @@ from .models import (
29
  SessionEvent,
30
  ToolCall,
31
  )
 
32
 
33
  logger = logging.getLogger(__name__)
34
 
@@ -147,11 +148,51 @@ class SessionAnalyzer:
147
  # =============================================================================
148
 
149
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
150
  def _build_digest(project: ProjectInfo, sessions: list[SessionData]) -> str:
151
  """Build a token-efficient text digest of all session events.
152
 
153
  The digest includes:
154
  - Project context
 
155
  - Per-session summaries with condensed event streams
156
  - Error outputs (truncated), success indicators, user messages
157
  """
@@ -173,6 +214,12 @@ def _build_digest(project: ProjectInfo, sessions: list[SessionData]) -> str:
173
  lines.append(f"Tokens used: {total_tokens_in:,} in / {total_tokens_out:,} out")
174
  lines.append("")
175
 
 
 
 
 
 
 
176
  # Budget tracking — stop adding events when we approach the limit
177
  # Rough estimate: 4 chars per token
178
  char_budget = _MAX_DIGEST_TOKENS * 4
@@ -289,6 +336,24 @@ Rules:
289
  - Do NOT produce tautological rules (e.g., "use python3 not python3")
290
  - Do NOT produce rules about things that only happened once (transient errors)
291
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
292
  Return ONLY valid JSON matching this schema — no other text:
293
  {
294
  "context_file_rules": [
 
29
  SessionEvent,
30
  ToolCall,
31
  )
32
+ from .writer import extract_marker_block
33
 
34
  logger = logging.getLogger(__name__)
35
 
 
148
  # =============================================================================
149
 
150
 
151
+ def _build_prior_patterns_section(project: ProjectInfo) -> str:
152
+ """Format the current marker blocks from CLAUDE.md / MEMORY.md for the LLM.
153
+
154
+ Returns "" when neither file exists nor contains a marker block. When at
155
+ least one file has a block, returns a header + labeled raw blocks so the
156
+ LLM can treat them as the starting baseline. See the "Prior Learned
157
+ Patterns" rule in _SYSTEM_PROMPT for the contract with the model.
158
+ """
159
+ parts: list[tuple[str, str]] = [] # (label, block)
160
+ candidates = (
161
+ ("CLAUDE.md (CONTEXT_FILE, project-level stable facts)", project.context_file),
162
+ ("MEMORY.md (MEMORY_FILE, session-level evolving preferences)", project.memory_file),
163
+ )
164
+ for label, path in candidates:
165
+ if path is None or not path.exists():
166
+ continue
167
+ block = extract_marker_block(path.read_text())
168
+ if block:
169
+ parts.append((label, block))
170
+
171
+ if not parts:
172
+ return ""
173
+
174
+ lines = [
175
+ "=== Prior Learned Patterns ===",
176
+ (
177
+ f"These patterns are currently written to {project.name}'s context "
178
+ f"files. They are your starting baseline — see the 'Prior Learned "
179
+ f"Patterns' rule in the system prompt for how to integrate them."
180
+ ),
181
+ "",
182
+ ]
183
+ for label, block in parts:
184
+ lines.append(f"--- From {label} ---")
185
+ lines.append(block)
186
+ lines.append("")
187
+ return "\n".join(lines)
188
+
189
+
190
  def _build_digest(project: ProjectInfo, sessions: list[SessionData]) -> str:
191
  """Build a token-efficient text digest of all session events.
192
 
193
  The digest includes:
194
  - Project context
195
+ - Prior learned patterns (if any) from CLAUDE.md / MEMORY.md
196
  - Per-session summaries with condensed event streams
197
  - Error outputs (truncated), success indicators, user messages
198
  """
 
214
  lines.append(f"Tokens used: {total_tokens_in:,} in / {total_tokens_out:,} out")
215
  lines.append("")
216
 
217
+ # Prior learned patterns (if any) — gives the LLM the current baseline so
218
+ # it can produce complete updated sections instead of condensed deltas.
219
+ prior_section = _build_prior_patterns_section(project)
220
+ if prior_section:
221
+ lines.append(prior_section)
222
+
223
  # Budget tracking — stop adding events when we approach the limit
224
  # Rough estimate: 4 chars per token
225
  char_budget = _MAX_DIGEST_TOKENS * 4
 
336
  - Do NOT produce tautological rules (e.g., "use python3 not python3")
337
  - Do NOT produce rules about things that only happened once (transient errors)
338
 
339
+ Prior Learned Patterns:
340
+ - The input may contain a "Prior Learned Patterns" section showing what is
341
+ already written to the project's CLAUDE.md / MEMORY.md. Treat those as the
342
+ starting baseline for your analysis.
343
+ - When you re-emit a section heading that appears in the prior block, your
344
+ output REPLACES that prior section wholesale — so your section must be the
345
+ COMPLETE updated version:
346
+ * Preserve prior bullets that remain accurate (copy them forward)
347
+ * Revise bullets when new evidence refines them (merge, don't duplicate)
348
+ * Drop a prior bullet only when contradicted by clear new evidence
349
+ - Sections from prior runs that you do NOT re-emit are preserved automatically
350
+ by the writer, so focus only on sections where you have something to add or
351
+ change. Do NOT re-emit a prior section just to echo it verbatim — that wastes
352
+ output tokens without changing the outcome.
353
+ - Do NOT write bullets that reference prior siblings you are about to drop
354
+ (e.g., "X is ALSO large — same rule as Y, Z") unless Y and Z are also present
355
+ in your current output or preserved in the prior block.
356
+
357
  Return ONLY valid JSON matching this schema — no other text:
358
  {
359
  "context_file_rules": [
headroom/learn/writer.py CHANGED
@@ -91,6 +91,17 @@ def _build_section(recommendations: list[Recommendation]) -> str:
91
  _TOKENS_ANNOTATION_PATTERN = re.compile(r"\*~([\d,]+) tokens/session saved\*\n?")
92
 
93
 
 
 
 
 
 
 
 
 
 
 
 
94
  def _parse_prior_recommendations(existing: str) -> list[Recommendation]:
95
  """Parse recommendations out of a prior marker block.
96
 
 
91
  _TOKENS_ANNOTATION_PATTERN = re.compile(r"\*~([\d,]+) tokens/session saved\*\n?")
92
 
93
 
94
+ def extract_marker_block(file_content: str) -> str | None:
95
+ """Return the raw text of the headroom:learn marker block, or None.
96
+
97
+ Unlike _parse_prior_recommendations, this returns the block verbatim
98
+ (including the start/end markers) so it can be fed back to an LLM as
99
+ context without losing formatting. Returns None if no block is present.
100
+ """
101
+ match = _MARKER_PATTERN.search(file_content)
102
+ return match.group(0) if match else None
103
+
104
+
105
  def _parse_prior_recommendations(existing: str) -> list[Recommendation]:
106
  """Parse recommendations out of a prior marker block.
107
 
tests/test_learn/test_analyzer.py CHANGED
@@ -153,6 +153,117 @@ class TestDigestBuilder:
153
  assert "0 sessions" in digest or "test-project" in digest
154
 
155
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
156
  # =============================================================================
157
  # LLM Response Parser Tests
158
  # =============================================================================
 
153
  assert "0 sessions" in digest or "test-project" in digest
154
 
155
 
156
+ # =============================================================================
157
+ # Prior Patterns Injection Tests
158
+ # =============================================================================
159
+
160
+
161
+ _MARKER_BLOCK = (
162
+ "<!-- headroom:learn:start -->\n"
163
+ "## Headroom Learned Patterns\n"
164
+ "*Auto-generated by `headroom learn` on 2026-04-01 — do not edit manually*\n"
165
+ "\n"
166
+ "### Large Files\n"
167
+ "- `src/App.tsx` is very large (~40k tokens) — use offset/limit reads\n"
168
+ "- `src/lib.rs` frequently exceeds 10k tokens\n"
169
+ "\n"
170
+ "<!-- headroom:learn:end -->"
171
+ )
172
+
173
+
174
+ def _project_with_files(tmp_path: Path, claude_md_text: str | None, memory_md_text: str | None) -> ProjectInfo:
175
+ """Build a ProjectInfo pointing at temp CLAUDE.md / MEMORY.md files."""
176
+ proj_dir = tmp_path / "proj"
177
+ proj_dir.mkdir()
178
+ data_dir = tmp_path / "data"
179
+ (data_dir / "memory").mkdir(parents=True)
180
+
181
+ context_file: Path | None = None
182
+ if claude_md_text is not None:
183
+ context_file = proj_dir / "CLAUDE.md"
184
+ context_file.write_text(claude_md_text)
185
+
186
+ memory_file: Path | None = None
187
+ if memory_md_text is not None:
188
+ memory_file = data_dir / "memory" / "MEMORY.md"
189
+ memory_file.write_text(memory_md_text)
190
+
191
+ return ProjectInfo(
192
+ name="proj",
193
+ project_path=proj_dir,
194
+ data_path=data_dir,
195
+ context_file=context_file,
196
+ memory_file=memory_file,
197
+ )
198
+
199
+
200
+ class TestPriorPatternsInjection:
201
+ """The digest should include the prior marker block so the LLM can emit
202
+ COMPLETE updated sections instead of condensed deltas that reference
203
+ now-dropped siblings (the "X is also large — same rule as Y, Z" bug)."""
204
+
205
+ def test_digest_includes_prior_block_from_claude_md(self, tmp_path):
206
+ project = _project_with_files(tmp_path, claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n", memory_md_text=None)
207
+ digest = _build_digest(project, [])
208
+ assert "Prior Learned Patterns" in digest
209
+ assert "### Large Files" in digest
210
+ assert "App.tsx" in digest
211
+
212
+ def test_digest_includes_prior_block_from_memory_md(self, tmp_path):
213
+ project = _project_with_files(tmp_path, claude_md_text=None, memory_md_text=f"{_MARKER_BLOCK}\n")
214
+ digest = _build_digest(project, [])
215
+ assert "Prior Learned Patterns" in digest
216
+ assert "MEMORY.md" in digest
217
+ assert "### Large Files" in digest
218
+
219
+ def test_digest_omits_section_when_no_files_exist(self, tmp_path):
220
+ project = _project_with_files(tmp_path, claude_md_text=None, memory_md_text=None)
221
+ digest = _build_digest(project, [])
222
+ assert "Prior Learned Patterns" not in digest
223
+ assert "<!-- headroom:learn" not in digest
224
+
225
+ def test_digest_omits_section_when_file_has_no_marker_block(self, tmp_path):
226
+ """CLAUDE.md exists but has no headroom block → no prior section emitted."""
227
+ project = _project_with_files(
228
+ tmp_path,
229
+ claude_md_text="# Project\n\nJust a regular readme, no headroom block.\n",
230
+ memory_md_text=None,
231
+ )
232
+ digest = _build_digest(project, [])
233
+ assert "Prior Learned Patterns" not in digest
234
+
235
+ def test_digest_surfaces_both_files_when_both_present(self, tmp_path):
236
+ project = _project_with_files(
237
+ tmp_path,
238
+ claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n",
239
+ memory_md_text=f"{_MARKER_BLOCK}\n",
240
+ )
241
+ digest = _build_digest(project, [])
242
+ assert digest.count("### Large Files") >= 2 # once per file
243
+ assert "CLAUDE.md" in digest
244
+ assert "MEMORY.md" in digest
245
+
246
+ @patch("headroom.learn.analyzer._call_llm")
247
+ def test_analyze_passes_prior_block_through_to_llm(self, mock_call_llm: MagicMock, tmp_path):
248
+ """End-to-end: SessionAnalyzer.analyze() → _call_llm receives digest
249
+ containing the prior marker block content."""
250
+ mock_call_llm.return_value = {"context_file_rules": [], "memory_file_rules": []}
251
+ project = _project_with_files(tmp_path, claude_md_text=f"# Project\n\n{_MARKER_BLOCK}\n", memory_md_text=None)
252
+ sessions = [
253
+ SessionData(
254
+ session_id="s1",
255
+ tool_calls=[_tc(msg_index=0, is_error=True, output="error")],
256
+ )
257
+ ]
258
+
259
+ SessionAnalyzer(model="test-model").analyze(project, sessions)
260
+
261
+ mock_call_llm.assert_called_once()
262
+ digest_arg = mock_call_llm.call_args[0][0]
263
+ assert "Prior Learned Patterns" in digest_arg
264
+ assert "App.tsx" in digest_arg
265
+
266
+
267
  # =============================================================================
268
  # LLM Response Parser Tests
269
  # =============================================================================
tests/test_learn/test_writer.py CHANGED
@@ -8,6 +8,7 @@ from headroom.learn.writer import (
8
  _MARKER_START,
9
  ClaudeCodeWriter,
10
  _parse_prior_recommendations,
 
11
  )
12
 
13
 
@@ -239,3 +240,43 @@ class TestParsePriorRecommendations:
239
  assert len(recs) == 1
240
  assert recs[0].section == "Real Section"
241
  assert "real bullet" in recs[0].content
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  _MARKER_START,
9
  ClaudeCodeWriter,
10
  _parse_prior_recommendations,
11
+ extract_marker_block,
12
  )
13
 
14
 
 
240
  assert len(recs) == 1
241
  assert recs[0].section == "Real Section"
242
  assert "real bullet" in recs[0].content
243
+
244
+
245
+ class TestExtractMarkerBlock:
246
+ """Direct coverage for extract_marker_block."""
247
+
248
+ def test_returns_raw_block_when_present(self):
249
+ """Marker block is returned verbatim with delimiters, for LLM prompts."""
250
+ content = (
251
+ "# Project README\n\n"
252
+ "Some text.\n\n"
253
+ f"{_MARKER_START}\n"
254
+ "## Headroom Learned Patterns\n"
255
+ "### Environment\n"
256
+ "- Use uv run python\n"
257
+ f"{_MARKER_END}\n"
258
+ "Trailing text.\n"
259
+ )
260
+ block = extract_marker_block(content)
261
+ assert block is not None
262
+ assert block.startswith(_MARKER_START)
263
+ assert block.endswith(_MARKER_END)
264
+ assert "### Environment" in block
265
+ assert "Use uv run python" in block
266
+ assert "Trailing text." not in block
267
+
268
+ def test_returns_none_when_absent(self):
269
+ """File without any marker delimiters yields None."""
270
+ assert extract_marker_block("# Project\n\nJust a regular README.\n") is None
271
+
272
+ def test_returns_none_when_only_start_marker(self):
273
+ """Partial/malformed block (start only) yields None — writer expects both delimiters."""
274
+ content = f"prefix\n{_MARKER_START}\n### Something\n- content\n"
275
+ assert extract_marker_block(content) is None
276
+
277
+ def test_returns_empty_block_when_markers_are_adjacent(self):
278
+ """A block with nothing between the markers is still returned (caller's choice what to do)."""
279
+ content = f"prefix\n{_MARKER_START}\n{_MARKER_END}\nsuffix\n"
280
+ block = extract_marker_block(content)
281
+ assert block is not None
282
+ assert block == f"{_MARKER_START}\n{_MARKER_END}"