Rifqi Hafizuddin Claude Fable 5 commited on
Commit
a7bdeae
·
1 Parent(s): f0cf168

[NOTICKET] fix(knowledge_extraction): show the model the section heading

Browse files

The evidence block sent to the model carried only `chunk.text`, so the section
heading never reached it. Two consequences, both silent:

- Indonesian standards name the term in the heading and open the body with the
definition — "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan
fisik…" — and the body often never repeats the term. The model was being
asked to define a term its evidence never named.
- It broke the invariant that what the model reads is exactly what the span
check searches. `validate.evidence_text` already composed heading + text, so
a field quoting a section title would have been rejected as unlocatable — and
the model could not have quoted one anyway.

Found while verifying the claim in the flow illustration: "Physical of
Availability" appeared in the prompt only because the prompt file's own worked
example contains that string, never from the document being processed.

The deterministic `source_wording` fallback masked the effect, which is why the
live pilot still produced the right answer.

New local test asserts the invariant directly: every line the model is shown
must be locatable by the span check.

Verification: ruff clean; import main OK; full suite 495 passed, 7 skipped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

src/knowledge_extraction/extract/base.py CHANGED
@@ -42,12 +42,26 @@ def cacheable(branch: str) -> bool:
42
 
43
  def evidence_block(chunks: list[Chunk], scores: list[float] | None = None) -> str:
44
  """Evidence labelled with chunk_id, section and page so the model can cite
45
- provenance and we can trace which evidence produced which field."""
 
 
 
 
 
 
 
 
 
 
 
 
 
46
  parts = []
47
  for i, chunk in enumerate(chunks):
48
  score = f" score={scores[i]:.1f}" if scores and i < len(scores) else ""
 
49
  parts.append(
50
  f"[chunk_id={chunk.chunk_id} section={chunk.section_no or '-'} "
51
- f"page={chunk.page_start}{score}]\n{chunk.text}"
52
  )
53
  return "EVIDENCE\n" + "\n\n---\n\n".join(parts)
 
42
 
43
  def evidence_block(chunks: list[Chunk], scores: list[float] | None = None) -> str:
44
  """Evidence labelled with chunk_id, section and page so the model can cite
45
+ provenance and we can trace which evidence produced which field.
46
+
47
+ **The heading is included, and must stay included.** Two reasons:
48
+
49
+ 1. Indonesian standards name the term in the heading and open the body with
50
+ the definition — "2.1.3 Physical of Availability (PA)" / "Adalah
51
+ ketersediaan fisik…" — so the body often never repeats the term. Without
52
+ the heading the model is asked to define a term the evidence never names.
53
+ 2. It keeps one invariant true: **what the model reads is exactly what the
54
+ span check searches.** `validate.evidence_text` composes heading + text;
55
+ if this block showed only the text, the model could never quote a
56
+ section title, and any field that did quote one would be rejected as
57
+ unlocatable.
58
+ """
59
  parts = []
60
  for i, chunk in enumerate(chunks):
61
  score = f" score={scores[i]:.1f}" if scores and i < len(scores) else ""
62
+ head = f"{chunk.heading}\n" if chunk.heading else ""
63
  parts.append(
64
  f"[chunk_id={chunk.chunk_id} section={chunk.section_no or '-'} "
65
+ f"page={chunk.page_start}{score}]\n{head}{chunk.text}"
66
  )
67
  return "EVIDENCE\n" + "\n\n---\n\n".join(parts)