[NOTICKET] fix(knowledge_extraction): show the model the section heading
Browse filesThe evidence block sent to the model carried only `chunk.text`, so the section
heading never reached it. Two consequences, both silent:
- Indonesian standards name the term in the heading and open the body with the
definition — "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan
fisik…" — and the body often never repeats the term. The model was being
asked to define a term its evidence never named.
- It broke the invariant that what the model reads is exactly what the span
check searches. `validate.evidence_text` already composed heading + text, so
a field quoting a section title would have been rejected as unlocatable — and
the model could not have quoted one anyway.
Found while verifying the claim in the flow illustration: "Physical of
Availability" appeared in the prompt only because the prompt file's own worked
example contains that string, never from the document being processed.
The deterministic `source_wording` fallback masked the effect, which is why the
live pilot still produced the right answer.
New local test asserts the invariant directly: every line the model is shown
must be locatable by the span check.
Verification: ruff clean; import main OK; full suite 495 passed, 7 skipped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@@ -42,12 +42,26 @@ def cacheable(branch: str) -> bool:
|
|
| 42 |
|
| 43 |
def evidence_block(chunks: list[Chunk], scores: list[float] | None = None) -> str:
|
| 44 |
"""Evidence labelled with chunk_id, section and page so the model can cite
|
| 45 |
-
provenance and we can trace which evidence produced which field.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
parts = []
|
| 47 |
for i, chunk in enumerate(chunks):
|
| 48 |
score = f" score={scores[i]:.1f}" if scores and i < len(scores) else ""
|
|
|
|
| 49 |
parts.append(
|
| 50 |
f"[chunk_id={chunk.chunk_id} section={chunk.section_no or '-'} "
|
| 51 |
-
f"page={chunk.page_start}{score}]\n{chunk.text}"
|
| 52 |
)
|
| 53 |
return "EVIDENCE\n" + "\n\n---\n\n".join(parts)
|
|
|
|
| 42 |
|
| 43 |
def evidence_block(chunks: list[Chunk], scores: list[float] | None = None) -> str:
|
| 44 |
"""Evidence labelled with chunk_id, section and page so the model can cite
|
| 45 |
+
provenance and we can trace which evidence produced which field.
|
| 46 |
+
|
| 47 |
+
**The heading is included, and must stay included.** Two reasons:
|
| 48 |
+
|
| 49 |
+
1. Indonesian standards name the term in the heading and open the body with
|
| 50 |
+
the definition — "2.1.3 Physical of Availability (PA)" / "Adalah
|
| 51 |
+
ketersediaan fisik…" — so the body often never repeats the term. Without
|
| 52 |
+
the heading the model is asked to define a term the evidence never names.
|
| 53 |
+
2. It keeps one invariant true: **what the model reads is exactly what the
|
| 54 |
+
span check searches.** `validate.evidence_text` composes heading + text;
|
| 55 |
+
if this block showed only the text, the model could never quote a
|
| 56 |
+
section title, and any field that did quote one would be rejected as
|
| 57 |
+
unlocatable.
|
| 58 |
+
"""
|
| 59 |
parts = []
|
| 60 |
for i, chunk in enumerate(chunks):
|
| 61 |
score = f" score={scores[i]:.1f}" if scores and i < len(scores) else ""
|
| 62 |
+
head = f"{chunk.heading}\n" if chunk.heading else ""
|
| 63 |
parts.append(
|
| 64 |
f"[chunk_id={chunk.chunk_id} section={chunk.section_no or '-'} "
|
| 65 |
+
f"page={chunk.page_start}{score}]\n{head}{chunk.text}"
|
| 66 |
)
|
| 67 |
return "EVIDENCE\n" + "\n\n---\n\n".join(parts)
|