[NOTICKET] fix(knowledge-parsing): keep the heading line out of Chunk.text
Browse filesThe heading was briefly copied into `text` as its first segment, to stop the
chunk that defines a term from containing no mention of that term — the BUMA
standard names the term in the heading and opens the body with "Adalah ..."
without repeating it.
The extraction half already solves this, and better, because it measured it on
the document: `validate/span_check.py` builds its haystack as heading + text,
and `rank/evidence.py` treats a heading that names the term as a mention at
offset 0.
So copying it into `text` only causes harm: the heading appears twice in that
haystack, and its occurrence becomes a real mention, inflating mention_count.
That count is what the frozen prototype baseline is compared against (169
mentions -> 66 clusters), so v2 numbers would drift for a reason that has
nothing to do with extraction quality.
`heading` stays verbatim, which is what both halves actually needed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@@ -55,15 +55,10 @@ class Chunk(BaseModel):
|
|
| 55 |
|
| 56 |
# Content. VERBATIM — never reflowed or normalised. See note above.
|
| 57 |
#
|
| 58 |
-
#
|
| 59 |
-
#
|
| 60 |
-
#
|
| 61 |
-
#
|
| 62 |
-
# the chunk that actually *defines* a term contains no mention of it, and
|
| 63 |
-
# ranks below calculation sections that merely use it repeatedly.
|
| 64 |
-
#
|
| 65 |
-
# It also makes this field the single place a span check has to look:
|
| 66 |
-
# `TermRecord.source_wording` is findable here, not in a second field.
|
| 67 |
text: str
|
| 68 |
|
| 69 |
# Location in the source document.
|
|
|
|
| 55 |
|
| 56 |
# Content. VERBATIM — never reflowed or normalised. See note above.
|
| 57 |
#
|
| 58 |
+
# Does NOT include the section's heading line; that lives in `heading`.
|
| 59 |
+
# Consumers that need both compose them — the extraction half already builds
|
| 60 |
+
# its span-check haystack as `heading + "\n" + text` and treats a heading
|
| 61 |
+
# that names the term as a mention at offset 0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
text: str
|
| 63 |
|
| 64 |
# Location in the source document.
|
|
@@ -205,19 +205,21 @@ def normalisasi(items: list[dict[str, Any]], doc_id: str) -> list[Chunk]:
|
|
| 205 |
berjalan = buka("text", page, section_no=nomor, heading=judul)
|
| 206 |
berjalan.source_items = [i]
|
| 207 |
berjalan.bbox = item.get("bbox")
|
| 208 |
-
# Baris judul
|
|
|
|
| 209 |
#
|
| 210 |
-
#
|
| 211 |
-
#
|
| 212 |
-
#
|
| 213 |
-
#
|
| 214 |
-
#
|
| 215 |
-
#
|
| 216 |
#
|
| 217 |
-
#
|
| 218 |
-
#
|
| 219 |
-
#
|
| 220 |
-
|
|
|
|
| 221 |
continue
|
| 222 |
|
| 223 |
if berjalan is None:
|
|
|
|
| 205 |
berjalan = buka("text", page, section_no=nomor, heading=judul)
|
| 206 |
berjalan.source_items = [i]
|
| 207 |
berjalan.bbox = item.get("bbox")
|
| 208 |
+
# Baris judul TIDAK dimasukkan ke `text` — cukup di field `heading`,
|
| 209 |
+
# verbatim.
|
| 210 |
#
|
| 211 |
+
# Sempat dicoba sebaliknya, karena di standar BUMA judul menyebut
|
| 212 |
+
# istilahnya lalu badan section mulai "Adalah ..." tanpa mengulangnya,
|
| 213 |
+
# sehingga chunk yang mendefinisikan istilah tidak memuat istilah itu.
|
| 214 |
+
# Tapi sisi ekstraksi sudah menangani ini: haystack span-check-nya
|
| 215 |
+
# disusun `heading + text`, dan ranker-nya memperlakukan judul sebagai
|
| 216 |
+
# mention di offset 0.
|
| 217 |
#
|
| 218 |
+
# Menyalinnya ke `text` justru merugikan: judul terhitung dua kali di
|
| 219 |
+
# haystack, dan kemunculannya jadi mention nyata sehingga
|
| 220 |
+
# `mention_count` menggelembung — padahal angka itu yang dibandingkan
|
| 221 |
+
# ke baseline beku (169 mention -> 66 cluster). Perbandingan v2 akan
|
| 222 |
+
# bergeser karena sebab yang tidak ada hubungannya dengan mutu.
|
| 223 |
continue
|
| 224 |
|
| 225 |
if berjalan is None:
|