Commit ·
5fed261
1
Parent(s): cb2fd31
[NOTICKET] docs(knowledge-parsing): say what decides a chunk boundary
Browse filesHarry read the contract and asked whether chunks are per page. They are not,
and the document never said so — the gap was real.
Records the actual priority: a numbered section heading is the primary
boundary; a table or chart is its own chunk; the 6000-character cap is a
safety net that only matters for documents with no detected headings.
With the measured spread, because the shape alone does not convey it: 1 to 7
chunks per page on the reference run, median text length 48 words. The cap
effectively never fires on a well-structured standard.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md
CHANGED
|
@@ -141,6 +141,29 @@ Measured on the same document and gold set, changing only the parse:
|
|
| 141 |
|
| 142 |
The markup is not discarded, because the formula branch needs exactly that form.
|
| 143 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
---
|
| 145 |
|
| 146 |
## 3 · Reference run — what 9 pages actually produced
|
|
|
|
| 141 |
|
| 142 |
The markup is not discarded, because the formula branch needs exactly that form.
|
| 143 |
|
| 144 |
+
### What decides a chunk boundary — **not the page**
|
| 145 |
+
|
| 146 |
+
A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition
|
| 147 |
+
that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking
|
| 148 |
+
follows the document's own structure, in this priority:
|
| 149 |
+
|
| 150 |
+
| | Opens a new chunk | |
|
| 151 |
+
|---|---|---|
|
| 152 |
+
| 1 | **A numbered section heading** | the primary boundary |
|
| 153 |
+
| 2 | **A table or a chart** | one MinerU item → one chunk |
|
| 154 |
+
| 3 | **6 000 characters (~1 500 tokens)** | safety net **only** for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so `text` stays verbatim |
|
| 155 |
+
|
| 156 |
+
Measured on the reference run — the spread is content-driven, not fixed:
|
| 157 |
+
|
| 158 |
+
| | |
|
| 159 |
+
|---|---|
|
| 160 |
+
| Chunks per page | **1 to 7** (page 0 → 1, page 5 → 7) |
|
| 161 |
+
| Chunks spanning >1 page | 0 of 31 here, though `page_idxs` is a list precisely because it can happen |
|
| 162 |
+
| `text` length | median **48 words**, max 444, one empty (the caption-less chart) |
|
| 163 |
+
|
| 164 |
+
Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It
|
| 165 |
+
matters for the mid-chapter textbook pages, which carry no headings at all.
|
| 166 |
+
|
| 167 |
---
|
| 168 |
|
| 169 |
## 3 · Reference run — what 9 pages actually produced
|