sofhiaazzhr Claude Opus 5 commited on
Commit
5fed261
·
1 Parent(s): cb2fd31

[NOTICKET] docs(knowledge-parsing): say what decides a chunk boundary

Browse files

Harry read the contract and asked whether chunks are per page. They are not,
and the document never said so — the gap was real.

Records the actual priority: a numbered section heading is the primary
boundary; a table or chart is its own chunk; the 6000-character cap is a
safety net that only matters for documents with no detected headings.

With the measured spread, because the shape alone does not convey it: 1 to 7
chunks per page on the reference run, median text length 48 words. The cap
effectively never fires on a well-structured standard.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

KNOWLEDGE_PARSING_OUTPUT_CONTRACT.md CHANGED
@@ -141,6 +141,29 @@ Measured on the same document and gold set, changing only the parse:
141
 
142
  The markup is not discarded, because the formula branch needs exactly that form.
143
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
  ---
145
 
146
  ## 3 · Reference run — what 9 pages actually produced
 
141
 
142
  The markup is not discarded, because the formula branch needs exactly that form.
143
 
144
+ ### What decides a chunk boundary — **not the page**
145
+
146
+ A page break is a printing artifact, not a meaning boundary. Cutting there would sever a definition
147
+ that happens to straddle one, and the evidence ranker downstream reasons about sections. So chunking
148
+ follows the document's own structure, in this priority:
149
+
150
+ | | Opens a new chunk | |
151
+ |---|---|---|
152
+ | 1 | **A numbered section heading** | the primary boundary |
153
+ | 2 | **A table or a chart** | one MinerU item → one chunk |
154
+ | 3 | **6 000 characters (~1 500 tokens)** | safety net **only** for documents where no heading is detected; without it one chunk could swallow a whole document, blunting evidence ranking and inflating the summary branch's token share. Splits at an item boundary, so `text` stays verbatim |
155
+
156
+ Measured on the reference run — the spread is content-driven, not fixed:
157
+
158
+ | | |
159
+ |---|---|
160
+ | Chunks per page | **1 to 7** (page 0 → 1, page 5 → 7) |
161
+ | Chunks spanning >1 page | 0 of 31 here, though `page_idxs` is a list precisely because it can happen |
162
+ | `text` length | median **48 words**, max 444, one empty (the caption-less chart) |
163
+
164
+ Rule 3 therefore almost never fires on a well-structured standard — the headings do the work. It
165
+ matters for the mid-chapter textbook pages, which carry no headings at all.
166
+
167
  ---
168
 
169
  ## 3 · Reference run — what 9 pages actually produced