Commit History
-
[NOTICKET] feat(knowledge-parsing): add 1-based page_no (S6b)
A computed `page_no` (= page_idx + 1) on Chunk, so the human/review-UI page number lives in the artifact instead of each consumer converting it. - Derived, never stored: cannot drift from page_idx, and neither normaliser has to set it. page_idx stays 0-based and raw. - Additive, schema 0.3.2 -> 0.3.3; older artifacts and consumers are unaffected. - Closes S6b (open since 2026-08-19); reverses the earlier "convert in the UI" note now that a real reviewer is about to read these. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>717452f -
[NOTICKET] feat(knowledge-parsing): wire Mistral OCR backend end-to-end
Makes Mistral OCR the primary parser (MinerU kept as a fallback), runnable as `python -m src.knowledge_parsing.run --backend mistral` with no GPU. - config: backend="mistral" default + Mistral settings (include_blocks, bbox_annotation, table_format). fingerprint() adds the Mistral settings ONLY on that backend, so existing MinerU cache keys stay byte-identical. - mistral_ocr.py (new): the API call — upload -> ocr.process, key from env/.env, file_id with a signed-url fallback, optional per-image descriptions. - parse.py: _parse_mistral caches the response like the MinerU path; cache_key uses the model id as the parser version for Mistral. - run.py: dispatches to normalise_mistral for the Mistral backend (skips the MinerU-only content-drift check and word-boundary vocabulary), stamps parser_name/normaliser_version per backend. - pyproject: mistralai in the knowledge-parsing extra. NOTE: it needs pydantic>=2.11.2, which the pinned pydantic==2.10.3 does not satisfy; the repo…6856ea8 -
[NOTICKET] feat(knowledge-parsing): Mistral OCR block normalizer (section default)
Mistral OCR (include_blocks=True) block stream -> per-section Chunk[], the counterpart of normalize.py for the Mistral backend. - Reuses normalize._heading and render.py so section boundaries and prose rendering match the MinerU path; artifact shape is identical (seam unchanged, extraction never learns which parser ran). - Strips markdown markup Mistral adds (not present in the source PDF) so the extraction span-check still matches quotes literally; whitespace kept verbatim. - granularity="section" (default) populates heading_path/heading/section_no, which extraction needs for outline + source_wording; "page" is recall-only. - Validated on the BUMA fixture through extraction's own adapter + span filter: section recall 0.878, page 0.951 (both above MinerU section 0.829). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>81b4f4e -
[NOTICKET] docs(knowledge): v3 delta in the workstream docs
Rifqi Hafizuddin +1: Claude Opus 5Completes the doc sync for `172d8dc`, alongside `6a9fb78`. - TODO: new X26 for the v3 reshape; X24 annotated as superseded for the domain branch. Numbered X26 rather than X25 because X25 ("baselines expire") already exists - task ids are cited from other docs and must not be reused. - CALIBRATION: the guarded-field table read `summary | (none - generation cannot be span-checked at all)`. That was a principle, not just a fact, and a future change would reasonably have built on it. It now reads `title, purpose_verbatim`, with the reason the exception disappeared: the branch was asked to LOCATE rather than SUMMARISE, so there is something to check against. - CONTEXT: the review-stage mermaid node "Brief Context" -> "Domain Context", and the per-branch model-tier principle amended - Summary was classed with Rule as reasoning/generation needing a larger tier "because a plausible summary cannot be span-checked". That premise no longer holds. This closes the X26 referenc…58e2002 -
[NOTICKET] docs(knowledge): record the DomainContext v3 reshape
Rifqi Hafizuddin +1: Claude Opus 5Brings the two canonical docs up to `172d8dc`. REPO_STATUS section 14 gains a dated delta for the v3 reshape; DEV_PLAN section 0.8 gains K12. Both now state that the domain branch is no longer the pipeline's one unverifiable output, and WHY that was achievable without a bigger model: the branch was asked a different question. "Summarise" cannot be located in the source by construction; "locate the document's own statement of purpose" can, so `GUARDED_FIELDS["summary"]` could move from `()` to `("title", "purpose_verbatim")`. Also records the review-queue contract change Harry needs — rows now carry `row_kind`, and a `key_parameter` row has no term_id, definition or mention count. One correction made in passing: REPO_STATUS described the extraction stages as `... validate -> diff -> queue`, omitting `link`, which has run between them since the v2 schema landed on 2026-08-26. Docs only; no code in this commit. The knowledge checklist (X26), the calibration guarded-field table and the…6a9fb78 -
[NOTICKET] feat(knowledge-extraction): subdomains + counts — v3 complete
Rifqi Hafizuddin +1: Claude Opus 5The last two DomainContext v3 fields, both deterministic. No LLM, no prompt change, no spend. subdomains Rolled up from the glossary entries' own `subdomain_tags`, never asked of the model. Each tag was already chosen once, per term, with that term's evidence in front of the model; a second document-level classification would be the same judgement made with less context and no way to check it. Ordered by tag frequency descending with an ALPHABETICAL TIEBREAK, which is load-bearing rather than tidy: on the reference document `production` and `equipment` both score 5, and the entries list them in differing order, so without the tiebreak two runs of one document would disagree. n_terms / n_formulas / n_rules Counts of the other three artifacts. Cheap orientation for a reviewer deciding whether a queue is worth opening, and the fastest way to spot a run that silently extracted nothing. Both are computed in `_finalise_domain`, called at the END of extract_all rathe…172d8dc -
[NOTICKET] feat(knowledge-extraction): span-guard the domain branch
Rifqi Hafizuddin +1: Claude Opus 5The domain branch was the pipeline's ONE unverifiable output: GUARDED_FIELDS ["summary"] was `()`, so `title` and `purpose` were unchecked model prose on nano, the tier whose failure mode is least detectable. Every other branch had to prove its claims against the source; this one did not. The fix is not a bigger model. It is asking a different question: v2 "write a short orientation brief" -> composed -> unprovable v3 "locate the document's own statement" -> quoted -> checkable A summary cannot be found in the source by construction. A quote can. So `purpose` becomes `purpose_verbatim`, prompts/summary.txt is rewritten from summarise to locate, and both text fields join GUARDED_FIELDS + _TRANSCRIBED. Also wires validation into `_run_summary`, which never called validate_entry - the guard table alone would have been inert. The haystack is the WHOLE document, unlike the other branches: this one is handed every chunk, so a quote may legitimately come from anywhere in it. …e2865ac -
[NOTICKET] feat(knowledge-extraction): derived outline + corroborated key parameters
Rifqi Hafizuddin +1: Claude Opus 5The two DETERMINISTIC halves of the DomainContext v3 reshape. No LLM, no spend, no prompt change - which is exactly why they ship ahead of the rest: the parts that need a prompt rewrite and an eval run are split off and still proposed. outline Derived from the artifact's heading_path, in reading order, de-duplicated. The blocker was upstream: adapter.py was dropping heading_path on the floor, so the internal Chunk never carried it - the same way it once dropped `latex`, and with the same consequence, a branch inventing prose about a document whose own structure was sitting unread. Fixed at the seam. It is the only field on this branch carrying the strong guarantee for free: verbatim source structure, nothing to hallucinate. On the BUMA fixture it yields 11 entries from 29 of 31 chunks, including the verbatim "Physical of Availability (PA)" the pipeline is required to surface rather than tidy. key_parameters Was list[str]; now list[KeyParameter{surface, term_id}],…6f5d619
-
[NOTICKET] refactor(knowledge-extraction): BriefContext -> DomainContext
Rifqi Hafizuddin +1: Claude Opus 5The workstream diagram has always called this box "Domain Knowledge", while the model and its fields described a short orientation card. Rename the entity to match the intent before anything is persisted: once an expert has approved rows keyed on these ids, changing the `b_` prefix is an id migration that loses the approvals - the exact failure the deterministic-id scheme exists to prevent. BriefContext -> DomainContext brief_id -> domain_id (prefix b_ -> d_) ExtractionResult.brief -> .domain brief_context.json -> domain_context.json Deliberately unchanged: - `Branch = "summary"` and GUARDED_FIELDS["summary"]. The branch literal is the measurement key in CallUsage.branch / RejectedField.branch; renaming it would break comparability with the committed eval baseline. - prompts/summary.txt. Editing prompt text needs the eval path, and the rewrite belongs with the v3 schema change, not with a rename. No field, prompt or guarding change - the …7f47c61
-
[NOTICKET] feat(knowledge-parsing): carry the document's human-readable title
`doc_id` is on its way to becoming an identifier rather than a label. Every `term_id` / `formula_id` / `rule_id` / `brief_id` downstream is a digest of it, so it must stop depending on what someone typed into a Save dialog — which it does today, being the file stem. A long name makes the cost visible: chunk_id: "STD_2026_006_MNO_ Rev.0.0 - Production Parameter and ECA::0000" That name is embedded in all 31 chunks and then in every entry derived from them. Renaming `Rev.0.0` to `Rev 0.0` re-keys the lot. `source_title` is the field the name moves to. It equals `doc_id` today, and that is the point: the two are now separate, so the identifier can change without the human-readable name going with it. `Provenance` is what a reviewing expert actually reads, and it carries `doc_id` — without this field they would be shown `f10807178b20531e` instead of the standard's name. Added ahead of the switch because it is needed whichever identifier wins (content hash or the Go service's upload…df571be
-
[NOTICKET] feat(knowledge-parsing): commit the reference artifact, and stop the stamp churning
Three changes, all aimed at one problem: the two halves of the pipeline could not reliably agree on which artifact they were measuring. **Metadata now precedes `chunks` in the artifact.** Pydantic serialises in declaration order, so it decides what a person sees on opening the file. With `chunks` first, checking which parser or normaliser produced an artifact meant scrolling past thousands of lines to the end — and that is a routine check now, not a rare one. JSON objects are unordered by specification and every consumer reads by key, so this changes nothing for code and `schema_version` is unchanged: no field, type or meaning moved. **`contracts.py` is out of the normaliser hash.** The artifact's SHAPE is already versioned by `schema_version`, so hashing it again counted the same change twice. The reordering above proved the cost: the stamp moved while the chunks came out byte-identical, which is a false "your artifact is stale". A signal that cries wolf gets ignored, and this one e…a0e59e4 -
[NOTICKET] eval(knowledge): commit the BUMA fixture artifact; K9/K10 close
Rifqi Hafizuddin +1: Claude Opus 5The 2026-08-27T13:28 re-parse of STD_2026_006_MNO becomes the committed fixture (S7/K10). Same document and content_hash as the 08-26 parse, same parser_config, five corrected chunk boundaries — of which ::0030 is the one that matters: it previously held only "Keterangan grafik:" and now carries the hierarchy-level paragraph, moving Controllable / Hierarki Level Analysis from E1 miss to hit. One paid run on it (90 calls, 170 s): E1 0.8293 / E2 2.556x / E3 1.000, all three above their kill lines, rule recall 0.4286 (6/14). 0 fields rejected by the span check, rules_without_term_link 0. Two caveats recorded beside the numbers rather than left for a reader to rediscover: E3's precision 1.000 sits on a scoreable base of 6 (57 of 63 entries abstain, recall 0.33 against the 18 gold terms with criteria), and the rule-recall delta is partly a denominator change — the same 6 rules matched as in the 08:29 run, the gold base went 15 to 14 when X23 closed. The harness now runs with data/knowled…e5f4abe -
[NOTICKET] feat(knowledge-parsing): stamp the artifact with the code that built it
A chunks.json is a frozen snapshot of the normalisation code at the moment it ran. When that code is fixed, existing artifacts do not update themselves — and nothing in their metadata said so. `parser_config` fingerprints MinerU's settings; `schema_version` versions the artifact's SHAPE, which a normalisation bugfix does not change. So a stale artifact and a correct one declared identical metadata, and the only observable difference was the character count. That cost a day. A pre-fix artifact reached the extraction half, term recall came back 0.7073 against a 0.8049 baseline, and it read as a regression in extraction's own work. It was neither side's setup: same filter code, different input. Verified here — scoring that artifact reproduces 169 mentions and 0.7073 exactly, against 186 and 0.8293 for the rebuilt one. `normaliser_version` is a hash of the modules that decide what a chunk CONTAINS: normalize.py, render.py, checks.py, contracts.py. A source hash rather than a hand-mainta…c64a0cb -
[NOTICKET] docs(knowledge-parsing): naming convention, and why it is a stopgap
The file name is currently a primary key, not a label: `doc_id` is the file stem, `chunk_id` is built from it, and `term_id` / `formula_id` / `rule_id` / `brief_id` are all digests of `doc_id` in `knowledge_extraction/ids.py`. Renaming a file after ingestion therefore re-keys every identifier an expert's approve/reject decisions hang on. That was worth writing down, because nothing in the code says it. The convention itself is ordinary — stable doc code, machine-readable revision suffix, no spaces, never `(1)`/`(2)`, never rename after ingestion. `(2)` gets its own line because it means "downloaded twice", not "version 2", and ingested as-is it becomes a second legitimate document with a full set of identifiers. What matters more is the section saying it is a STOPGAP. It holds only because we control the file names while parsing is an admin-triggered batch. Behind the upload endpoint that is planned after eval and the playground it stops holding: people will upload `Copy of SOP final…942f14e -
[NOTICKET] fix(knowledge-parsing): long names, name clashes, and a GPU check in the wrong place
Four defects, all found by asking what happens when a real person uploads a real file rather than the two well-named documents in the test folder. **Long file names failed the parse.** MinerU names its output folder after `pdf_file_names` and then writes images into it under a full SHA-256 filename. Measured on this repo the fixed part of the path costs 217 characters, leaving 43 for the name — and the BUMA standard's own name is 56. `_mineru_name` now passes a bounded name instead. Head AND tail are kept, because the names that collide in practice share a long prefix and differ at the end ("… and ECA" versus "… and ECA (2)"); keeping only the head would render exactly those indistinguishable. Verified with a real parse of the 56-character name: 239 characters, 31 chunks, where before it died on MAX_PATH. Uniqueness does not depend on that name — the parent folder is the cache key, which is the full SHA-256 of the file contents. It is cosmetic, exists so the cache can be browsed whil…9083dd1 -
[NOTICKET] revert(knowledge_extraction): X20 — it measured a net -1 (X20 closed)
Rifqi HafizuddinReverts X20's three edits to rule.txt on Rifqi's decision, after the same-day A/B measured them as a net loss. X20 in 5/14, all 5 runs X20 reverted 6/14, all 5 runs Zero variance in both arms. One rule differed: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without, because X20's closing clause -- "a statement that is mostly symbols, = and numbers is an equation" -- cannot distinguish a computation from a symbolic CONDITION, and that gold rule reads "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:". X20's own target, GAINLOSS_UNITS, fired 0% in both arms, so the 26 Aug 6/7 was the day and not the prompt. A conditional carve-out (jika / apabila / maka -> a rule however symbolic its condition) was considered and declined: X20 had a measured cost and no measured benefit, and refining it would spend further runs defending a change that has yet to earn any. The underlying intent -- stop the rule branch duplicating the formula branch -- stays valid for anyone who wants to attempt it…21640d6 -
[NOTICKET] eval(knowledge): X20 measures net -1; close X23 on expert ruling; instrument X25
Rifqi HafizuddinThree things, all from the same session. X20 -- FIRST CONTROLLED A/B IN THIS PROJECT, AND IT SAYS X20 COSTS A RULE. Arms alternated inside one session so drift hit both equally. Arm B reverts X20's three edits and KEEPS X23's synthetic examples, so the only variable is X20 itself. Rule branch only: 15 calls per run, ~35s, five rounds each. X20 in 5/14, every one of 5 runs X20 reverted 6/14, every one of 5 runs Zero variance in both arms, and exactly one rule differs: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without. The cause is X20's own final clause. That gold rule is a genuine conditional whose CONDITION is written symbolically -- "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:" -- and X20 ends with "a statement that is mostly symbols, = and numbers is an equation". The heuristic cannot tell a computation from a symbolic condition, so it nulls a real rule. X20 is also not buying what it was written to buy. GAINLOSS_UNITS, the rule it targets, fires 0% in BOTH arms …e6ed414 -
[NOTICKET] feat(knowledge_extraction): cue class for declarative composition (X24b cause 3)
Rifqi HafizuddinThe rule branch could not see three gold rules because the cue taxonomy had no slot for the voice they are written in. It had conditional cues (jika, apabila, harus) and definitional ones (adalah, merupakan). This corpus states most of its calculation policy in a third voice -- declarative composition: "Nilai PA ... EKUIVALEN DENGAN rata-rata tertimbang ... dengan Quantity sebagai FAKTOR PEMBOBOT" "Perhitungan Pty ... DIPEROLEH DARI hasil pembobotan ..." "Gain/Loss grouping ... merupakan PENJUMLAHAN DARI Gain/Loss model unit" Those are calculation rules by the extraction prompt's own definition -- how a value is computed, sourced or aggregated -- and no existing cue reached any of them. Not a tuning gap; a missing class. Five patterns, chosen by measuring which chunks each reaches rather than by intuition, every one load-bearing for a previously invisible gold rule. Candidates 8 -> 15. \bdihitung menggunakan\b was REJECTED despite matching 5 chunks: it precedes every formu…e6d9d29 -
[NOTICKET] docs(knowledge): correct the rule-ceiling diagnosis — it is not tables
Rifqi HafizuddinYesterday's note said 6 of the 9 never-firing rules live in Tabel 2.1-2.5 and that candidate generation over tables was the next gain. Traced properly today against the artifact, that is wrong: 5 of the 8 candidates already come from tabular chunks, so the cue regex reaches tables fine. Three separate causes, only one of which is anywhere near the prompt. Two gold rules -- PTY_PRODUCTION_SOURCE and PTY_WORKING_HOURS_DEFINITION -- have no matching text in ANY chunk. They are not extractable at all, so either the gold was written against content the parse does not produce, or the parse drops it. Worth knowing which; it is not an extraction problem either way. Notably PTY_PRODUCTION_SOURCE is the rule that used to score. It was worked example 1 in rule.txt verbatim, so the model was reproducing it from the PROMPT rather than from the document. X23 removing that example is why it now reads 0/7, and this is the confirmation: there was never anything in the document to find. One call ret…892f387
-
[NOTICKET] docs(knowledge-parsing): record B3 — MinerU reaches parity, and why
B3 is answered. MinerU hybrid/high through the PRODUCTION module scores E1 0.8293 (34/41) against PyMuPDF's frozen 0.8537 (35/41) — parity with the scratch converter calibration §10 measured, now from code that ships. §10 is EXTENDED, not corrected. Its numbers stand and its attribution was right: the scratch converter did handle list blocks, so `Uncontrollable` really was "present as prose but missed" in that run, exactly as recorded. What B3 adds is that the production module had regressed against that converter, and by how much. The residual 0.0244 is not parsing's to close, and the docs say so rather than leaving it as an open action. `Breakdown` appears three times in the text and sits in the SAME chunks as `Total Hours`, which the filter does find — same chunk, same sentence, `( Total Hours - Breakdown )`, one tagged and one not. So chunk arrangement cannot explain it. Chunk counts rule out over-splitting too: 31 chunks = 15 prose (11 headed, 9 distinct sections) + 14 tables + …e65e685 -
[NOTICKET] fix(knowledge-parsing): normalize.py silently dropped every list block
MinerU keeps list content in `list_items` and gives such items no `text` field. They therefore fell through the generic text path, which reads `item["text"]`, came back empty, and were skipped — no error, no warning, no trace. This is the same shape of bug the chart/image branch was written to prevent, and that branch's comment describes it exactly: "pictures have no such field — so BOTH its caption and its description vanish in silence". Lists have no such field either, and nobody noticed. Measured cost on the BUMA standard: two dropped blocks took five gold terms with them — Joint survey, Truck count, EWH, Controllable, Hierarki Level Analysis. None of the five were "missed by the model"; none were in the artifact at all. Term-filter recall 0.7073 -> 0.8293, +0.1220. List content joins the running text chunk rather than opening its own, because it is prose inside its section. It deliberately skips heading detection: a short bullet can look like a heading, and letting one open a se…6edfbe7 -
[NOTICKET] eval(knowledge): rule-branch noise band — X20 confirmed, X23 answered
Rifqi HafizuddinFive repeat runs at identical config, plus the two from earlier today: seven runs of the same pipeline on the same artifact at temperature 0. 3/15 x1 (the run carrying the latex-guard defect) 4/15 x5 6/15 x1 min 3, median 4, max 6, mean 4.14 The branch is much less noisy than one pair of runs suggested. Five of seven land on exactly 4/15. The rule from here on is: quote the median and the band, never a single run. Per-rule stability turned out to be the useful view, not the total. X20 IS CONFIRMED, and by the rule it names. GAINLOSS_UNITS -- the units sentence that shared a chunk with a Gain/Loss equation, the exact case the fix was written for -- fires in 6 of 7 runs against 0 at the 3/15 baseline. That is a stable +1 sitting outside the noise band, not a lucky sample. Three other rules fire 7/7 and two fire 1/7; those two are the whole of the variance. X23'S GOLD QUESTION IS ANSWERED. With the few-shot that forbade it removed, WATERFALL_TWO_METHODS still f…f754631 -
[NOTICKET] fix(knowledge_extraction): audit the latex normaliser for overfitting
Rifqi Hafizuddin +1: Claude Opus 5Rifqi asked whether the normalisation rules are real or tailored to one document, before spending on repeat runs. Audited against cases this corpus does NOT contain -- every fixture here is BUMA, so the suite could not have caught a rule that is really a fact about one document. Two defects and three scoped assumptions came out of it. The properties that matter held. Different operands, precedence, operand order, sub- versus superscript, and + versus - all stay distinct. That follows from preserving structure, not from anything about this corpus. FIXED -- \limits was never actually stripped. The pattern was \limits\b, and \b does not match between `s` and `_` because `_` is a word character, so \sum\limits_{i} kept its \limits while \limits x lost it. That is the SAME defect X22 found in the eval scorer, in a different file: where a LaTeX command can be followed by a subscript, \b is the wrong boundary. Now (?![A-Za-z]). FIXED -- \dfrac and \tfrac were unhandled, so a model using ei…7d6dedf -
[NOTICKET] fix(knowledge_extraction): the latex guard was rejecting correct transcriptions
Rifqi Hafizuddin +1: Claude Opus 5First real run of the new runner, on Sofhia's hybrid artifact, and it caught a defect I shipped two commits ago. 4 of 6 formulas had formula_latex nulled -- Qty twice, PA and UA -- and all four rejections were FALSE. The cause is the same identity trap, one level deeper. The model reads RENDERED PROSE, and the document writes `Qty = (INPR Hours) / (Total Hours)`. So it answers with a slash and the document's own parentheses, while the source markup carries `\frac{INPR Hours}{Total Hours}` with neither. Two spellings of one division, and the guard called them different formulas. Three fixes, all notational: Division canonicalises to the prose spelling: `\frac{A}{B}` -> `(A)/(B)`, so a slash and a \frac converge instead of diverging. The old `frac(A)(B)` form was also unsafe on its own terms -- it used parentheses as delimiters, so a literal paren in the numerator collided with the delimiter and `\frac{(a)}{(b)}` could never match `\frac{a}{b}`. Redundant grouping parens collapse: `(…22eb3d4 -
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin +1: Claude Opus 5Sofhia supplied the same 11 equations as MinerU transcribed them under four backends -- 44 real strings, committed as a fixture. Running the shipped normaliser over them reproduced her figures exactly: 11/11 for vlm/medium, 10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference. Her independent check and this code agree on both the counts and which equations differ, which is the point of her having run it separately. The remaining gaps were in the command inventory, and the inventory is now evidence rather than guesswork. \mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets the whole of equation 7 under pipeline, including its multiplication sign, and \displaystyle appears in equation 4. Both are handled now, and the wrapped-x rule generalises across every presentational wrapper rather than being \mathrm specific. Equation 4 needed two more, and one of them was a decision Sofhia deliberately left open. The array ALIGNMENT SPEC arrives…01c76dd -
[NOTICKET] eval(knowledge): the runner the harness has been missing
Rifqi Hafizuddin +1: Claude Opus 5The 2026-08-24 figures were assembled by hand. That is why the rule-branch number in the committed result file is 1/15 measured with a scorer that has since been fixed, and why nobody could re-derive it. This is the reproducible version, following the house convention the other four harnesses already use: module mode, timestamped results, never overwritten. It takes a ParsedDocument artifact and never a PDF. MinerU is not invoked and is not required -- that is the seam doing its job, and it is what makes the pending X20 and X23 measurements possible on a machine with no GPU and no parser installed. Free by default: E1 term-filter recall and E2 compression, plus the funnel, with zero API calls. --extract adds the paid branches, E3 schema fill and rule recall. --dry-run prints the token estimate and the per-branch cache-floor status. --mock exercises every branch with no spend, and stamps the result NOT SCOREABLE so a fabricated number can never be quoted as a measurement. Two refusal…1edf61c -
[NOTICKET] docs(knowledge-parsing): record that Chunk.latex is now load-bearing
Until a998ea48 this field was carried but unread — a parser change touching it would have broken nothing. Extraction's formula guard now validates `formula_latex` against it, so that is no longer true, and the contract has to say so before someone tidies it in good faith. The failure mode is the dangerous kind and gets the emphasis: reflowing or dropping this field does not error. The guard simply has less to check against, and claims it can no longer verify are recorded as unverifiable rather than caught. Fewer things fail and nothing looks wrong. The two obligations run OPPOSITE to the rule for `text`, which is why they are stated rather than left to inference. `text` must stay verbatim because the span check searches it literally. `latex` must stay UNCLEANED because extraction canonicalises before comparing — pre-cleaning removes information the canonicaliser was built to absorb. Character-spacing, `\mathrm{}` wrappers and `~` are all expected input, not damage. Also draws the li…a0df6d3 -
[NOTICKET] fix(knowledge_extraction): rule.txt few-shots stop contaminating the gold set (X23)
Rifqi Hafizuddin +1: Claude Opus 5Two of the four worked examples were lifted from the reference corpus, and they contaminated the measurement in opposite directions. Example 2 used "Waterfall Analysis dapat di-review melalui dua metode" and taught that it is NOT a rule. The gold set records that exact sentence as WATERFALL_TWO_METHODS. The prompt was instructing the model to fail one gold rule by construction, so the branch could not score above 14/15. Example 1 was the inverse and had gone unnoticed: its evidence is gold rule PTY_PRODUCTION_SOURCE verbatim (joint survey / truck count). That is a free hit inflating the score, not capping it. Both are now synthetic, which is the reasoning examples 3 and 4 already carried in the prompt itself. The closing note is generalised: all four are synthetic, so neither an answer nor a NON-answer for a real document leaks in. The gold set was deliberately NOT touched, following the X22 precedent -- the prompt is tuned constantly and the gold is the measuring instrument, so th…7e8614f -
[NOTICKET] docs(knowledge-parsing): B1 — Colab, and the two decisions it forces
Pak Ricky prefers a Colab subscription over provisioning a GPU. That is consistent with what §4 already argued — the invoice is decided by the billing model, not the compute — so the structure holds and only the recommendation moves. Colab is also not a migration: every backend figure in §3 was measured there already. Measured against the same corpus: Colab Pro is $9.99/month for 100 compute units and a T4 burns ~1.19 CU/hour, so the allowance is ~84 T4-hours and the entire 16-hour worst-case backfill costs ~19 CU — a fifth of one month, five times over-covered. Effective $0.119/GPU-hour against Azure's $0.752. The spread between best and worst provisioning widens from ~80x to ~290x. Azure figures deliberately STAY. Every hour-to-dollar conversion in §3 and §5 derives from that rate, so removing it would strip the tables of their basis — and it is the fallback if the governance question below lands the wrong way. New §4.1 records the two decisions that are not ours, because the chea…2d2f905 -
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin +1: Claude Opus 5Three paths leave formula_latex populated and they are not equally strong: located in the source markup, no markup to check against, or an operator the source cannot express. Until now the output could not tell them apart, so a verified formula and an unverifiable one reached the expert as identical rows. That gap is not theoretical. MinerU's pipeline backend writes multiplication as a bare letter x, so every product formula from such an artifact is unverifiable -- including the Production formula this contract uses as its own worked example. Every artifact on Sofhia's machine is pipeline, because she has no GPU. FormulaEntry gains latex_verification: verified | unverified_no_markup | unverified_operator, null when there is no claim to characterise. Set by the span check, never by the model -- the model cannot know which guard ran, and a field it could write would be a claim rather than a record. _mark_latex is a never-throw seam like the rest of validation: an entry type without th…f81a227 -
[NOTICKET] fix(knowledge_extraction): compare formula_latex by notation, not identity
Rifqi Hafizuddin +1: Claude Opus 59c8a6e4 gave formula_latex the right haystack and the wrong test. Sofhia ran it against real artifacts and found `Production = MOHH \times Qty \times ...` rejected: the `pipeline` backend transcribes the multiplication sign as a bare letter x, so the claim had nothing to match. Probing further, the problem was wider than the case she hit. Against a source that DOES carry \times, the guard also rejected a model that dropped \mathrm{}, one that used \cdot, and one that wrote idiomatic LaTeX -- 4 of 5 plausible correct outputs. It passed only when the model mirrored MinerU's notation character for character. The test that shipped green passed only because I hand-wrote the fixture to mirror the source. That is the same category error as the original defect, one level up. First the haystack was the wrong FORMAT; then it was the wrong NOTION OF CORRECTNESS. The prompt asks the model to transcribe INTO LaTeX, which is a translation, and identity is not how you check a translation. normalis…a998ea4 -
[NOTICKET] docs(knowledge-parsing): B1 — cost justification for the BUMA corpus
Owed from the 13 Aug executive review. Every number was already measured; this is the write-up, plus one finding that only surfaced while doing the arithmetic. The finding: $0.752/hour is derived from a monthly VM price ($548.96 / 730 h). So $6.89 for the full corpus is only true if the VM is billed per hour and deallocated afterwards. Left running — the default unless someone says otherwise — the same 9.2 hours of work bills $548.96/month. That is an ~80x spread for identical work, and it is the only decision in the document that moves real money, so it leads. Parsing is an admin-triggered offline batch and the service never imports MinerU at runtime, so there is no reason to hold a GPU idle between runs. Also recorded: - Size from the heaviest document, not the average. Density spread is 2.66x (the 9-page standard is 8.51 s/page against the 20-page handbook's 3.20), and the dense document is the shorter one. Planning range $4.55-$12.09; budget the $12 end, since BUM…609e7ba -
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin +1: Claude Opus 5The field was guarded in name only. It sat in GUARDED_FIELDS but not in _TRANSCRIBED, so its value was never searched in the source -- only its provenance.span was. A structurally wrong transcription with a valid prose span passed, which is exactly the shape X21 produced: a \frac that lost its division reads as a product and states something the document does not. Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose and can never contain LaTeX, so that would have nulled every formula entry and presented as a model failure rather than a validation bug. The haystack was the problem, not the check. Chunk.latex is now the haystack. The parsing half has always emitted it; extraction's adapter dropped it on the floor, which is what left the claim unprovable. It now crosses the seam and is compared with whitespace removed ENTIRELY rather than collapsed. That is a different notion of equality, not a weaker one. In LaTeX, whitespace outside \text{} is not significa…9c8a6e4 -
[NOTICKET] docs(knowledge-parsing): settle determinism; finish the English pass
Determinism was the one thing left open on the backend decision: the VLM family is not deterministic in general, so a frozen baseline might drift and no eval number could be trusted. It does not drift. Two runs over the same documents, with the parse cache moved aside so both genuinely re-parsed (from_cache: false on all four records), produced 0 differences across 50 chunks. All four image descriptions were present in both runs and character-for-character identical (192, 256, 77, 146 chars) — those are the only model-written content in the artifact and were the likeliest to wobble. Existing baselines therefore stay valid and need no re-measurement, and hybrid/high is final rather than provisional. Marked settled in config.py, calibration §13, and REPO_STATUS §14. The parsing README was still Indonesian in the how-to-use sections written last week, while the .py files had already moved to English; it is now English throughout. Its console example was also stale — it showed the old I…d5ed50c -
[NOTICKET] feat(knowledge_extraction): output schema v2 — entity ids, links, 16 fields cut
Rifqi Hafizuddin +1: Claude Fable 5Harry's review of the output contract asked for four things: an id on every entity, a formula referenced rather than restated, rules related to formulas, and a smaller schema. All four are adopted. Two defects surfaced while adding the ids, and they matter more than the ids do. Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank ordinal (`#c000` over a list sorted by mention count), so one changed count reshuffled every id after it, and `rule_id` was written by the model, so the same rule could return a different slug next run. The product is an expert working a review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable ids lose which 40. Approval continuity, not linking, is why these exist. New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never positional, never model-supplied. New `link.py` resolves every reference after the branches run — deterministic, no LLM, no spend — and returns dangle counts rather than…49dfe4d -
[NOTICKET] docs(knowledge-parsing): record the backend measurement, and fix three stale facts
The backend decision was in the code and in a commit message, but the evidence behind it had no home in the repo. Anyone asking "why hybrid/high" would find the conclusion and none of the data. **Calibration §13 — all four backends measured.** One GPU, the same 29 pages. `vlm` and `hybrid/high` produce IDENTICAL output (100% LaTeX overlap, same image descriptions), so `vlm` only costs more time. `pipeline` is the only backend copying the multiplication sign as a literal letter, which fuses MOHH/Qty/PA/UA/ Pty into one token — and it is slower than hybrid/medium, so it loses twice. The choice narrows to medium vs high, i.e. whether image descriptions are wanted: $2.67 against $6.89 across 6,800 pages. Also records what the descriptions actually cost in coverage. MinerU only describes a picture it first classified into a sub_type, and the pattern held 6 of 6: no sub_type, no description. Coverage was 4 of 6, and the miss is not explained by image quality — a plainly legible waterfall c…b0cc507 -
[NOTICKET] docs(knowledge): X20 landed unevaluated; X22 resolved; X23 opened
Rifqi Hafizuddin +1: Claude Opus 5Syncs the two knowledge docs to what the branch actually holds after 296cdd7. X20 goes to in-progress, not done. The prompt change is committed; the eval CLAUDE.md 7B requires is not run, and the row says so rather than implying the regression is fixed. It records what the three edits are, that worked example 3 is synthetic on purpose (MTTR, absent from all 15 gold rules) so no gold answer leaks into the prompt, and that example 1 already is PTY_PRODUCTION_SOURCE verbatim -- pre-existing contamination the new example deliberately did not add to. X22 goes to done. Rifqi fixed it in abae344 on 2026-08-24 by changing the scorer, not the gold: underscores are deleted rather than spaced, because spacing turns qty_a into "qty a" which still does not contain qtya. Marked RESOLVED with the reasoning rather than deleted, per house convention. That resolution moves X20's target. The 1/15 in v2_full_document_2026-08-24_093051.json was measured with the pre-X22 scorer, so the number any X20 re-…e84487d -
[NOTICKET] fix(knowledge_extraction): rule.txt excludes bare equations (X20)
Rifqi Hafizuddin +1: Claude Opus 5The rule branch was spending calls re-extracting equations the formula branch already captures, and dropping the operational prose it exists for. 4 of 8 v2 rule statements were bare equations; in the prototype 0 of 7 were. The cause is not the model. Before rendering, formulas reached it as mangled character-spaced LaTeX and did not read as statements. Rendered to clean prose they read exactly like prescriptive rules, and rule.txt never excluded them -- it defined a rule as "a statement that constrains or directs behaviour", which a formula arguably is. Rendering gained +7.3 recall points on the term filter, so the fix belongs in the prompt, not in undoing the rendering. Three edits. A rule is now PROSE, with a paragraph saying an equation belongs to the formula branch and that "calculation policy" means prose stating how, when or on what basis a calculation applies. A new requirement 3 sets statement to null when the only prescriptive content is a formula. And -- the clause that mat…296cdd7
-
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments, manifest keys and console output. Everything else in the repo is English, and this module is the newest work, so it is the one to bring into line rather than the older code around it. Renamed across the module: parse_dokumen -> parse_document, normalisasi -> normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak -> trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen -> DocumentRecord, and the locals with them. Docstrings and comments were REWRITTEN rather than translated word for word. Several carry the reasoning behind a decision — why `\frac` is unwrapped before the braced wrappers, why a heading never enters `text`, why the digit threshold is 3 — and a mechanical translation would have flattened exactly the part worth keeping. Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan -> warnings, jenis -> kind, galat -> error. That is a data change, …7f153d6
-
[NOTICKET] docs(knowledge-parsing): how to run it, and how to read what comes back
The README still said `python -m src.knowledge_parsing.run` with no environment set, and pointed at `uv sync --extra knowledge-parsing` — a command that fails. Four sections that answer questions actually asked this week. **How to run it**, and what a successful run looks like, with real console output. First-time readers cannot tell whether `⚠ 3 peringatan` is routine or alarming, and that is the one line worth reading: warnings are normal, `ok (cache)` means nothing was re-parsed, and only `ok / gagal` needs attention. **Backend**, with the measured table and why hybrid/high won — including that `vlm` is identical but slower and `pipeline` fuses the multiplication sign. Stability is marked open, not settled. **Running without a GPU**: the runner drops to `pipeline` on its own, and the result is valid but LOWER FIDELITY — no image descriptions, multiplication copied as a letter. Easy to forget, so it is stated where it will be seen. **Installing MinerU**, with an explicit do-not-…b3eae0f -
[NOTICKET] feat(knowledge-parsing): one runner, in Python rather than shell
Running the pipeline meant six environment variables and an interpreter path from another repo, typed out every time. This wraps it: py scripts/parse.py # Windows python3 scripts/parse.py # Linux Python, not shell, and that was learned the hard way. A bash version and a PowerShell version were both written first, and between them hit two traps in twenty minutes: - CRLF. core.autocrlf=true rewrites .sh on Windows checkout, and bash then dies on line one with `set: pipefail` — a message that names nothing. - A trailing backslash. The default input path contains spaces, so PowerShell quotes it, the final backslash escapes the closing quote, and the NEXT argument is swallowed: `--backend pipeline` ended up inside `--input`. `subprocess` takes an argument list and never goes through a shell, so neither is possible here. One file instead of two, and readable by anyone on the team without knowing both shells. The backend is chosen from…74acbf7 -
[NOTICKET] feat(knowledge-parsing): default to hybrid/high, the measured choice
All four backends run on the same GPU, same two documents, 29 pages: hybrid/high 4.85 s/page image descriptions YES hybrid/medium 1.88 s/page descriptions FORCE-DISABLED by MinerU pipeline 2.67 s/page never analyses images at all vlm 6.96 s/page output IDENTICAL to hybrid/high, 100% Two findings decided it. `vlm` and `hybrid/high` produce the same LaTeX and the same descriptions, so `vlm` only costs more time for nothing. And `pipeline` loses twice: slower than hybrid/medium, and the only backend that copies the multiplication sign as a literal `x`, which fuses `MOHH x Qty x PA` into one token and takes five terms with it. The `high` / `medium` gap is $6.89 against $2.67 across the whole 6,800-page BUMA corpus — $4, too small to trade image descriptions for. `effort` moves to "high" in the same commit, and that is not cosmetic: MinerU force-disables image analysis at medium, so flipping the backend alone would have produced hybrid WI…52e1bd4 -
[NOTICKET] fix(eval/knowledge): stop the scorer treating `_` as a word character (X22)
Rifqi Hafizuddin +1: Claude Opus 5`norm()` flattened every punctuation mark to a space except the underscore, because `_` is inside `\w`. The gold writes notation bare (`QtyA`) while the renderer folds the LaTeX subscript and emits `Qty_A`. Those normalised to `qtya` and `qty_a`, the substring test failed, and a correct entry was scored as a miss — the instrument was wrong, not the extraction. Underscores are deleted rather than spaced. Spacing only turns `qty_a` into `qty a`, which still does not contain `qtya`; deletion is what makes the two notations meet. The gold is NOT edited. Verified: none of the 15 rules contains an underscore, so no gold fragment changes under the new normaliser and matches can only move from miss to hit. The alternative — rewriting the gold to `Qty_A` — was rejected because it binds the frozen instrument to one renderer's current output, and rendering has already changed once this week and will change again with the backend decision. Two rules use the affected notation, GAINLOSS_QTY_ZERO_…abae344 -
[NOTICKET] fix(knowledge_extraction): refuse an artifact that will not name its backend
Rifqi Hafizuddin +1: Claude Opus 5MinerU's `pipeline` and `vlm` backends emit different text from the same PDF. The same equation arrives as a semantic `\times` from one and as a spaced literal `x` from the other, and word boundaries that survive in `\text{...}` under `vlm` are gone under `pipeline`. Two extraction results measured on different backends are therefore not comparable, and nothing downstream can tell them apart after the fact. This is not hypothetical: the two halves spent a week comparing numbers taken from two different parses and read the disagreement as a rendering bug. Each side was measuring correctly, on a different artifact. The provenance already existed and was walked around. `ParsedDocument` carries `parser_backend`, and the parsing half reads it from MinerU's own `_middle.json` rather than from config, precisely so the recorded backend is the one actually used. It never reached extraction because what crossed the seam was a bare `content_list.json`, stripped of its envelope. So the CLI now …4db5f14 -
[NOTICKET] fix(knowledge): settle the chunk_id separator on `::`
Rifqi Hafizuddin +1: Claude Opus 5The two halves minted provenance keys differently: the parsing half emits `<doc_id>::<seq>`, while the extraction adapter fell back to `<doc_id>#<seq>` for artifacts arriving without one. Nothing broke in practice — the adapter prefers the supplied id, so real runs always carried `::` — but the two shapes could diverge the moment a fixture or an older artifact took the fallback. `::` wins: it is what the producer actually emits, the calibration doc already uses it in prose, and `#` is URL-fragment syntax that would bite once these ids become MCP resource identifiers. The `KNOWLEDGE_OUTPUT_CONTRACT.md` cited as the source of `#` does not exist in the repo or its history. The separator is now stated in the contract next to the field, because this is the second time an undocumented detail of the seam's shape has cost both halves a round trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>e8cc26c -
[NOTICKET] docs(knowledge-parsing): contract 0.3.0 — image kind + generated_description
Records what the code change added: `image` joins ChunkKind, and `generated_description` carries MinerU's model-written description of a picture. Two things a consumer would otherwise get wrong. Backends disagree on the label — a figure `pipeline` calls a chart, `vlm` calls an image — so branching on one of them silently misses half the cases. And whether descriptions exist at all depends on the backend: never on `pipeline`, disabled outright on `hybrid` medium, on for `hybrid` high and always for `vlm`. An artifact carrying nulls here is not a parsing failure, it is the backend, which is one more reason `parser_backend` has to travel with the data. Also states why the field is kept out of `text`: that field is what the extraction guardrail searches to prove a quote is verbatim, and generated prose there would let a hallucination pass the check built to catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>f0e54e9 -
[NOTICKET] feat(knowledge-parsing): carry image descriptions, apart from the text
MinerU describes images and charts, but only on some backends. Read from its own source (is_image_analysis_option_visible): never on `pipeline`, always on `vlm`, and on `hybrid` only at effort=high — medium disables it outright. Two faults kept that description from ever arriving. `normalize.py` had no branch for `image` items at all. Different backends label the same picture differently: what `pipeline` calls a chart, `vlm` calls an image. Image items fell through to the generic text path, which reads only item["text"] — a field pictures do not have — so caption AND description were dropped in silence. That is most of what looked like "vlm loses chart detection" in yesterday's comparison: vlm found them, we discarded them. The second is worse and was waiting to fire. `_teks_chart` appended `content` straight into Chunk.text. Harmless today because `pipeline` always leaves it empty — but the moment the backend flips to vlm, model-written prose starts flowing into the field the extra…c113bf8 -
Merge commit 'refs/pr/20' of https://huggingface.co/spaces/DataEyond/Agentic-Service-Data-Eyond-Catalog into pr/20
6e78f1e -
[NOTICKET] docs: reorganise into docs/ and give every doc a status banner
Rifqi Hafizuddin +1: Claude Opus 5The root carried 14 markdown files and no way to tell which were authoritative. That knowledge lived in CLAUDE.md §1 and in Rifqi's head, which works for us and fails for anyone else — and one of the stale ones already has a named failure mode ("The Stale-Doc Believer") for being implemented from. Root now holds four: REPO_STATUS, DEV_PLAN, API_CONTRACT_BE_PYTHON, README. Everything else moves under docs/, by git mv so history follows: docs/knowledge/ context, TODO, calibration, parsing output contract docs/plans/ SPINE_V2_PLAN (approved, W3 deferred), DS_TOOLS_PLAN (proposal) docs/archive/ API_ENDPOINTS_RESTRUCTURE, CODE_REVIEW_2026-07-23, PROJECT_BRD docs/reference/ API_CONTRACT_BE_GOLANG (the Go repo's, a copy that can drift) Every doc gains a one-line status banner under its H1, marked <!-- doc-status -->: LIVE / LIVE PLAN / PROPOSAL / ARCHIVED / REFERENCE. Archived ones name what superseded them — API_ENDPOINTS_RESTRUCTURE now opens with "DO NOT IMPLEMENT FROM…f1342ce -
[NOTICKET] fix(knowledge-parsing): let the source prove itself before trusting it
Word-boundary recovery treats the PDF text layer as the arbiter. That is safe when the layer is intact and dangerous when it is HALF there: a legitimate word looks unknown, and gets split. An empty text layer was already safe — no vocabulary, no recovery. The gap was the middle ground, and nothing measured it. So the page now has to prove itself: how many of MinerU's own prose words on that page are actually found in its text layer. A low score means the two read the same page differently, and a source that disagrees with the parser is not fit to arbitrate it. Pages scoring below 90%, or carrying too little prose to judge at all, lose their vocabulary and skip recovery — left damaged but honest, rather than repaired into something the document never said. This replaces a constant tuned against one document with a measure that assesses each page on its own evidence. Measured across three documents, two of them a McGraw-Hill textbook in English with nothing in common with the BUMA st…22c6ad1