Commit History
-
[NOTICKET] feat(knowledge-extraction): subdomains + counts — v3 complete
Rifqi Hafizuddin +1: Claude Opus 5The last two DomainContext v3 fields, both deterministic. No LLM, no prompt change, no spend. subdomains Rolled up from the glossary entries' own `subdomain_tags`, never asked of the model. Each tag was already chosen once, per term, with that term's evidence in front of the model; a second document-level classification would be the same judgement made with less context and no way to check it. Ordered by tag frequency descending with an ALPHABETICAL TIEBREAK, which is load-bearing rather than tidy: on the reference document `production` and `equipment` both score 5, and the entries list them in differing order, so without the tiebreak two runs of one document would disagree. n_terms / n_formulas / n_rules Counts of the other three artifacts. Cheap orientation for a reviewer deciding whether a queue is worth opening, and the fastest way to spot a run that silently extracted nothing. Both are computed in `_finalise_domain`, called at the END of extract_all rathe…172d8dc -
[NOTICKET] feat(knowledge-extraction): span-guard the domain branch
Rifqi Hafizuddin +1: Claude Opus 5The domain branch was the pipeline's ONE unverifiable output: GUARDED_FIELDS ["summary"] was `()`, so `title` and `purpose` were unchecked model prose on nano, the tier whose failure mode is least detectable. Every other branch had to prove its claims against the source; this one did not. The fix is not a bigger model. It is asking a different question: v2 "write a short orientation brief" -> composed -> unprovable v3 "locate the document's own statement" -> quoted -> checkable A summary cannot be found in the source by construction. A quote can. So `purpose` becomes `purpose_verbatim`, prompts/summary.txt is rewritten from summarise to locate, and both text fields join GUARDED_FIELDS + _TRANSCRIBED. Also wires validation into `_run_summary`, which never called validate_entry - the guard table alone would have been inert. The haystack is the WHOLE document, unlike the other branches: this one is handed every chunk, so a quote may legitimately come from anywhere in it. …e2865ac -
[NOTICKET] feat(knowledge-extraction): derived outline + corroborated key parameters
Rifqi Hafizuddin +1: Claude Opus 5The two DETERMINISTIC halves of the DomainContext v3 reshape. No LLM, no spend, no prompt change - which is exactly why they ship ahead of the rest: the parts that need a prompt rewrite and an eval run are split off and still proposed. outline Derived from the artifact's heading_path, in reading order, de-duplicated. The blocker was upstream: adapter.py was dropping heading_path on the floor, so the internal Chunk never carried it - the same way it once dropped `latex`, and with the same consequence, a branch inventing prose about a document whose own structure was sitting unread. Fixed at the seam. It is the only field on this branch carrying the strong guarantee for free: verbatim source structure, nothing to hallucinate. On the BUMA fixture it yields 11 entries from 29 of 31 chunks, including the verbatim "Physical of Availability (PA)" the pipeline is required to surface rather than tidy. key_parameters Was list[str]; now list[KeyParameter{surface, term_id}],…6f5d619
-
[NOTICKET] refactor(knowledge-extraction): BriefContext -> DomainContext
Rifqi Hafizuddin +1: Claude Opus 5The workstream diagram has always called this box "Domain Knowledge", while the model and its fields described a short orientation card. Rename the entity to match the intent before anything is persisted: once an expert has approved rows keyed on these ids, changing the `b_` prefix is an id migration that loses the approvals - the exact failure the deterministic-id scheme exists to prevent. BriefContext -> DomainContext brief_id -> domain_id (prefix b_ -> d_) ExtractionResult.brief -> .domain brief_context.json -> domain_context.json Deliberately unchanged: - `Branch = "summary"` and GUARDED_FIELDS["summary"]. The branch literal is the measurement key in CallUsage.branch / RejectedField.branch; renaming it would break comparability with the committed eval baseline. - prompts/summary.txt. Editing prompt text needs the eval path, and the rewrite belongs with the v3 schema change, not with a rename. No field, prompt or guarding change - the …7f47c61
-
[NOTICKET] revert(knowledge_extraction): X20 — it measured a net -1 (X20 closed)
Rifqi HafizuddinReverts X20's three edits to rule.txt on Rifqi's decision, after the same-day A/B measured them as a net loss. X20 in 5/14, all 5 runs X20 reverted 6/14, all 5 runs Zero variance in both arms. One rule differed: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without, because X20's closing clause -- "a statement that is mostly symbols, = and numbers is an equation" -- cannot distinguish a computation from a symbolic CONDITION, and that gold rule reads "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:". X20's own target, GAINLOSS_UNITS, fired 0% in both arms, so the 26 Aug 6/7 was the day and not the prompt. A conditional carve-out (jika / apabila / maka -> a rule however symbolic its condition) was considered and declined: X20 had a measured cost and no measured benefit, and refining it would spend further runs defending a change that has yet to earn any. The underlying intent -- stop the rule branch duplicating the formula branch -- stays valid for anyone who wants to attempt it…21640d6 -
[NOTICKET] eval(knowledge): X20 measures net -1; close X23 on expert ruling; instrument X25
Rifqi HafizuddinThree things, all from the same session. X20 -- FIRST CONTROLLED A/B IN THIS PROJECT, AND IT SAYS X20 COSTS A RULE. Arms alternated inside one session so drift hit both equally. Arm B reverts X20's three edits and KEEPS X23's synthetic examples, so the only variable is X20 itself. Rule branch only: 15 calls per run, ~35s, five rounds each. X20 in 5/14, every one of 5 runs X20 reverted 6/14, every one of 5 runs Zero variance in both arms, and exactly one rule differs: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without. The cause is X20's own final clause. That gold rule is a genuine conditional whose CONDITION is written symbolically -- "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:" -- and X20 ends with "a statement that is mostly symbols, = and numbers is an equation". The heuristic cannot tell a computation from a symbolic condition, so it nulls a real rule. X20 is also not buying what it was written to buy. GAINLOSS_UNITS, the rule it targets, fires 0% in BOTH arms …e6ed414 -
[NOTICKET] feat(knowledge_extraction): cue class for declarative composition (X24b cause 3)
Rifqi HafizuddinThe rule branch could not see three gold rules because the cue taxonomy had no slot for the voice they are written in. It had conditional cues (jika, apabila, harus) and definitional ones (adalah, merupakan). This corpus states most of its calculation policy in a third voice -- declarative composition: "Nilai PA ... EKUIVALEN DENGAN rata-rata tertimbang ... dengan Quantity sebagai FAKTOR PEMBOBOT" "Perhitungan Pty ... DIPEROLEH DARI hasil pembobotan ..." "Gain/Loss grouping ... merupakan PENJUMLAHAN DARI Gain/Loss model unit" Those are calculation rules by the extraction prompt's own definition -- how a value is computed, sourced or aggregated -- and no existing cue reached any of them. Not a tuning gap; a missing class. Five patterns, chosen by measuring which chunks each reaches rather than by intuition, every one load-bearing for a previously invisible gold rule. Candidates 8 -> 15. \bdihitung menggunakan\b was REJECTED despite matching 5 chunks: it precedes every formu…e6d9d29
-
[NOTICKET] fix(knowledge_extraction): audit the latex normaliser for overfitting
Rifqi Hafizuddin +1: Claude Opus 5Rifqi asked whether the normalisation rules are real or tailored to one document, before spending on repeat runs. Audited against cases this corpus does NOT contain -- every fixture here is BUMA, so the suite could not have caught a rule that is really a fact about one document. Two defects and three scoped assumptions came out of it. The properties that matter held. Different operands, precedence, operand order, sub- versus superscript, and + versus - all stay distinct. That follows from preserving structure, not from anything about this corpus. FIXED -- \limits was never actually stripped. The pattern was \limits\b, and \b does not match between `s` and `_` because `_` is a word character, so \sum\limits_{i} kept its \limits while \limits x lost it. That is the SAME defect X22 found in the eval scorer, in a different file: where a LaTeX command can be followed by a subscript, \b is the wrong boundary. Now (?![A-Za-z]). FIXED -- \dfrac and \tfrac were unhandled, so a model using ei…7d6dedf -
[NOTICKET] fix(knowledge_extraction): the latex guard was rejecting correct transcriptions
Rifqi Hafizuddin +1: Claude Opus 5First real run of the new runner, on Sofhia's hybrid artifact, and it caught a defect I shipped two commits ago. 4 of 6 formulas had formula_latex nulled -- Qty twice, PA and UA -- and all four rejections were FALSE. The cause is the same identity trap, one level deeper. The model reads RENDERED PROSE, and the document writes `Qty = (INPR Hours) / (Total Hours)`. So it answers with a slash and the document's own parentheses, while the source markup carries `\frac{INPR Hours}{Total Hours}` with neither. Two spellings of one division, and the guard called them different formulas. Three fixes, all notational: Division canonicalises to the prose spelling: `\frac{A}{B}` -> `(A)/(B)`, so a slash and a \frac converge instead of diverging. The old `frac(A)(B)` form was also unsafe on its own terms -- it used parentheses as delimiters, so a literal paren in the numerator collided with the delimiter and `\frac{(a)}{(b)}` could never match `\frac{a}{b}`. Redundant grouping parens collapse: `(…22eb3d4 -
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin +1: Claude Opus 5Sofhia supplied the same 11 equations as MinerU transcribed them under four backends -- 44 real strings, committed as a fixture. Running the shipped normaliser over them reproduced her figures exactly: 11/11 for vlm/medium, 10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference. Her independent check and this code agree on both the counts and which equations differ, which is the point of her having run it separately. The remaining gaps were in the command inventory, and the inventory is now evidence rather than guesswork. \mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets the whole of equation 7 under pipeline, including its multiplication sign, and \displaystyle appears in equation 4. Both are handled now, and the wrapped-x rule generalises across every presentational wrapper rather than being \mathrm specific. Equation 4 needed two more, and one of them was a decision Sofhia deliberately left open. The array ALIGNMENT SPEC arrives…01c76dd -
[NOTICKET] fix(knowledge_extraction): rule.txt few-shots stop contaminating the gold set (X23)
Rifqi Hafizuddin +1: Claude Opus 5Two of the four worked examples were lifted from the reference corpus, and they contaminated the measurement in opposite directions. Example 2 used "Waterfall Analysis dapat di-review melalui dua metode" and taught that it is NOT a rule. The gold set records that exact sentence as WATERFALL_TWO_METHODS. The prompt was instructing the model to fail one gold rule by construction, so the branch could not score above 14/15. Example 1 was the inverse and had gone unnoticed: its evidence is gold rule PTY_PRODUCTION_SOURCE verbatim (joint survey / truck count). That is a free hit inflating the score, not capping it. Both are now synthetic, which is the reasoning examples 3 and 4 already carried in the prompt itself. The closing note is generalised: all four are synthetic, so neither an answer nor a NON-answer for a real document leaks in. The gold set was deliberately NOT touched, following the X22 precedent -- the prompt is tuned constantly and the gold is the measuring instrument, so th…7e8614f -
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin +1: Claude Opus 5Three paths leave formula_latex populated and they are not equally strong: located in the source markup, no markup to check against, or an operator the source cannot express. Until now the output could not tell them apart, so a verified formula and an unverifiable one reached the expert as identical rows. That gap is not theoretical. MinerU's pipeline backend writes multiplication as a bare letter x, so every product formula from such an artifact is unverifiable -- including the Production formula this contract uses as its own worked example. Every artifact on Sofhia's machine is pipeline, because she has no GPU. FormulaEntry gains latex_verification: verified | unverified_no_markup | unverified_operator, null when there is no claim to characterise. Set by the span check, never by the model -- the model cannot know which guard ran, and a field it could write would be a claim rather than a record. _mark_latex is a never-throw seam like the rest of validation: an entry type without th…f81a227 -
[NOTICKET] fix(knowledge_extraction): compare formula_latex by notation, not identity
Rifqi Hafizuddin +1: Claude Opus 59c8a6e4 gave formula_latex the right haystack and the wrong test. Sofhia ran it against real artifacts and found `Production = MOHH \times Qty \times ...` rejected: the `pipeline` backend transcribes the multiplication sign as a bare letter x, so the claim had nothing to match. Probing further, the problem was wider than the case she hit. Against a source that DOES carry \times, the guard also rejected a model that dropped \mathrm{}, one that used \cdot, and one that wrote idiomatic LaTeX -- 4 of 5 plausible correct outputs. It passed only when the model mirrored MinerU's notation character for character. The test that shipped green passed only because I hand-wrote the fixture to mirror the source. That is the same category error as the original defect, one level up. First the haystack was the wrong FORMAT; then it was the wrong NOTION OF CORRECTNESS. The prompt asks the model to transcribe INTO LaTeX, which is a translation, and identity is not how you check a translation. normalis…a998ea4 -
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin +1: Claude Opus 5The field was guarded in name only. It sat in GUARDED_FIELDS but not in _TRANSCRIBED, so its value was never searched in the source -- only its provenance.span was. A structurally wrong transcription with a valid prose span passed, which is exactly the shape X21 produced: a \frac that lost its division reads as a product and states something the document does not. Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose and can never contain LaTeX, so that would have nulled every formula entry and presented as a model failure rather than a validation bug. The haystack was the problem, not the check. Chunk.latex is now the haystack. The parsing half has always emitted it; extraction's adapter dropped it on the floor, which is what left the claim unprovable. It now crosses the seam and is compared with whitespace removed ENTIRELY rather than collapsed. That is a different notion of equality, not a weaker one. In LaTeX, whitespace outside \text{} is not significa…9c8a6e4 -
[NOTICKET] feat(knowledge_extraction): output schema v2 — entity ids, links, 16 fields cut
Rifqi Hafizuddin +1: Claude Fable 5Harry's review of the output contract asked for four things: an id on every entity, a formula referenced rather than restated, rules related to formulas, and a smaller schema. All four are adopted. Two defects surfaced while adding the ids, and they matter more than the ids do. Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank ordinal (`#c000` over a list sorted by mention count), so one changed count reshuffled every id after it, and `rule_id` was written by the model, so the same rule could return a different slug next run. The product is an expert working a review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable ids lose which 40. Approval continuity, not linking, is why these exist. New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never positional, never model-supplied. New `link.py` resolves every reference after the branches run — deterministic, no LLM, no spend — and returns dangle counts rather than…49dfe4d -
[NOTICKET] fix(knowledge_extraction): rule.txt excludes bare equations (X20)
Rifqi Hafizuddin +1: Claude Opus 5The rule branch was spending calls re-extracting equations the formula branch already captures, and dropping the operational prose it exists for. 4 of 8 v2 rule statements were bare equations; in the prototype 0 of 7 were. The cause is not the model. Before rendering, formulas reached it as mangled character-spaced LaTeX and did not read as statements. Rendered to clean prose they read exactly like prescriptive rules, and rule.txt never excluded them -- it defined a rule as "a statement that constrains or directs behaviour", which a formula arguably is. Rendering gained +7.3 recall points on the term filter, so the fix belongs in the prompt, not in undoing the rendering. Three edits. A rule is now PROSE, with a paragraph saying an equation belongs to the formula branch and that "calculation policy" means prose stating how, when or on what basis a calculation applies. A new requirement 3 sets statement to null when the only prescriptive content is a formula. And -- the clause that mat…296cdd7
-
[NOTICKET] fix(knowledge_extraction): refuse an artifact that will not name its backend
Rifqi Hafizuddin +1: Claude Opus 5MinerU's `pipeline` and `vlm` backends emit different text from the same PDF. The same equation arrives as a semantic `\times` from one and as a spaced literal `x` from the other, and word boundaries that survive in `\text{...}` under `vlm` are gone under `pipeline`. Two extraction results measured on different backends are therefore not comparable, and nothing downstream can tell them apart after the fact. This is not hypothetical: the two halves spent a week comparing numbers taken from two different parses and read the disagreement as a rendering bug. Each side was measuring correctly, on a different artifact. The provenance already existed and was walked around. `ParsedDocument` carries `parser_backend`, and the parsing half reads it from MinerU's own `_middle.json` rather than from config, precisely so the recorded backend is the one actually used. It never reached extraction because what crossed the seam was a bare `content_list.json`, stripped of its envelope. So the CLI now …4db5f14 -
[NOTICKET] fix(knowledge): settle the chunk_id separator on `::`
Rifqi Hafizuddin +1: Claude Opus 5The two halves minted provenance keys differently: the parsing half emits `<doc_id>::<seq>`, while the extraction adapter fell back to `<doc_id>#<seq>` for artifacts arriving without one. Nothing broke in practice — the adapter prefers the supplied id, so real runs always carried `::` — but the two shapes could diverge the moment a fixture or an older artifact took the fallback. `::` wins: it is what the producer actually emits, the calibration doc already uses it in prose, and `#` is URL-fragment syntax that would bite once these ids become MCP resource identifiers. The `KNOWLEDGE_OUTPUT_CONTRACT.md` cited as the source of `#` does not exist in the repo or its history. The separator is now stated in the contract next to the field, because this is the second time an undocumented detail of the seam's shape has cost both halves a round trip. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>e8cc26c -
[NOTICKET] eval(knowledge): full document through both halves — E3 0.90 PASS; fix legend blank-line bug
Rifqi Hafizuddin +1: Claude Fable 5First full-document run with the parsing module building the artifact and the extraction module running every branch. Result committed beside the frozen baseline: eval/knowledge/results/v2_full_document_2026-08-24_093051.json E1 term-filter recall 0.8049 (prototype 0.8537) PASS, kill 0.70 E3 schema fill 0.90 (prototype 0.75) PASS, kill 0.80 rule-branch recall 0.0667 (prototype 0.2667) REGRESSION E3 is the headline: the one experiment that FAILED in the prototype now clears its line. It needs the same care the original number did — the scoreable base is 10 entries, so one adjudication moves it ~10 points, and nano is not deterministic at temperature 0. The single failure (produksi) is the same one the prototype recorded and reads as a gold-adjudication case, not a model error. E1 sits below the baseline because the CHUNKING differs, not the filter: the parsing module splits to a token budget (31 chunks) where the earlier comparison used 14, so th…ec7d3d4
-
[NOTICKET] fix(knowledge_extraction): show the model the section heading
Rifqi Hafizuddin +1: Claude Fable 5The evidence block sent to the model carried only `chunk.text`, so the section heading never reached it. Two consequences, both silent: - Indonesian standards name the term in the heading and open the body with the definition — "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik…" — and the body often never repeats the term. The model was being asked to define a term its evidence never named. - It broke the invariant that what the model reads is exactly what the span check searches. `validate.evidence_text` already composed heading + text, so a field quoting a section title would have been rejected as unlocatable — and the model could not have quoted one anyway. Found while verifying the claim in the flow illustration: "Physical of Availability" appeared in the prompt only because the prompt file's own worked example contains that string, never from the document being processed. The deterministic `source_wording` fallback masked the effect, which is why th…a7bdeae -
[NOTICKET] feat(knowledge_extraction): consume the parsing half's ParsedDocument
Rifqi Hafizuddin +1: Claude Fable 5Wires the adapter to the contract Sofhia landed in src/knowledge_parsing, so the seam is exercised with the real types on both sides rather than a hand-written dict. - accepts a ParsedDocument instance directly, not just a dict - page_idx / page_idxs / page_idx_end are now the primary field names; the older draft and prototype names still load so existing fixtures keep working - page numbers pass through 0-BASED with no conversion anywhere. Converting is the review UI's job, done once at display time — an off-by-one here would be invisible until an expert opened the wrong page - parser_backend is folded into parser_version, because the same MinerU build emits different text from `pipeline` and `vlm`; a shift in extraction output has to stay attributable to one or the other New local seam tests import src.knowledge_parsing.contracts directly, so a field rename on either side fails there instead of silently degrading a corpus run. They pin the three properties the seam exists…f132cc1
-
[NOTICKET] feat(knowledge_extraction): paid extraction stage + validate, diff, queue
Rifqi Hafizuddin +1: Claude Fable 5Completes every stage that does not depend on the seam. The pipeline now runs end to end: artifact -> filters -> cluster -> rank -> LLM -> validate -> diff -> frequency-sorted review queue. extract/ client on the nano __54n quad; structured output is PROBED not assumed (json_schema, falling back to json_object + validate-retry, recording which applied); cached tokens read from the API, never modelled. Four branches, one call per TERM CLUSTER. Prompts stay as files so the fixed prefix is byte-identical and caching keeps engaging. validate/ span check (whitespace-only normalisation, never repairs, unlocatable provenance rejects every guarded field), escalation, conflict detection by token overlap — explainable to the reviewer, and it never picks a winner. diff/ new / duplicate / conflicting against an explicitly supplied active glossary, so the duplicate and conflicting paths are reachable. The …ab5ea78 -
[NOTICKET] feat(knowledge_extraction): span filter — E1 recall 0.8537, matches the frozen baseline
Rifqi Hafizuddin +1: Claude Fable 5Adds the term-filter stage, the last free stage before the paid extraction call. Dependencies (approved): gliner>=0.2.13 (resolved 0.2.28), torch>=2.6 (resolved 2.11.0+cpu). Install verified clean — `import torchgen` succeeds, so not the corrupt-wheel failure recorded in the calibration doc. filters/span_filter.py runs the model on EVERY chunk, unconditionally. There is deliberately no relevance gate in front of it: a pre-filter's false negatives drop content before extraction sees it, and dropped content never reaches expert review to be caught. "Zero candidate spans" is the relevance signal instead. Two mechanical traps handled, both of which silently cost recall: - the model truncates long inputs and WARNS rather than failing, so chunks are fed as overlapping 130-word windows with character offsets remapped. Offsets are walked over the original spacing, never re-joined — re-joining would shift every offset in a chunk containing newlines and break every span check. - the ~12…3a01634 -
[NOTICKET] fix(knowledge_extraction): move runner into the package; fix two ranking defects
Rifqi Hafizuddin +1: Claude Fable 5Runner relocated scripts/run_knowledge_extraction.py -> the package as src/knowledge_extraction/cli.py. scripts/ is gitignored, and this runner is the pipeline's operator entry point, so it has to ship with the module: uv run --no-sync python -m src.knowledge_extraction.cli <artifact.json> pyproject: narrow the T201 lint exemption to that one CLI file rather than all of scripts/, so the ignore traces to this change and nothing else. Two ranking defects found while verifying against the real 9-page BUMA standard (both recorded as deviations V1/V2 in KNOWLEDGE_PIPELINE_CALIBRATION.md §9): V1 — a heading that names the term now counts as a mention at offset 0 for cue-proximity scoring. Indonesian standards are written "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik...": the heading names the term and the body opens with the definitional cue WITHOUT repeating it, so the definition section earned no cue bonus at all. Measured before the fix, PA's own definition se…72605ff -
[NOTICKET] feat(knowledge_extraction): free stages — models, seam adapter, filters, cluster, ranking
Rifqi Hafizuddin +1: Claude Fable 5First slice of extraction v2, rebuilt in this repo (the kex prototype is not being ported). All stages here are free: zero API calls, CPU only. New module src/knowledge_extraction/ (named to avoid colliding with src/knowledge/, the existing OCR->pgvector service, which is untouched): - models.py pydantic v2 contracts. Encodes the three invariants: content fields Optional, subdomain_tags as an enum, provenance.span mandatory. Adds source_wording so the literal source wording survives instead of being normalised away. - adapter.py THE ONLY file that knows the seam's shape. Stages depend on an internal Chunk subset, so the artifact contract can settle either way without touching a stage. Verified against both live shapes: Sofhia's bare-list draft with its `kind` discriminator, and the document-level envelope proposed in the checklist.…024c30a