[NOTICKET] feat(knowledge-extraction): span-guard the domain branch
Rifqi Hafizuddin+1: Claude Opus 5
e2865ac
The domain branch was the pipeline's ONE unverifiable output: GUARDED_FIELDS
["summary"] was `()`, so `title` and `purpose` were unchecked model prose on
nano, the tier whose failure mode is least detectable. Every other branch had to
prove its claims against the source; this one did not.
The fix is not a bigger model. It is asking a different question:
v2 "write a short orientation brief" -> composed -> unprovable
v3 "locate the document's own statement" -> quoted -> checkable
A summary cannot be found in the source by construction. A quote can. So
`purpose` becomes `purpose_verbatim`, prompts/summary.txt is rewritten from
summarise to locate, and both text fields join GUARDED_FIELDS + _TRANSCRIBED.
Also wires validation into `_run_summary`, which never called validate_entry -
the guard table alone would have been inert. The haystack is the WHOLE document,
unlike the other branches: this one is handed every chunk, so a quote may
legitimately come from anywhere in it.
…
[NOTICKET] fix(knowledge_extraction): audit the latex normaliser for overfitting
Rifqi Hafizuddin+1: Claude Opus 5
7d6dedf
Rifqi asked whether the normalisation rules are real or tailored to one
document, before spending on repeat runs. Audited against cases this corpus does
NOT contain -- every fixture here is BUMA, so the suite could not have caught a
rule that is really a fact about one document. Two defects and three scoped
assumptions came out of it.
The properties that matter held. Different operands, precedence, operand order,
sub- versus superscript, and + versus - all stay distinct. That follows from
preserving structure, not from anything about this corpus.
FIXED -- \limits was never actually stripped. The pattern was \limits\b, and \b
does not match between `s` and `_` because `_` is a word character, so
\sum\limits_{i} kept its \limits while \limits x lost it. That is the SAME
defect X22 found in the eval scorer, in a different file: where a LaTeX command
can be followed by a subscript, \b is the wrong boundary. Now (?![A-Za-z]).
FIXED -- \dfrac and \tfrac were unhandled, so a model using ei…
[NOTICKET] fix(knowledge_extraction): the latex guard was rejecting correct transcriptions
Rifqi Hafizuddin+1: Claude Opus 5
22eb3d4
First real run of the new runner, on Sofhia's hybrid artifact, and it caught a
defect I shipped two commits ago. 4 of 6 formulas had formula_latex nulled --
Qty twice, PA and UA -- and all four rejections were FALSE.
The cause is the same identity trap, one level deeper. The model reads RENDERED
PROSE, and the document writes `Qty = (INPR Hours) / (Total Hours)`. So it
answers with a slash and the document's own parentheses, while the source
markup carries `\frac{INPR Hours}{Total Hours}` with neither. Two spellings of
one division, and the guard called them different formulas.
Three fixes, all notational:
Division canonicalises to the prose spelling: `\frac{A}{B}` -> `(A)/(B)`, so a
slash and a \frac converge instead of diverging. The old `frac(A)(B)` form was
also unsafe on its own terms -- it used parentheses as delimiters, so a literal
paren in the numerator collided with the delimiter and `\frac{(a)}{(b)}` could
never match `\frac{a}{b}`.
Redundant grouping parens collapse: `(…
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin+1: Claude Opus 5
01c76dd
Sofhia supplied the same 11 equations as MinerU transcribed them under four
backends -- 44 real strings, committed as a fixture. Running the shipped
normaliser over them reproduced her figures exactly: 11/11 for vlm/medium,
10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference.
Her independent check and this code agree on both the counts and which
equations differ, which is the point of her having run it separately.
The remaining gaps were in the command inventory, and the inventory is now
evidence rather than guesswork.
\mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets
the whole of equation 7 under pipeline, including its multiplication sign, and
\displaystyle appears in equation 4. Both are handled now, and the wrapped-x
rule generalises across every presentational wrapper rather than being \mathrm
specific.
Equation 4 needed two more, and one of them was a decision Sofhia deliberately
left open. The array ALIGNMENT SPEC arrives…
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin+1: Claude Opus 5
f81a227
Three paths leave formula_latex populated and they are not equally strong:
located in the source markup, no markup to check against, or an operator the
source cannot express. Until now the output could not tell them apart, so a
verified formula and an unverifiable one reached the expert as identical rows.
That gap is not theoretical. MinerU's pipeline backend writes multiplication as
a bare letter x, so every product formula from such an artifact is unverifiable
-- including the Production formula this contract uses as its own worked
example. Every artifact on Sofhia's machine is pipeline, because she has no GPU.
FormulaEntry gains latex_verification: verified | unverified_no_markup |
unverified_operator, null when there is no claim to characterise. Set by the
span check, never by the model -- the model cannot know which guard ran, and a
field it could write would be a claim rather than a record.
_mark_latex is a never-throw seam like the rest of validation: an entry type
without th…
[NOTICKET] fix(knowledge_extraction): compare formula_latex by notation, not identity
Rifqi Hafizuddin+1: Claude Opus 5
a998ea4
9c8a6e4 gave formula_latex the right haystack and the wrong test. Sofhia ran it
against real artifacts and found `Production = MOHH \times Qty \times ...`
rejected: the `pipeline` backend transcribes the multiplication sign as a bare
letter x, so the claim had nothing to match.
Probing further, the problem was wider than the case she hit. Against a source
that DOES carry \times, the guard also rejected a model that dropped \mathrm{},
one that used \cdot, and one that wrote idiomatic LaTeX -- 4 of 5 plausible
correct outputs. It passed only when the model mirrored MinerU's notation
character for character. The test that shipped green passed only because I
hand-wrote the fixture to mirror the source.
That is the same category error as the original defect, one level up. First the
haystack was the wrong FORMAT; then it was the wrong NOTION OF CORRECTNESS. The
prompt asks the model to transcribe INTO LaTeX, which is a translation, and
identity is not how you check a translation.
normalis…
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin+1: Claude Opus 5
9c8a6e4
The field was guarded in name only. It sat in GUARDED_FIELDS but not in
_TRANSCRIBED, so its value was never searched in the source -- only its
provenance.span was. A structurally wrong transcription with a valid prose span
passed, which is exactly the shape X21 produced: a \frac that lost its division
reads as a product and states something the document does not.
Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose
and can never contain LaTeX, so that would have nulled every formula entry and
presented as a model failure rather than a validation bug. The haystack was the
problem, not the check.
Chunk.latex is now the haystack. The parsing half has always emitted it;
extraction's adapter dropped it on the floor, which is what left the claim
unprovable. It now crosses the seam and is compared with whitespace removed
ENTIRELY rather than collapsed.
That is a different notion of equality, not a weaker one. In LaTeX, whitespace
outside \text{} is not significa…
Harry's review of the output contract asked for four things: an id on every
entity, a formula referenced rather than restated, rules related to formulas,
and a smaller schema. All four are adopted.
Two defects surfaced while adding the ids, and they matter more than the ids do.
Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank
ordinal (`#c000` over a list sorted by mention count), so one changed count
reshuffled every id after it, and `rule_id` was written by the model, so the same
rule could return a different slug next run. The product is an expert working a
review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable
ids lose which 40. Approval continuity, not linking, is why these exist.
New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never
positional, never model-supplied. New `link.py` resolves every reference after
the branches run — deterministic, no LLM, no spend — and returns dangle counts
rather than…
Completes every stage that does not depend on the seam. The pipeline now runs
end to end: artifact -> filters -> cluster -> rank -> LLM -> validate -> diff ->
frequency-sorted review queue.
extract/ client on the nano __54n quad; structured output is PROBED not assumed
(json_schema, falling back to json_object + validate-retry, recording
which applied); cached tokens read from the API, never modelled. Four
branches, one call per TERM CLUSTER. Prompts stay as files so the
fixed prefix is byte-identical and caching keeps engaging.
validate/ span check (whitespace-only normalisation, never repairs, unlocatable
provenance rejects every guarded field), escalation, conflict
detection by token overlap — explainable to the reviewer, and it
never picks a winner.
diff/ new / duplicate / conflicting against an explicitly supplied active
glossary, so the duplicate and conflicting paths are reachable. The
…
