[NOTICKET] docs(knowledge): v3 delta in the workstream docs
Rifqi Hafizuddin+1: Claude Opus 5
58e2002
Completes the doc sync for `172d8dc`, alongside `6a9fb78`.
- TODO: new X26 for the v3 reshape; X24 annotated as superseded for the domain
branch. Numbered X26 rather than X25 because X25 ("baselines expire") already
exists - task ids are cited from other docs and must not be reused.
- CALIBRATION: the guarded-field table read
`summary | (none - generation cannot be span-checked at all)`. That was a
principle, not just a fact, and a future change would reasonably have built on
it. It now reads `title, purpose_verbatim`, with the reason the exception
disappeared: the branch was asked to LOCATE rather than SUMMARISE, so there is
something to check against.
- CONTEXT: the review-stage mermaid node "Brief Context" -> "Domain Context",
and the per-branch model-tier principle amended - Summary was classed with
Rule as reasoning/generation needing a larger tier "because a plausible
summary cannot be span-checked". That premise no longer holds.
This closes the X26 referenc…
The last two DomainContext v3 fields, both deterministic. No LLM, no prompt
change, no spend.
subdomains
Rolled up from the glossary entries' own `subdomain_tags`, never asked of the
model. Each tag was already chosen once, per term, with that term's evidence
in front of the model; a second document-level classification would be the
same judgement made with less context and no way to check it.
Ordered by tag frequency descending with an ALPHABETICAL TIEBREAK, which is
load-bearing rather than tidy: on the reference document `production` and
`equipment` both score 5, and the entries list them in differing order, so
without the tiebreak two runs of one document would disagree.
n_terms / n_formulas / n_rules
Counts of the other three artifacts. Cheap orientation for a reviewer
deciding whether a queue is worth opening, and the fastest way to spot a run
that silently extracted nothing.
Both are computed in `_finalise_domain`, called at the END of extract_all rathe…
[NOTICKET] feat(knowledge-extraction): span-guard the domain branch
Rifqi Hafizuddin+1: Claude Opus 5
e2865ac
The domain branch was the pipeline's ONE unverifiable output: GUARDED_FIELDS
["summary"] was `()`, so `title` and `purpose` were unchecked model prose on
nano, the tier whose failure mode is least detectable. Every other branch had to
prove its claims against the source; this one did not.
The fix is not a bigger model. It is asking a different question:
v2 "write a short orientation brief" -> composed -> unprovable
v3 "locate the document's own statement" -> quoted -> checkable
A summary cannot be found in the source by construction. A quote can. So
`purpose` becomes `purpose_verbatim`, prompts/summary.txt is rewritten from
summarise to locate, and both text fields join GUARDED_FIELDS + _TRANSCRIBED.
Also wires validation into `_run_summary`, which never called validate_entry -
the guard table alone would have been inert. The haystack is the WHOLE document,
unlike the other branches: this one is handed every chunk, so a quote may
legitimately come from anywhere in it.
…
The two DETERMINISTIC halves of the DomainContext v3 reshape. No LLM, no spend,
no prompt change - which is exactly why they ship ahead of the rest: the parts
that need a prompt rewrite and an eval run are split off and still proposed.
outline
Derived from the artifact's heading_path, in reading order, de-duplicated.
The blocker was upstream: adapter.py was dropping heading_path on the floor,
so the internal Chunk never carried it - the same way it once dropped `latex`,
and with the same consequence, a branch inventing prose about a document whose
own structure was sitting unread. Fixed at the seam.
It is the only field on this branch carrying the strong guarantee for free:
verbatim source structure, nothing to hallucinate. On the BUMA fixture it
yields 11 entries from 29 of 31 chunks, including the verbatim "Physical of
Availability (PA)" the pipeline is required to surface rather than tidy.
key_parameters
Was list[str]; now list[KeyParameter{surface, term_id}],…
The workstream diagram has always called this box "Domain Knowledge", while
the model and its fields described a short orientation card. Rename the entity
to match the intent before anything is persisted: once an expert has approved
rows keyed on these ids, changing the `b_` prefix is an id migration that loses
the approvals - the exact failure the deterministic-id scheme exists to prevent.
BriefContext -> DomainContext
brief_id -> domain_id (prefix b_ -> d_)
ExtractionResult.brief -> .domain
brief_context.json -> domain_context.json
Deliberately unchanged:
- `Branch = "summary"` and GUARDED_FIELDS["summary"]. The branch literal is
the measurement key in CallUsage.branch / RejectedField.branch; renaming it
would break comparability with the committed eval baseline.
- prompts/summary.txt. Editing prompt text needs the eval path, and the
rewrite belongs with the v3 schema change, not with a rename.
No field, prompt or guarding change - the …
[NOTICKET] eval(knowledge): commit the BUMA fixture artifact; K9/K10 close
Rifqi Hafizuddin+1: Claude Opus 5
e5f4abe
The 2026-08-27T13:28 re-parse of STD_2026_006_MNO becomes the committed
fixture (S7/K10). Same document and content_hash as the 08-26 parse, same
parser_config, five corrected chunk boundaries — of which ::0030 is the one
that matters: it previously held only "Keterangan grafik:" and now carries the
hierarchy-level paragraph, moving Controllable / Hierarki Level Analysis from
E1 miss to hit.
One paid run on it (90 calls, 170 s): E1 0.8293 / E2 2.556x / E3 1.000, all
three above their kill lines, rule recall 0.4286 (6/14). 0 fields rejected by
the span check, rules_without_term_link 0.
Two caveats recorded beside the numbers rather than left for a reader to
rediscover: E3's precision 1.000 sits on a scoreable base of 6 (57 of 63
entries abstain, recall 0.33 against the 18 gold terms with criteria), and the
rule-recall delta is partly a denominator change — the same 6 rules matched as
in the 08:29 run, the gold base went 15 to 14 when X23 closed.
The harness now runs with data/knowled…
[NOTICKET] revert(knowledge_extraction): X20 — it measured a net -1 (X20 closed)
Rifqi Hafizuddin
21640d6
Reverts X20's three edits to rule.txt on Rifqi's decision, after the same-day
A/B measured them as a net loss.
X20 in 5/14, all 5 runs
X20 reverted 6/14, all 5 runs
Zero variance in both arms. One rule differed: GAINLOSS_QTY_ZERO_PLAN, 0% with
X20 and 100% without, because X20's closing clause -- "a statement that is
mostly symbols, = and numbers is an equation" -- cannot distinguish a
computation from a symbolic CONDITION, and that gold rule reads "Jika (Qty_P(M)
= 0, Qty_A(M) > 0), maka:". X20's own target, GAINLOSS_UNITS, fired 0% in both
arms, so the 26 Aug 6/7 was the day and not the prompt.
A conditional carve-out (jika / apabila / maka -> a rule however symbolic its
condition) was considered and declined: X20 had a measured cost and no measured
benefit, and refining it would spend further runs defending a change that has
yet to earn any. The underlying intent -- stop the rule branch duplicating the
formula branch -- stays valid for anyone who wants to attempt it…
[NOTICKET] eval(knowledge): X20 measures net -1; close X23 on expert ruling; instrument X25
Rifqi Hafizuddin
e6ed414
Three things, all from the same session.
X20 -- FIRST CONTROLLED A/B IN THIS PROJECT, AND IT SAYS X20 COSTS A RULE.
Arms alternated inside one session so drift hit both equally. Arm B reverts
X20's three edits and KEEPS X23's synthetic examples, so the only variable is
X20 itself. Rule branch only: 15 calls per run, ~35s, five rounds each.
X20 in 5/14, every one of 5 runs
X20 reverted 6/14, every one of 5 runs
Zero variance in both arms, and exactly one rule differs:
GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without.
The cause is X20's own final clause. That gold rule is a genuine conditional
whose CONDITION is written symbolically -- "Jika (Qty_P(M) = 0, Qty_A(M) > 0),
maka:" -- and X20 ends with "a statement that is mostly symbols, = and numbers
is an equation". The heuristic cannot tell a computation from a symbolic
condition, so it nulls a real rule.
X20 is also not buying what it was written to buy. GAINLOSS_UNITS, the rule it
targets, fires 0% in BOTH arms …
[NOTICKET] feat(knowledge_extraction): cue class for declarative composition (X24b cause 3)
Rifqi Hafizuddin
e6d9d29
The rule branch could not see three gold rules because the cue taxonomy had no
slot for the voice they are written in. It had conditional cues (jika, apabila,
harus) and definitional ones (adalah, merupakan). This corpus states most of its
calculation policy in a third voice -- declarative composition:
"Nilai PA ... EKUIVALEN DENGAN rata-rata tertimbang ... dengan Quantity
sebagai FAKTOR PEMBOBOT"
"Perhitungan Pty ... DIPEROLEH DARI hasil pembobotan ..."
"Gain/Loss grouping ... merupakan PENJUMLAHAN DARI Gain/Loss model unit"
Those are calculation rules by the extraction prompt's own definition -- how a
value is computed, sourced or aggregated -- and no existing cue reached any of
them. Not a tuning gap; a missing class.
Five patterns, chosen by measuring which chunks each reaches rather than by
intuition, every one load-bearing for a previously invisible gold rule.
Candidates 8 -> 15.
\bdihitung menggunakan\b was REJECTED despite matching 5 chunks: it precedes
every formu…
[NOTICKET] docs(knowledge): correct the rule-ceiling diagnosis — it is not tables
Rifqi Hafizuddin
892f387
Yesterday's note said 6 of the 9 never-firing rules live in Tabel 2.1-2.5 and
that candidate generation over tables was the next gain. Traced properly today
against the artifact, that is wrong: 5 of the 8 candidates already come from
tabular chunks, so the cue regex reaches tables fine.
Three separate causes, only one of which is anywhere near the prompt.
Two gold rules -- PTY_PRODUCTION_SOURCE and PTY_WORKING_HOURS_DEFINITION -- have
no matching text in ANY chunk. They are not extractable at all, so either the
gold was written against content the parse does not produce, or the parse drops
it. Worth knowing which; it is not an extraction problem either way.
Notably PTY_PRODUCTION_SOURCE is the rule that used to score. It was worked
example 1 in rule.txt verbatim, so the model was reproducing it from the PROMPT
rather than from the document. X23 removing that example is why it now reads
0/7, and this is the confirmation: there was never anything in the document to
find.
One call ret…
[NOTICKET] docs(knowledge-parsing): record B3 — MinerU reaches parity, and why
B3 is answered. MinerU hybrid/high through the PRODUCTION module scores E1
0.8293 (34/41) against PyMuPDF's frozen 0.8537 (35/41) — parity with the scratch
converter calibration §10 measured, now from code that ships.
§10 is EXTENDED, not corrected. Its numbers stand and its attribution was right:
the scratch converter did handle list blocks, so `Uncontrollable` really was
"present as prose but missed" in that run, exactly as recorded. What B3 adds is
that the production module had regressed against that converter, and by how much.
The residual 0.0244 is not parsing's to close, and the docs say so rather than
leaving it as an open action. `Breakdown` appears three times in the text and
sits in the SAME chunks as `Total Hours`, which the filter does find — same
chunk, same sentence, `( Total Hours - Breakdown )`, one tagged and one not. So
chunk arrangement cannot explain it. Chunk counts rule out over-splitting too:
31 chunks = 15 prose (11 headed, 9 distinct sections) + 14 tables + …
[NOTICKET] eval(knowledge): rule-branch noise band — X20 confirmed, X23 answered
Rifqi Hafizuddin
f754631
Five repeat runs at identical config, plus the two from earlier today: seven
runs of the same pipeline on the same artifact at temperature 0.
3/15 x1 (the run carrying the latex-guard defect)
4/15 x5
6/15 x1
min 3, median 4, max 6, mean 4.14
The branch is much less noisy than one pair of runs suggested. Five of seven
land on exactly 4/15. The rule from here on is: quote the median and the band,
never a single run.
Per-rule stability turned out to be the useful view, not the total.
X20 IS CONFIRMED, and by the rule it names. GAINLOSS_UNITS -- the units sentence
that shared a chunk with a Gain/Loss equation, the exact case the fix was
written for -- fires in 6 of 7 runs against 0 at the 3/15 baseline. That is a
stable +1 sitting outside the noise band, not a lucky sample. Three other rules
fire 7/7 and two fire 1/7; those two are the whole of the variance.
X23'S GOLD QUESTION IS ANSWERED. With the few-shot that forbade it removed,
WATERFALL_TWO_METHODS still f…
[NOTICKET] fix(knowledge_extraction): audit the latex normaliser for overfitting
Rifqi Hafizuddin+1: Claude Opus 5
7d6dedf
Rifqi asked whether the normalisation rules are real or tailored to one
document, before spending on repeat runs. Audited against cases this corpus does
NOT contain -- every fixture here is BUMA, so the suite could not have caught a
rule that is really a fact about one document. Two defects and three scoped
assumptions came out of it.
The properties that matter held. Different operands, precedence, operand order,
sub- versus superscript, and + versus - all stay distinct. That follows from
preserving structure, not from anything about this corpus.
FIXED -- \limits was never actually stripped. The pattern was \limits\b, and \b
does not match between `s` and `_` because `_` is a word character, so
\sum\limits_{i} kept its \limits while \limits x lost it. That is the SAME
defect X22 found in the eval scorer, in a different file: where a LaTeX command
can be followed by a subscript, \b is the wrong boundary. Now (?![A-Za-z]).
FIXED -- \dfrac and \tfrac were unhandled, so a model using ei…
[NOTICKET] fix(knowledge_extraction): close the notation rules against real backend markup
Rifqi Hafizuddin+1: Claude Opus 5
01c76dd
Sofhia supplied the same 11 equations as MinerU transcribed them under four
backends -- 44 real strings, committed as a fixture. Running the shipped
normaliser over them reproduced her figures exactly: 11/11 for vlm/medium,
10/11 for hybrid/medium, 8/11 for pipeline, against hybrid/high as reference.
Her independent check and this code agree on both the counts and which
equations differ, which is the point of her having run it separately.
The remaining gaps were in the command inventory, and the inventory is now
evidence rather than guesswork.
\mathsf and \displaystyle occur in this corpus and were missing -- \mathsf sets
the whole of equation 7 under pipeline, including its multiplication sign, and
\displaystyle appears in equation 4. Both are handled now, and the wrapped-x
rule generalises across every presentational wrapper rather than being \mathrm
specific.
Equation 4 needed two more, and one of them was a decision Sofhia deliberately
left open. The array ALIGNMENT SPEC arrives…
[NOTICKET] docs(knowledge-parsing): record that Chunk.latex is now load-bearing
Until a998ea48 this field was carried but unread — a parser change touching it
would have broken nothing. Extraction's formula guard now validates
`formula_latex` against it, so that is no longer true, and the contract has to
say so before someone tidies it in good faith.
The failure mode is the dangerous kind and gets the emphasis: reflowing or
dropping this field does not error. The guard simply has less to check against,
and claims it can no longer verify are recorded as unverifiable rather than
caught. Fewer things fail and nothing looks wrong.
The two obligations run OPPOSITE to the rule for `text`, which is why they are
stated rather than left to inference. `text` must stay verbatim because the span
check searches it literally. `latex` must stay UNCLEANED because extraction
canonicalises before comparing — pre-cleaning removes information the
canonicaliser was built to absorb. Character-spacing, `\mathrm{}` wrappers and
`~` are all expected input, not damage.
Also draws the li…
[NOTICKET] fix(knowledge_extraction): rule.txt few-shots stop contaminating the gold set (X23)
Rifqi Hafizuddin+1: Claude Opus 5
7e8614f
Two of the four worked examples were lifted from the reference corpus, and they
contaminated the measurement in opposite directions.
Example 2 used "Waterfall Analysis dapat di-review melalui dua metode" and
taught that it is NOT a rule. The gold set records that exact sentence as
WATERFALL_TWO_METHODS. The prompt was instructing the model to fail one gold
rule by construction, so the branch could not score above 14/15.
Example 1 was the inverse and had gone unnoticed: its evidence is gold rule
PTY_PRODUCTION_SOURCE verbatim (joint survey / truck count). That is a free hit
inflating the score, not capping it.
Both are now synthetic, which is the reasoning examples 3 and 4 already carried
in the prompt itself. The closing note is generalised: all four are synthetic,
so neither an answer nor a NON-answer for a real document leaks in.
The gold set was deliberately NOT touched, following the X22 precedent -- the
prompt is tuned constantly and the gold is the measuring instrument, so th…
[NOTICKET] docs(knowledge-parsing): B1 — Colab, and the two decisions it forces
Pak Ricky prefers a Colab subscription over provisioning a GPU. That is
consistent with what §4 already argued — the invoice is decided by the billing
model, not the compute — so the structure holds and only the recommendation
moves. Colab is also not a migration: every backend figure in §3 was measured
there already.
Measured against the same corpus: Colab Pro is $9.99/month for 100 compute
units and a T4 burns ~1.19 CU/hour, so the allowance is ~84 T4-hours and the
entire 16-hour worst-case backfill costs ~19 CU — a fifth of one month, five
times over-covered. Effective $0.119/GPU-hour against Azure's $0.752. The
spread between best and worst provisioning widens from ~80x to ~290x.
Azure figures deliberately STAY. Every hour-to-dollar conversion in §3 and §5
derives from that rate, so removing it would strip the tables of their basis —
and it is the fallback if the governance question below lands the wrong way.
New §4.1 records the two decisions that are not ours, because the chea…
[NOTICKET] feat(knowledge_extraction): record which guarantee formula_latex carries
Rifqi Hafizuddin+1: Claude Opus 5
f81a227
Three paths leave formula_latex populated and they are not equally strong:
located in the source markup, no markup to check against, or an operator the
source cannot express. Until now the output could not tell them apart, so a
verified formula and an unverifiable one reached the expert as identical rows.
That gap is not theoretical. MinerU's pipeline backend writes multiplication as
a bare letter x, so every product formula from such an artifact is unverifiable
-- including the Production formula this contract uses as its own worked
example. Every artifact on Sofhia's machine is pipeline, because she has no GPU.
FormulaEntry gains latex_verification: verified | unverified_no_markup |
unverified_operator, null when there is no claim to characterise. Set by the
span check, never by the model -- the model cannot know which guard ran, and a
field it could write would be a claim rather than a record.
_mark_latex is a never-throw seam like the rest of validation: an entry type
without th…
[NOTICKET] fix(knowledge_extraction): compare formula_latex by notation, not identity
Rifqi Hafizuddin+1: Claude Opus 5
a998ea4
9c8a6e4 gave formula_latex the right haystack and the wrong test. Sofhia ran it
against real artifacts and found `Production = MOHH \times Qty \times ...`
rejected: the `pipeline` backend transcribes the multiplication sign as a bare
letter x, so the claim had nothing to match.
Probing further, the problem was wider than the case she hit. Against a source
that DOES carry \times, the guard also rejected a model that dropped \mathrm{},
one that used \cdot, and one that wrote idiomatic LaTeX -- 4 of 5 plausible
correct outputs. It passed only when the model mirrored MinerU's notation
character for character. The test that shipped green passed only because I
hand-wrote the fixture to mirror the source.
That is the same category error as the original defect, one level up. First the
haystack was the wrong FORMAT; then it was the wrong NOTION OF CORRECTNESS. The
prompt asks the model to transcribe INTO LaTeX, which is a translation, and
identity is not how you check a translation.
normalis…
[NOTICKET] docs(knowledge-parsing): B1 — cost justification for the BUMA corpus
Owed from the 13 Aug executive review. Every number was already measured; this
is the write-up, plus one finding that only surfaced while doing the arithmetic.
The finding: $0.752/hour is derived from a monthly VM price ($548.96 / 730 h).
So $6.89 for the full corpus is only true if the VM is billed per hour and
deallocated afterwards. Left running — the default unless someone says
otherwise — the same 9.2 hours of work bills $548.96/month. That is an ~80x
spread for identical work, and it is the only decision in the document that
moves real money, so it leads.
Parsing is an admin-triggered offline batch and the service never imports
MinerU at runtime, so there is no reason to hold a GPU idle between runs.
Also recorded:
- Size from the heaviest document, not the average. Density spread is 2.66x
(the 9-page standard is 8.51 s/page against the 20-page handbook's 3.20),
and the dense document is the shorter one. Planning range $4.55-$12.09;
budget the $12 end, since BUM…
[NOTICKET] fix(knowledge_extraction): formula_latex is checked against Chunk.latex
Rifqi Hafizuddin+1: Claude Opus 5
9c8a6e4
The field was guarded in name only. It sat in GUARDED_FIELDS but not in
_TRANSCRIBED, so its value was never searched in the source -- only its
provenance.span was. A structurally wrong transcription with a valid prose span
passed, which is exactly the shape X21 produced: a \frac that lost its division
reads as a product and states something the document does not.
Adding the field to _TRANSCRIBED was not the fix. `text` carries rendered prose
and can never contain LaTeX, so that would have nulled every formula entry and
presented as a model failure rather than a validation bug. The haystack was the
problem, not the check.
Chunk.latex is now the haystack. The parsing half has always emitted it;
extraction's adapter dropped it on the floor, which is what left the claim
unprovable. It now crosses the seam and is compared with whitespace removed
ENTIRELY rather than collapsed.
That is a different notion of equality, not a weaker one. In LaTeX, whitespace
outside \text{} is not significa…
[NOTICKET] docs(knowledge-parsing): settle determinism; finish the English pass
Determinism was the one thing left open on the backend decision: the VLM
family is not deterministic in general, so a frozen baseline might drift and
no eval number could be trusted.
It does not drift. Two runs over the same documents, with the parse cache
moved aside so both genuinely re-parsed (from_cache: false on all four
records), produced 0 differences across 50 chunks. All four image
descriptions were present in both runs and character-for-character identical
(192, 256, 77, 146 chars) — those are the only model-written content in the
artifact and were the likeliest to wobble.
Existing baselines therefore stay valid and need no re-measurement, and
hybrid/high is final rather than provisional. Marked settled in config.py,
calibration §13, and REPO_STATUS §14.
The parsing README was still Indonesian in the how-to-use sections written
last week, while the .py files had already moved to English; it is now
English throughout. Its console example was also stale — it showed the old
I…
Harry's review of the output contract asked for four things: an id on every
entity, a formula referenced rather than restated, rules related to formulas,
and a smaller schema. All four are adopted.
Two defects surfaced while adding the ids, and they matter more than the ids do.
Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank
ordinal (`#c000` over a list sorted by mention count), so one changed count
reshuffled every id after it, and `rule_id` was written by the model, so the same
rule could return a different slug next run. The product is an expert working a
review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable
ids lose which 40. Approval continuity, not linking, is why these exist.
New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never
positional, never model-supplied. New `link.py` resolves every reference after
the branches run — deterministic, no LLM, no spend — and returns dangle counts
rather than…
[NOTICKET] docs(knowledge-parsing): record the backend measurement, and fix three stale facts
The backend decision was in the code and in a commit message, but the evidence
behind it had no home in the repo. Anyone asking "why hybrid/high" would find
the conclusion and none of the data.
**Calibration §13 — all four backends measured.** One GPU, the same 29 pages.
`vlm` and `hybrid/high` produce IDENTICAL output (100% LaTeX overlap, same image
descriptions), so `vlm` only costs more time. `pipeline` is the only backend
copying the multiplication sign as a literal letter, which fuses MOHH/Qty/PA/UA/
Pty into one token — and it is slower than hybrid/medium, so it loses twice. The
choice narrows to medium vs high, i.e. whether image descriptions are wanted:
$2.67 against $6.89 across 6,800 pages.
Also records what the descriptions actually cost in coverage. MinerU only
describes a picture it first classified into a sub_type, and the pattern held 6
of 6: no sub_type, no description. Coverage was 4 of 6, and the miss is not
explained by image quality — a plainly legible waterfall c…
Syncs the two knowledge docs to what the branch actually holds after 296cdd7.
X20 goes to in-progress, not done. The prompt change is committed; the eval
CLAUDE.md 7B requires is not run, and the row says so rather than implying the
regression is fixed. It records what the three edits are, that worked example 3
is synthetic on purpose (MTTR, absent from all 15 gold rules) so no gold answer
leaks into the prompt, and that example 1 already is PTY_PRODUCTION_SOURCE
verbatim -- pre-existing contamination the new example deliberately did not add
to.
X22 goes to done. Rifqi fixed it in abae344 on 2026-08-24 by changing the
scorer, not the gold: underscores are deleted rather than spaced, because
spacing turns qty_a into "qty a" which still does not contain qtya. Marked
RESOLVED with the reasoning rather than deleted, per house convention.
That resolution moves X20's target. The 1/15 in
v2_full_document_2026-08-24_093051.json was measured with the pre-X22 scorer,
so the number any X20 re-…
[NOTICKET] refactor(knowledge-parsing): English throughout the module
The parsing half was written in Indonesian — identifiers, docstrings, comments,
manifest keys and console output. Everything else in the repo is English, and
this module is the newest work, so it is the one to bring into line rather than
the older code around it.
Renamed across the module: parse_dokumen -> parse_document, normalisasi ->
normalise, pulihkan_batas_kata -> restore_word_boundaries, kosakata_layak ->
trustworthy_vocabulary, periksa_items -> check_items, CatatanDokumen ->
DocumentRecord, and the locals with them.
Docstrings and comments were REWRITTEN rather than translated word for word.
Several carry the reasoning behind a decision — why `\frac` is unwrapped before
the braced wrappers, why a heading never enters `text`, why the digit threshold
is 3 — and a mechanical translation would have flattened exactly the part worth
keeping.
Manifest keys move too: detik_per_halaman -> seconds_per_page, peringatan ->
warnings, jenis -> kind, galat -> error. That is a data change, …
Records what the code change added: `image` joins ChunkKind, and
`generated_description` carries MinerU's model-written description of a picture.
Two things a consumer would otherwise get wrong. Backends disagree on the label
— a figure `pipeline` calls a chart, `vlm` calls an image — so branching on one
of them silently misses half the cases. And whether descriptions exist at all
depends on the backend: never on `pipeline`, disabled outright on `hybrid`
medium, on for `hybrid` high and always for `vlm`. An artifact carrying nulls
here is not a parsing failure, it is the backend, which is one more reason
`parser_backend` has to travel with the data.
Also states why the field is kept out of `text`: that field is what the
extraction guardrail searches to prove a quote is verbatim, and generated prose
there would let a hallucination pass the check built to catch it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[NOTICKET] docs: reorganise into docs/ and give every doc a status banner
Rifqi Hafizuddin+1: Claude Opus 5
f1342ce
The root carried 14 markdown files and no way to tell which were authoritative.
That knowledge lived in CLAUDE.md §1 and in Rifqi's head, which works for us and
fails for anyone else — and one of the stale ones already has a named failure
mode ("The Stale-Doc Believer") for being implemented from.
Root now holds four: REPO_STATUS, DEV_PLAN, API_CONTRACT_BE_PYTHON, README.
Everything else moves under docs/, by git mv so history follows:
docs/knowledge/ context, TODO, calibration, parsing output contract
docs/plans/ SPINE_V2_PLAN (approved, W3 deferred), DS_TOOLS_PLAN (proposal)
docs/archive/ API_ENDPOINTS_RESTRUCTURE, CODE_REVIEW_2026-07-23, PROJECT_BRD
docs/reference/ API_CONTRACT_BE_GOLANG (the Go repo's, a copy that can drift)
Every doc gains a one-line status banner under its H1, marked <!-- doc-status -->:
LIVE / LIVE PLAN / PROPOSAL / ARCHIVED / REFERENCE. Archived ones name what
superseded them — API_ENDPOINTS_RESTRUCTURE now opens with "DO NOT IMPLEMENT FROM…
