Commit History
-
[NOTICKET] feat(knowledge-extraction): span-guard the domain branch
Rifqi Hafizuddin +1: Claude Opus 5The domain branch was the pipeline's ONE unverifiable output: GUARDED_FIELDS ["summary"] was `()`, so `title` and `purpose` were unchecked model prose on nano, the tier whose failure mode is least detectable. Every other branch had to prove its claims against the source; this one did not. The fix is not a bigger model. It is asking a different question: v2 "write a short orientation brief" -> composed -> unprovable v3 "locate the document's own statement" -> quoted -> checkable A summary cannot be found in the source by construction. A quote can. So `purpose` becomes `purpose_verbatim`, prompts/summary.txt is rewritten from summarise to locate, and both text fields join GUARDED_FIELDS + _TRANSCRIBED. Also wires validation into `_run_summary`, which never called validate_entry - the guard table alone would have been inert. The haystack is the WHOLE document, unlike the other branches: this one is handed every chunk, so a quote may legitimately come from anywhere in it. …e2865ac -
[NOTICKET] feat(knowledge-extraction): derived outline + corroborated key parameters
Rifqi Hafizuddin +1: Claude Opus 5The two DETERMINISTIC halves of the DomainContext v3 reshape. No LLM, no spend, no prompt change - which is exactly why they ship ahead of the rest: the parts that need a prompt rewrite and an eval run are split off and still proposed. outline Derived from the artifact's heading_path, in reading order, de-duplicated. The blocker was upstream: adapter.py was dropping heading_path on the floor, so the internal Chunk never carried it - the same way it once dropped `latex`, and with the same consequence, a branch inventing prose about a document whose own structure was sitting unread. Fixed at the seam. It is the only field on this branch carrying the strong guarantee for free: verbatim source structure, nothing to hallucinate. On the BUMA fixture it yields 11 entries from 29 of 31 chunks, including the verbatim "Physical of Availability (PA)" the pipeline is required to surface rather than tidy. key_parameters Was list[str]; now list[KeyParameter{surface, term_id}],…6f5d619
-
[NOTICKET] refactor(knowledge-extraction): BriefContext -> DomainContext
Rifqi Hafizuddin +1: Claude Opus 5The workstream diagram has always called this box "Domain Knowledge", while the model and its fields described a short orientation card. Rename the entity to match the intent before anything is persisted: once an expert has approved rows keyed on these ids, changing the `b_` prefix is an id migration that loses the approvals - the exact failure the deterministic-id scheme exists to prevent. BriefContext -> DomainContext brief_id -> domain_id (prefix b_ -> d_) ExtractionResult.brief -> .domain brief_context.json -> domain_context.json Deliberately unchanged: - `Branch = "summary"` and GUARDED_FIELDS["summary"]. The branch literal is the measurement key in CallUsage.branch / RejectedField.branch; renaming it would break comparability with the committed eval baseline. - prompts/summary.txt. Editing prompt text needs the eval path, and the rewrite belongs with the v3 schema change, not with a rename. No field, prompt or guarding change - the …7f47c61
-
[NOTICKET] revert(knowledge_extraction): X20 — it measured a net -1 (X20 closed)
Rifqi HafizuddinReverts X20's three edits to rule.txt on Rifqi's decision, after the same-day A/B measured them as a net loss. X20 in 5/14, all 5 runs X20 reverted 6/14, all 5 runs Zero variance in both arms. One rule differed: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without, because X20's closing clause -- "a statement that is mostly symbols, = and numbers is an equation" -- cannot distinguish a computation from a symbolic CONDITION, and that gold rule reads "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:". X20's own target, GAINLOSS_UNITS, fired 0% in both arms, so the 26 Aug 6/7 was the day and not the prompt. A conditional carve-out (jika / apabila / maka -> a rule however symbolic its condition) was considered and declined: X20 had a measured cost and no measured benefit, and refining it would spend further runs defending a change that has yet to earn any. The underlying intent -- stop the rule branch duplicating the formula branch -- stays valid for anyone who wants to attempt it…21640d6 -
[NOTICKET] eval(knowledge): X20 measures net -1; close X23 on expert ruling; instrument X25
Rifqi HafizuddinThree things, all from the same session. X20 -- FIRST CONTROLLED A/B IN THIS PROJECT, AND IT SAYS X20 COSTS A RULE. Arms alternated inside one session so drift hit both equally. Arm B reverts X20's three edits and KEEPS X23's synthetic examples, so the only variable is X20 itself. Rule branch only: 15 calls per run, ~35s, five rounds each. X20 in 5/14, every one of 5 runs X20 reverted 6/14, every one of 5 runs Zero variance in both arms, and exactly one rule differs: GAINLOSS_QTY_ZERO_PLAN, 0% with X20 and 100% without. The cause is X20's own final clause. That gold rule is a genuine conditional whose CONDITION is written symbolically -- "Jika (Qty_P(M) = 0, Qty_A(M) > 0), maka:" -- and X20 ends with "a statement that is mostly symbols, = and numbers is an equation". The heuristic cannot tell a computation from a symbolic condition, so it nulls a real rule. X20 is also not buying what it was written to buy. GAINLOSS_UNITS, the rule it targets, fires 0% in BOTH arms …e6ed414
-
[NOTICKET] fix(knowledge_extraction): rule.txt few-shots stop contaminating the gold set (X23)
Rifqi Hafizuddin +1: Claude Opus 5Two of the four worked examples were lifted from the reference corpus, and they contaminated the measurement in opposite directions. Example 2 used "Waterfall Analysis dapat di-review melalui dua metode" and taught that it is NOT a rule. The gold set records that exact sentence as WATERFALL_TWO_METHODS. The prompt was instructing the model to fail one gold rule by construction, so the branch could not score above 14/15. Example 1 was the inverse and had gone unnoticed: its evidence is gold rule PTY_PRODUCTION_SOURCE verbatim (joint survey / truck count). That is a free hit inflating the score, not capping it. Both are now synthetic, which is the reasoning examples 3 and 4 already carried in the prompt itself. The closing note is generalised: all four are synthetic, so neither an answer nor a NON-answer for a real document leaks in. The gold set was deliberately NOT touched, following the X22 precedent -- the prompt is tuned constantly and the gold is the measuring instrument, so th…7e8614f -
[NOTICKET] feat(knowledge_extraction): output schema v2 — entity ids, links, 16 fields cut
Rifqi Hafizuddin +1: Claude Fable 5Harry's review of the output contract asked for four things: an id on every entity, a formula referenced rather than restated, rules related to formulas, and a smaller schema. All four are adopted. Two defects surfaced while adding the ids, and they matter more than the ids do. Neither existing identifier survived a re-run: `cluster_id` was a frequency-rank ordinal (`#c000` over a list sorted by mention count), so one changed count reshuffled every id after it, and `rule_id` was written by the model, so the same rule could return a different slug next run. The product is an expert working a review queue — if Mas Beta approves 40 entries and a prompt is retuned, unstable ids lose which 40. Approval continuity, not linking, is why these exist. New `ids.py` mints term/formula/rule/brief ids as sha256 over content: never positional, never model-supplied. New `link.py` resolves every reference after the branches run — deterministic, no LLM, no spend — and returns dangle counts rather than…49dfe4d -
[NOTICKET] fix(knowledge_extraction): rule.txt excludes bare equations (X20)
Rifqi Hafizuddin +1: Claude Opus 5The rule branch was spending calls re-extracting equations the formula branch already captures, and dropping the operational prose it exists for. 4 of 8 v2 rule statements were bare equations; in the prototype 0 of 7 were. The cause is not the model. Before rendering, formulas reached it as mangled character-spaced LaTeX and did not read as statements. Rendered to clean prose they read exactly like prescriptive rules, and rule.txt never excluded them -- it defined a rule as "a statement that constrains or directs behaviour", which a formula arguably is. Rendering gained +7.3 recall points on the term filter, so the fix belongs in the prompt, not in undoing the rendering. Three edits. A rule is now PROSE, with a paragraph saying an equation belongs to the formula branch and that "calculation policy" means prose stating how, when or on what basis a calculation applies. A new requirement 3 sets statement to null when the only prescriptive content is a formula. And -- the clause that mat…296cdd7
-
[NOTICKET] fix(knowledge_extraction): show the model the section heading
Rifqi Hafizuddin +1: Claude Fable 5The evidence block sent to the model carried only `chunk.text`, so the section heading never reached it. Two consequences, both silent: - Indonesian standards name the term in the heading and open the body with the definition — "2.1.3 Physical of Availability (PA)" / "Adalah ketersediaan fisik…" — and the body often never repeats the term. The model was being asked to define a term its evidence never named. - It broke the invariant that what the model reads is exactly what the span check searches. `validate.evidence_text` already composed heading + text, so a field quoting a section title would have been rejected as unlocatable — and the model could not have quoted one anyway. Found while verifying the claim in the flow illustration: "Physical of Availability" appeared in the prompt only because the prompt file's own worked example contains that string, never from the document being processed. The deterministic `source_wording` fallback masked the effect, which is why th…a7bdeae
-
[NOTICKET] feat(knowledge_extraction): paid extraction stage + validate, diff, queue
Rifqi Hafizuddin +1: Claude Fable 5Completes every stage that does not depend on the seam. The pipeline now runs end to end: artifact -> filters -> cluster -> rank -> LLM -> validate -> diff -> frequency-sorted review queue. extract/ client on the nano __54n quad; structured output is PROBED not assumed (json_schema, falling back to json_object + validate-retry, recording which applied); cached tokens read from the API, never modelled. Four branches, one call per TERM CLUSTER. Prompts stay as files so the fixed prefix is byte-identical and caching keeps engaging. validate/ span check (whitespace-only normalisation, never repairs, unlocatable provenance rejects every guarded field), escalation, conflict detection by token overlap — explainable to the reviewer, and it never picks a winner. diff/ new / duplicate / conflicting against an explicitly supplied active glossary, so the duplicate and conflicting paths are reachable. The …ab5ea78