Rifqi Hafizuddin
[NOTICKET] feat(knowledge_extraction): paid extraction stage + validate, diff, queue
ab5ea78
Raw
History Blame
1.75 kB
"""Knowledge-extraction pipeline: parsed document -> candidate knowledge entries.
Distinct from `src/knowledge/`, which is the existing OCR -> chunk -> pgvector
ingestion path for unstructured RAG. This package does not touch it.
Stage order, and which stages cost money:
adapter seam artifact -> internal Chunk free
filters cue / legend / span NER -> mentions free (CPU)
cluster normalise + cluster mentions free
rank evidence scoring, top-K selection free
extract one LLM call per TERM CLUSTER PAID
validate verbatim span check, escalation free
diff new / duplicate / conflicting free
queue frequency-sorted review queue free
`service.py` is the facade; `cli.py` is the operator entry point.
Design rationale: knowledge_pipeline_context.md
Calibrated constants and why: KNOWLEDGE_PIPELINE_CALIBRATION.md
"""
from .adapter import parsed_doc_from_artifact
from .models import (
AbbrevPair,
BriefContext,
CallUsage,
Chunk,
ClusterResult,
FilterResult,
FormulaEntry,
GlossaryEntry,
Mention,
ParsedDoc,
Provenance,
RuleCandidate,
RuleEntry,
TermCluster,
)
from .service import ExtractionResult, build_clusters, estimate_cost, extract_all, run_filters
__all__ = [
"AbbrevPair",
"BriefContext",
"CallUsage",
"Chunk",
"ClusterResult",
"ExtractionResult",
"FilterResult",
"FormulaEntry",
"GlossaryEntry",
"Mention",
"ParsedDoc",
"Provenance",
"RuleCandidate",
"RuleEntry",
"TermCluster",
"build_clusters",
"estimate_cost",
"extract_all",
"parsed_doc_from_artifact",
"run_filters",
]