HK-UTM-LLM / markdown
GordonUK's picture
Create markdown
b7ab33f verified
Raw
History Blame Contribute Delete
795 Bytes
# regx — Regulatory Ingest + Chunk (starter)
Minimal toolkit to:
- fetch official regulatory artifacts (HTML/PDF) with provenance
- parse into a hierarchical Node tree
- normalize and chunk into RAG‑ready units
- tag with lightweight concepts
## Quickstart
```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# 1) Fetch a page/PDF to data/ and write a manifest
regx fetch-url "https://example.com/your-source.html" --out data
# 2) Parse to a structured Node tree JSON
regx parse-structured data/your-source.html.json --out out/nodes.json
# 3) Chunk the structure to JSONL
regx chunk out/nodes.json --out out/chunks.jsonl --max-tokens 1024
# 4) Auto-tag concepts and write a mapping file
regx map-concepts out/chunks.jsonl --out out/mappings.jsonl