Add README.md

README.md

@@ -1,9 +1,367 @@
 ---
-library_name: mlx
 license: apache-2.0
-license_link: https://huggingface.co/Qwen/Qwen3-4B/blob/main/LICENSE
+language:
+  - en
+  - fr
+  - de
+  - ro
+library_name: transformers
 pipeline_tag: text-generation
-base_model: mlx-community/Qwen3-4B-4bit
+base_model: Qwen/Qwen3-4B
 tags:
-- mlx
+  - lora
+  - regulatory
+  - compliance
+  - ontology-extraction
+  - information-extraction
+  - mlx
+  - gguf
+  - flowx
+model-index:
+  - name: semantic-mapper
+    results:
+      - task:
+          type: text-generation
+          name: Regulatory ontology extraction (held-out, document-disjoint, EN/FR/DE/RO)
+        metrics:
+          - type: f1
+            name: concepts F1 (controlled 252-concept vocab)
+            value: 0.54
+          - type: f1
+            name: concepts F1 (fuzzy / semantic match)
+            value: 0.58
+          - type: f1
+            name: domain_tags F1 (free-text)
+            value: 0.55
+          - type: accuracy
+            name: entities field-accuracy
+            value: 0.53
+          - type: accuracy
+            name: JSON validity
+            value: 1.0
+          - type: accuracy
+            name: all-3-facets present
+            value: 1.0
 ---
+
+## Inference contract
+
+Running this model correctly requires its **frozen inference contract**: the exact
+system prompt, user-turn format, decode settings, and output JSON schema it was
+trained against. See [`inference_contract/`](./inference_contract):
+
+- [`INFERENCE.md`](./inference_contract/INFERENCE.md): wiring guide: system prompt, user turn (`Extract ontology from this chunk:\n\nCHUNK:\n<text>`), decode settings (`enable_thinking=False`, greedy/temp 0, `max_new_tokens` ~1024), and the note that `semantic.concepts` is constrained to the **252-concept controlled vocabulary** (`concept_taxonomy.yaml`, shipped at repo root).
+- [`prompt_mapper_sys_v1.txt`](./inference_contract/prompt_mapper_sys_v1.txt): the system prompt, verbatim.
+- [`schema_mapper_v1.json`](./inference_contract/schema_mapper_v1.json): output JSON Schema (structural / semantic / governance).
+
+Prompt version `mapper_sys_v1`. Do not edit the prompt/schema; the weights are trained against them.
+
+# FlowX Semantic Mapper (4B): the regulation-to-ontology tagger
+
+**Reads one chunk of regulatory or legal text and emits a structured ontology JSON: where the clause sits in its source, what it is about, and how it maps to policy and escalation.** Languages: EN, FR, DE, RO. Domains: banking, insurance, logistics, labor.
+
+Given a single clause (a subsection of a state insurance code, an EU regulation article, a labor-code paragraph, an ADR packing rule), the Mapper returns three facets: a **structural** locator (source id, document type, and the full hierarchy path), a **semantic** payload (free-text `domain_tags`, controlled-vocabulary `concepts`, and an actor/action/object/constraint entity triple), and a **governance** payload (`PDP.<domain>.<rule>` policy references plus a single `if ... THEN escalate` guard clause).
+
+The product insight that shapes everything: **an ontology field is only useful if it is consistent enough to join on.** The first version emitted `concepts` as open free text and produced 1062 unique ids, 88% of them singletons: unlearnable for the model and unmeasurable for us (concepts F1 sat at 0.24). Collapsing that to a **252-concept controlled taxonomy** (`concept_taxonomy.yaml`) is the central design choice of this release: it made `concepts` both learnable and scoreable (F1 0.24 → 0.54), and gave downstream policy code a stable key to join on. `domain_tags` stay free-text on purpose (for recall and human browsing) and are honestly noisier as a result.
+
+The Mapper is the first stage of a pipeline, not a standalone product: **Mapper tags a chunk → a policy layer → the [`flowxai/sentinel-gate`](https://huggingface.co/flowxai/sentinel-gate) escalation model decides DECIDE vs ESCALATE.** It is part of the FlowX on-device model family alongside [`flowxai/sentinel-gate`](https://huggingface.co/flowxai/sentinel-gate), [`flowxai/caveat`](https://huggingface.co/flowxai/caveat), and the consumer scam detectors [`flowxai/scam-guard-qwen06b`](https://huggingface.co/flowxai/scam-guard-qwen06b) / [`flowxai/scam-guard-qwen17b`](https://huggingface.co/flowxai/scam-guard-qwen17b).
+
+LoRA-fine-tuned from Apache-2.0 **`Qwen/Qwen3-4B`**. Formats: fp16 safetensors (transformers / CUDA / vLLM), **MLX-quantized** int4 + int8 (Apple Silicon), and **GGUF** `Q8_0` + `Q4_K_M` (llama.cpp / CPU / Ollama).
+
+---
+
+## How do I use it?
+
+Two copy-pasteable ways to turn a regulatory chunk into ontology JSON, plus a note.
+
+Real example input (a real clause from the California Insurance Code, section 790.03(h)(2)):
+
+```
+Failing to acknowledge and act reasonably promptly upon communications with respect
+to claims arising under insurance policies.
+```
+
+The user turn wraps it in the fixed template (see the inference contract):
+
+```
+Extract ontology from this chunk:
+
+CHUNK:
+Failing to acknowledge and act reasonably promptly upon communications with respect to claims arising under insurance policies.
+```
+
+### (a) transformers / CUDA (fp16 safetensors)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+SYSTEM_PROMPT = (
+    "You are a legal and regulatory ontology extractor.\n"
+    "Extract structured tags from document chunks. Output ONLY valid JSON."
+)
+
+tok = AutoTokenizer.from_pretrained("flowxai/semantic-mapper")
+model = AutoModelForCausalLM.from_pretrained("flowxai/semantic-mapper", device_map="cuda")
+
+chunk = ("Failing to acknowledge and act reasonably promptly upon communications "
+         "with respect to claims arising under insurance policies.")
+user = f"Extract ontology from this chunk:\n\nCHUNK:\n{chunk}"
+
+prompt = tok.apply_chat_template(
+    [{"role": "system", "content": SYSTEM_PROMPT},
+     {"role": "user", "content": user}],
+    add_generation_prompt=True, enable_thinking=False, tokenize=False,
+)
+ids = tok(prompt, return_tensors="pt").to(model.device)
+out = model.generate(**ids, max_new_tokens=1024, do_sample=False, temperature=0.0)
+raw = tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)
+```
+
+### (b) MLX (Apple Silicon)
+
+Off the MLX-quantized weights (int4 / int8):
+
+```bash
+mlx_lm.generate --model flowxai/semantic-mapper-mlx-int4 --temp 0 --max-tokens 1024 \
+  --system-prompt "$(printf 'You are a legal and regulatory ontology extractor.\nExtract structured tags from document chunks. Output ONLY valid JSON.')" \
+  --prompt "$(printf 'Extract ontology from this chunk:\n\nCHUNK:\nFailing to acknowledge and act reasonably promptly upon communications with respect to claims arising under insurance policies.')"
+```
+
+```python
+from mlx_lm import load, generate
+
+SYSTEM_PROMPT = (
+    "You are a legal and regulatory ontology extractor.\n"
+    "Extract structured tags from document chunks. Output ONLY valid JSON."
+)
+
+model, tok = load("flowxai/semantic-mapper-mlx-int4")
+chunk = ("Failing to acknowledge and act reasonably promptly upon communications "
+         "with respect to claims arising under insurance policies.")
+user = f"Extract ontology from this chunk:\n\nCHUNK:\n{chunk}"
+prompt = tok.apply_chat_template(
+    [{"role": "system", "content": SYSTEM_PROMPT},
+     {"role": "user", "content": user}],
+    add_generation_prompt=True, enable_thinking=False,
+)
+raw = generate(model, tok, prompt=prompt, max_tokens=1024, verbose=False)
+```
+
+Either backend returns the **same JSON** for this chunk (this is the real held-out target for the example above):
+
+```json
+{
+  "structural": {
+    "source_id": "US_CA_INS_790_03_h_2",
+    "hierarchy": [
+      "state_code", "california_insurance_code",
+      "section", "790.03",
+      "subdivision", "h",
+      "paragraph", "2"
+    ],
+    "document_type": "state_insurance_code"
+  },
+  "semantic": {
+    "domain_tags": ["insurance_regulation", "claims", "unfair_claims_practices", "settlement"],
+    "concepts": ["claim_acknowledgement", "prompt_communication", "settlement_timeline"],
+    "entities": {
+      "actor": "insurer",
+      "action": "acknowledge and act reasonably promptly upon communications",
+      "object": "claim_communication",
+      "constraint": { "condition": "reasonably_prompt" }
+    }
+  },
+  "governance": {
+    "policy_references": ["PDP.insurance.claims_settlement", "PDP.insurance.unfair_practices"],
+    "escalation_trigger": "if !claim_communication_acknowledged_promptly THEN escalate"
+  }
+}
+```
+
+### (c) A short note on decoding
+
+`enable_thinking=False` is not optional. Qwen3-4B is a thinking model, but this
+adapter was trained on pure JSON with no reasoning block, so the default template
+yields empty output. Decode greedily (temperature 0), one chunk per call, and cap
+at ~1024 new tokens. GGUF (`Q8_0` / `Q4_K_M`) runs the same contract on
+llama.cpp / Ollama. Full wiring is in [`inference_contract/INFERENCE.md`](./inference_contract/INFERENCE.md).
+
+---
+
+## How it works
+
+```
+regulatory
+text chunk  ---> FlowX Semantic Mapper 4B ---> structural  (source_id, hierarchy, document_type)
+                                          |
+                                          +--> semantic    (domain_tags, concepts, entities)
+                                          |
+                                          +--> governance  (policy_references, escalation_trigger)
+```
+
+Pipeline line:
+
+```
+chunk --> Semantic Mapper (this model) --> policy layer --> Sentinel Gate --> DECIDE | ESCALATE
+```
+
+Under the hood the Mapper is: `tokenizer → Qwen3-4B (LoRA fine-tuned) → greedy JSON
+decode (thinking disabled) → parse against schema_mapper_v1.json`. The
+`semantic.concepts` field targets the 252-concept controlled taxonomy
+(`concept_taxonomy.yaml`), so downstream code can filter out-of-vocabulary ids and
+join the rest straight into the policy layer.
+
+---
+
+## Output schema
+
+A single JSON object with three facets (full schema:
+[`schema_mapper_v1.json`](./inference_contract/schema_mapper_v1.json)):
+
+**`structural`**: where the clause lives.
+- `source_id`: stable id for the chunk, usually an uppercased normalization of the citation path (e.g. `US_CA_INS_790_03_h_2`).
+- `hierarchy`: ordered flat array of alternating `[level, value]` pairs from the outermost container down to the leaf.
+- `document_type`: snake_case document class (e.g. `state_insurance_code`, `eu_regulation`, `labor_code`, `adr_agreement`).
+
+**`semantic`**: what the clause is about.
+- `domain_tags`: free snake_case topic tags (**open vocabulary**; noisier by design, tuned for recall/browsing).
+- `concepts`: concept ids from the **252-concept controlled taxonomy** (`concept_taxonomy.yaml`). Treat any id not in the taxonomy as out-of-vocabulary.
+- `entities`: `{actor, action, object, constraint}`: who the obligation falls on, what must/may be done, on what, and a free key/value `constraint` map of qualifying conditions (e.g. `{"condition":"reasonably_prompt"}`).
+
+**`governance`**: how it maps to policy and escalation.
+- `policy_references`: ids in the form `PDP.<domain>.<rule>` (e.g. `PDP.insurance.claims_settlement`).
+- `escalation_trigger`: a single guard clause `if <condition> THEN escalate`, consumed by the Sentinel Gate.
+
+The controlled taxonomy holds **252 canonical concepts** across 6 categories
+(money, rights_waived, time_renewal, lease, insurance, data), each with an `id`, a
+`definition`, and a `primary_domain`. It ships as `concept_taxonomy.yaml` at the
+repo root.
+
+---
+
+## Evaluation
+
+Held-out set: **n = 112**, **document-disjoint** from training, multilingual
+(EN / FR / DE / RO). Read the honest numbers first, then the caveat.
+
+| Metric | Value |
+| --- | --- |
+| JSON validity | **1.00** |
+| all-3-facets present | **1.00** |
+| `domain_tags` F1 (free-text) | 0.55 |
+| `concepts` F1 (controlled 252-concept vocab) | **0.54** |
+| `concepts` F1 (fuzzy / semantic match) | 0.58 |
+| `entities` field-accuracy | 0.53 |
+
+**The headline improvement, quantified.** Adopting the controlled 252-concept
+vocabulary plus a real-regulation data expansion moved `concepts` F1 from **0.24**
+(open free-text vocab) to **0.54**, and `entities` from **unmeasured** (the facet
+was empty in the first corpus) to **0.53**. The format facets are saturated: JSON
+validity and all-three-facets-present are both 1.00, so the model reliably produces
+a parseable, structurally-complete object every time.
+
+**Honest reading.**
+- `concepts` is **partial** at ~0.54: it surfaces many of the right controlled ids, not all of them. Use it as a strong signal, not a complete tagging.
+- `domain_tags` are **free-text** and therefore less consistent than the controlled `concepts` (this is the intended trade: recall/browsability over join-stability).
+- `entities` field-accuracy of 0.53 reflects a genuinely hard four-slot extraction (actor/action/object/constraint), now measurable where before it was not.
+- **EN is strongest.** FR, DE, RO are supported and evaluated, but a bit weaker.
+
+> **Home-field caveat (read before citing these numbers).** The held-out set is
+> **document-disjoint** (no shared source document leaks between train and test),
+> which is the right discipline. But the annotations across train and test share a
+> **lineage**: the same controlled taxonomy and the same frontier-model-assisted
+> annotation style. The scores above measure how well the model reproduces *that
+> annotation convention* on unseen documents, which is exactly the deployment task,
+> but they are **not** a claim of agreement with an independent human legal
+> annotator. Treat 0.54 concepts F1 as "reproduces the FlowX taxonomy convention on
+> held-out documents at ~0.54", not "a legal-grade ontology at 0.54".
+
+---
+
+## Formats
+
+| Format | Path | Runtime | Notes |
+| --- | --- | --- | --- |
+| fp16 safetensors | repo root | transformers / CUDA / vLLM | full precision |
+| MLX int4 | `mlx-int4/` | Apple Silicon (MLX) | smallest Apple footprint |
+| MLX int8 | `mlx-int8/` | Apple Silicon (MLX) | higher-fidelity Apple path |
+| GGUF `Q8_0` | `gguf/` | llama.cpp / CPU / Ollama | recommended GGUF quant |
+| GGUF `Q4_K_M` | `gguf/` | llama.cpp / CPU / Ollama | smallest GGUF |
+
+All formats run the same frozen inference contract (system prompt, user-turn
+template, `enable_thinking=False`, greedy decode).
+
+---
+
+## Intended use & limitations
+
+**Intended use.** A **first-stage extractor** in a regulatory-compliance pipeline:
+it turns a clause of legislation into a structured ontology record that a policy
+layer and the [`flowxai/sentinel-gate`](https://huggingface.co/flowxai/sentinel-gate)
+escalation model can act on. It is meant to run over a knowledge base of regulatory
+chunks and produce consistent, joinable tags, with a human reviewing anything the
+Sentinel routes for escalation.
+
+**Out of scope & limitations.**
+- **Not legal advice.** The Mapper labels the structure and topic of a text; it does not interpret the law, give a legal opinion, or determine compliance. A human with legal responsibility owns any decision.
+- **Extraction is partial.** `concepts` F1 ~0.54 and `entities` accuracy ~0.53: expect misses and the occasional wrong slot. Do not treat the output as an exhaustive or authoritative tagging.
+- **Controlled vocab, not free-form.** `semantic.concepts` targets the 252-concept taxonomy. Filter out-of-vocabulary ids before joining. `domain_tags` are free-text and inherently noisier.
+- **Language skew.** EN is strongest; FR / DE / RO are supported but weaker.
+- **One chunk at a time.** Trained on single-chunk turns; batching multiple clauses into one user turn degrades output.
+- **Domain coverage.** Banking, insurance, logistics, labor. Text outside these domains is out of distribution.
+- **Annotation-lineage caveat** (see Evaluation): scores measure reproduction of the FlowX annotation convention, not agreement with an independent legal annotator.
+
+---
+
+## Training data
+
+- **763 records total**, split **651 train / 112 held-out**, **document-disjoint** (no source document appears in both splits).
+- **Real, verbatim regulatory / legislative text** from official public sources: US eCFR & state codes, EU EUR-Lex, France Légifrance, Germany Gesetze im Internet, Romania legislatie.just.ro, and UNECE/ADR. Every record carries a source URL. Official legislation is reproduced with source acknowledgement.
+- Annotated to the **controlled 252-concept taxonomy** with frontier-model assistance, then validated: concepts checked in-vocabulary, entities checked populated, source URL required per record.
+- Coverage spans four domains (banking / insurance / logistics / labor) and four languages (EN / FR / DE / RO).
+
+The controlled taxonomy (`concept_taxonomy.yaml`) is the pivot of this dataset:
+open free-text concepts (1062 unique, 88% singletons) were collapsed to 252
+canonical ids, which is what made the `concepts` facet learnable and measurable.
+
+---
+
+## Fine-tuning
+
+**LoRA** on **`Qwen/Qwen3-4B`** with **MLX-LM on Apple Silicon**.
+
+| Hyperparameter | Value |
+| --- | --- |
+| method | LoRA |
+| rank | 32 |
+| scale (alpha) | 16 |
+| dropout | 0.05 |
+| layers | `num_layers -1` (all) |
+| epochs | ~3 |
+| LR schedule | cosine 1e-4 → 1e-5 |
+| max_seq_length | 2048 |
+| seed | 42 |
+| thinking | disabled |
+
+The assistant turn in training is the exact ontology JSON the model must emit (no
+reasoning block), which is why inference must set `enable_thinking=False`.
+
+---
+
+## Not legal advice
+
+This model extracts structure and topic tags from legal/regulatory text. It does
+not provide legal advice, does not interpret the law, and does not determine
+compliance. Its output is an aid to a compliance pipeline under human ownership,
+not a substitute for a qualified professional.
+
+## Links
+
+- **Sibling (escalation gate):** [`flowxai/sentinel-gate`](https://huggingface.co/flowxai/sentinel-gate): consumes a case (with the Mapper's governance facet) and decides DECIDE vs ESCALATE.
+- **FlowX on-device family:** [`flowxai/caveat`](https://huggingface.co/flowxai/caveat), [`flowxai/scam-guard-qwen06b`](https://huggingface.co/flowxai/scam-guard-qwen06b), [`flowxai/scam-guard-qwen17b`](https://huggingface.co/flowxai/scam-guard-qwen17b).
+
+## License
+
+Apache-2.0 (weights and code). Base model `Qwen/Qwen3-4B` is Apache-2.0. Copyright
+2026 FlowX.AI; see the `NOTICE` file. Author: Bogdan Răduță, Head of Research,
+FlowX.AI.