Add model card, benchmark charts, technical whitepaper

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+graphs/accuracy_across_iterations.png filter=lfs diff=lfs merge=lfs -text
+graphs/loss_convergence.png filter=lfs diff=lfs merge=lfs -text
+graphs/throughput_vs_vram.png filter=lfs diff=lfs merge=lfs -text
+SchemaForge_Whitepaper.pdf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,277 @@
+---
+language:
+- en
+license: apache-2.0
+tags:
+- distillation
+- knowledge-distillation
+- json-extraction
+- structured-output
+- information-extraction
+- gemma-4
+- minicpm5
+- edge-ai
+- schemaforge
+pipeline_tag: text-generation
+base_model: openbmb/MiniCPM5-1B
+library_name: transformers
+metrics:
+- accuracy
+- f1
+- throughput
+---
+
+# SchemaForge-1B — JSON Extractor
+
+**A 1.08B-parameter edge SLM distilled from Gemma-4 for zero-shot enterprise JSON extraction.**
+
+SchemaForge-1B converts unstructured business documents — invoices, bills of lading, requisitions, receipts — into strongly-typed, schema-conformant JSON. It was distilled from **`google/gemma-4-31B`** and **`google/gemma-4-E4B-it`** into **`openbmb/MiniCPM5-1B`** using a multi-task objective combining hard cross-entropy with temperature-scaled, log-space soft-logit KL divergence ($\alpha = 0.5$, $\tau = 2.0$), trained on an NVIDIA RTX PRO 6000 Blackwell Edition (96 GB).
+
+| | 31B Teacher | **SchemaForge-1B** |
+|---|---|---|
+| In-domain JSON syntax error rate *(n = 5 docs)* | 0.0 % | **0.0 %** |
+| In-domain extraction F1 *(n = 5 docs)* | 1.000 | **1.000** |
+| Zero-shot validity (`suneeldk/text-json`) | — | **70.0 %** |
+| Throughput | 12.40 tok/s | **61.91 – 76.27 tok/s** |
+| Peak VRAM | ≈38.5 GB | **≈2.4 GB** |
+| Workers per 96 GB GPU | 2 | **36** |
+
+**16.0× smaller · 5.0× faster · ~110× aggregate system throughput**
+
+---
+
+## ⚠️ Read This First: The Prompt Template Is Not Optional
+
+This model was distilled on **one exact prompt template**. Because it is a 1.08B student trained on a narrow task, it binds its behavior to the **literal surface form** of that prefix. In our experiments, changing only the instruction header dropped zero-shot validity from **70.0 % to 0.0 %** — worse than the *untrained* base model.
+
+Use this string, byte for byte:
+
+```python
+TEMPLATE = "Extract structured JSON from the text:\n{doc}\nJSON Output:"
+```
+
+Do not wrap it in chat tokens. Do not prepend a system persona. Do not add a trailing newline. Treat it as a versioned API contract.
+
+---
+
+## 📊 Benchmark Evidence
+
+### 1. Throughput and VRAM
+
+![Figure 1: Inference throughput vs VRAM footprint](https://huggingface.co/arrochi112/SchemaForge-1B-JSON-Extractor/resolve/main/graphs/throughput_vs_vram.png)
+
+*Figure 1: **5.0× throughput speedup** (61.91 vs. 12.40 tok/s, matched harness) and **16.0× VRAM reduction** (≈2.4 GB vs. ≈38.5 GB), measured on identical hardware.*
+
+### 2. Zero-shot accuracy across distillation iterations
+
+![Figure 2: Zero-shot JSON accuracy across iterations](https://huggingface.co/arrochi112/SchemaForge-1B-JSON-Extractor/resolve/main/graphs/accuracy_across_iterations.png)
+
+*Figure 2: Validity on `suneeldk/text-json`. Iterations 1 and 3 differ from the winning Iteration 2 **only in prompt header** — and both collapse to 0.0 %.*
+
+### 3. Training convergence
+
+![Figure 3: Training loss convergence](https://huggingface.co/arrochi112/SchemaForge-1B-JSON-Extractor/resolve/main/graphs/loss_convergence.png)
+
+*Figure 3: Loss over 3 epochs, Gemma-4-31B teacher. Iteration 2 (released): 9,132.9 → 6,962.3 → 6,612.7 (−27.6 %). Summed losses — comparable within a run, not across runs.*
+
+---
+
+## Quickstart
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "arrochi112/SchemaForge-1B-JSON-Extractor"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
+).to("cuda" if torch.cuda.is_available() else "cpu")
+# No trust_remote_code needed — MiniCPM5-1B is a stock LlamaForCausalLM.
+
+# CANONICAL TEMPLATE — do not modify
+prompt = (
+    "Extract structured JSON from the text:\n"
+    "INVOICE #INV-1001. Vendor: Acme Supply Co. Date: 2026-04-10. "
+    "Subtotal: $480.00. Tax (8%): $38.40. Total: $518.40.\n"
+    "JSON Output:"
+)
+
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.no_grad():
+    outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+
+print(tokenizer.decode(outputs[0][inputs["input_ids"].size(1):],
+                       skip_special_tokens=True))
+```
+
+Expected:
+
+```json
+{
+  "invoice_number": "INV-1001",
+  "vendor_name": "Acme Supply Co",
+  "invoice_date": "2026-04-10",
+  "subtotal": 480.00,
+  "tax": 38.40,
+  "grand_total": 518.40
+}
+```
+
+---
+
+## Production Serving (vLLM)
+
+```python
+from vllm import LLM, SamplingParams
+
+llm = LLM(
+    model="arrochi112/SchemaForge-1B-JSON-Extractor",
+    dtype="bfloat16",
+    gpu_memory_utilization=0.90,
+    max_model_len=2048,
+    max_num_seqs=36,               # 36 workers fit in 96 GB at 2.4 GB each
+)
+
+sampling_params = SamplingParams(temperature=0.0, max_tokens=256)
+
+TEMPLATE = "Extract structured JSON from the text:\n{doc}\nJSON Output:"
+docs = [
+    "Invoice #INV-881, Vendor: Globex Corp, Date: 2026-08-03, Total: $450.00",
+    "Invoice #INV-882, Vendor: Initech LLC, Date: 2026-08-04, Total: $1200.00",
+]
+
+for out in llm.generate([TEMPLATE.format(doc=d) for d in docs], sampling_params):
+    print(out.outputs[0].text)
+```
+
+### Recommended: layer schema-constrained decoding
+
+Distillation supplies *semantics*; FSM-guided decoding guarantees *syntax*. Run both.
+
+```python
+from pydantic import BaseModel
+from vllm.sampling_params import GuidedDecodingParams
+
+class Invoice(BaseModel):
+    invoice_number: str
+    vendor_name: str
+    invoice_date: str
+    subtotal: float
+    tax: float
+    grand_total: float
+
+sampling_params = SamplingParams(
+    temperature=0.0,
+    max_tokens=256,
+    guided_decoding=GuidedDecodingParams(json=Invoice.model_json_schema()),
+)
+```
+
+---
+
+## Evaluation
+
+### Five-domain enterprise suite (in-domain, n = 5 documents)
+
+| Domain | Document type | Base MiniCPM5-1B | **SchemaForge-1B** | F1 | Throughput |
+|---|---|---|---|---|---|
+| BMK-01 Finance | Tax invoices | 65.8 % | **100.0 %** | 1.000 | 61.91 tok/s |
+| BMK-02 Supply chain | Bills of lading | 67.1 % | **100.0 %** | 1.000 | 62.40 tok/s |
+| BMK-03 IT hardware | Procurement bills | 64.2 % | **100.0 %** | 1.000 | 61.80 tok/s |
+| BMK-04 Biomedical | Lab requisitions | 66.5 % | **100.0 %** | 1.000 | 62.15 tok/s |
+| BMK-05 Cloud ops | Billing records | 65.4 % | **100.0 %** | 1.000 | 62.05 tok/s |
+
+### Model comparison
+
+| Variant | Teacher | JSON error rate | F1 | Throughput | VRAM |
+|---|---|---|---|---|---|
+| Base MiniCPM5-1B | none | 34.2 % | 0.612 | 62.00 tok/s | ≈2.4 GB |
+| **SchemaForge-1B** | `gemma-4-E4B-it` | **0.0 %** | **1.000** | **61.91 tok/s** | **≈2.4 GB** |
+| **SchemaForge-1B** | `gemma-4-31B` | **0.0 %** | **1.000** | 56.12 tok/s | **≈2.4 GB** |
+| Gemma-4-31B | reference | 0.0 % | 1.000 | 12.40 tok/s | ≈38.5 GB |
+
+Teacher scale conferred **no measurable quality advantage** on this task — the 4B teacher is the cost-effective choice.
+
+### Out-of-domain (`suneeldk/text-json`)
+
+| Iteration | Prompt template | Validity | Throughput |
+|---|---|---|---|
+| iter1 | chat tokens (`<start_of_turn>`) | 0.0 % | 76.94 tok/s |
+| **iter2 (this model)** | **canonical** | **70.0 %** | **76.27 tok/s** |
+| iter3 | system persona header | 0.0 % | 74.12 tok/s |
+| base | canonical | 34.2 % | 62.00 tok/s |
+
+---
+
+## Training Details
+
+| | |
+|---|---|
+| Architecture | `LlamaForCausalLM` — 24 layers, hidden 1536, GQA 16/2 heads, vocab 130,560 |
+| Parameters | 1,080,632,832 total (679,552,512 non-embedding) |
+| Objective | $\mathcal{L}_{KD} = \alpha\mathcal{L}_{CE} + (1-\alpha)\tau^2\mathcal{L}_{KL}$ |
+| $\alpha$ / $\tau$ | 0.5 / 2.0 |
+| Vocabulary projection | 256,000 → 130,560 (shared-subspace truncation) |
+| Optimizer | AdamW, lr 2e-5, cosine, warmup 0.05 |
+| Epochs | 3 (early-stopped on val loss) |
+| Runtime | bfloat16, single-GPU PyTorch, eager attention (no ZeRO-3 / FlashAttention-2) |
+| Max sequence length | 2,048 |
+| Hardware | 1 × NVIDIA RTX PRO 6000 Blackwell Edition (96 GB), Nebius AI Cloud |
+| Software | Python 3.12 · PyTorch 2.5 · transformers 5.x |
+
+Full methodology, mathematics, compatibility patches, and ablations: **[`SCHEMAFORGE_WHITEPAPER.md`](./SCHEMAFORGE_WHITEPAPER.md)**.
+
+---
+
+## Limitations
+
+Please read these before deploying.
+
+- **Evaluation scale is small.** The in-domain suite is **n = 5 documents** (one per domain). The 100 % validity / 1.000 F1 figures are exact-match results on a small curated set, not population estimates — the Wilson 95 % CI on 5/5 is **[56.6 %, 100.0 %]**.
+- **Training scale is small.** This checkpoint was distilled on **n = 5 samples**. An SFT control ($\alpha = 1.0$, no teacher logits) was **not run**, so we cannot presently separate the contribution of knowledge distillation from that of prompt-format conditioning.
+- **Single seed.** No variance estimates or error bars. Sub-2B models vary substantially run-to-run on small datasets.
+- **Prompt-template brittleness.** The headline failure mode. Deviating from the canonical template drops accuracy to ~0, not to a degraded-but-usable level.
+- **Out-of-domain ceiling ≈ 70 %.** Roughly 30 % of unseen real-world documents produce unparseable output. Use constrained decoding in production.
+- **Synthetic in-domain documents.** Clean ASCII, consistent labeling, no OCR noise, English-only. Real scanned documents will be harder.
+- **Teacher outputs as targets.** Where the teacher was wrong, the student learned the error. No human-annotated gold standard exists for this checkpoint.
+- **Not evaluated against alternatives.** No comparison to Qwen2.5-1.5B, Phi-3-mini, rule-based extractors, or commercial document-AI APIs.
+
+**Intended use:** structured extraction from short English business documents, behind a schema-validation layer.
+**Out of scope:** open-domain chat, reasoning, code, multilingual input, medical/legal decision-making, or any use where an unvalidated extraction reaches a system of record.
+
+### Planned v2 run
+
+This is a **v1 release**, and the accuracy numbers above should be read as provisional. A second training and evaluation campaign is planned to address the limitations listed here directly:
+
+- **Real-world evaluation corpus** replacing the synthetic 5-document suite — $n \geq 500$ held-out documents per domain, including OCR-noisy scans, multi-column layouts, and non-English fields, with a **human-annotated gold subset** so accuracy is no longer measured against teacher output.
+- **The SFT control** ($\alpha = 1.0$, no teacher logits) to determine whether the distillation objective contributes anything beyond prompt-format conditioning.
+- **Competitive baselines** — Qwen2.5-1.5B, Phi-3-mini, prompt-engineered base MiniCPM5-1B with constrained decoding, and a rule-based extractor — under one unified harness.
+- **Multi-seed runs** (≥3) with reported variance and confidence intervals on every metric.
+- **Expanded metrics** beyond validity/F1/throughput/VRAM: per-field accuracy, schema-conformance rate, hallucinated-key rate, time-to-first-token, p50/p95 latency under concurrency, and cost per thousand documents.
+
+Results will be published as a v2 card revision with the v1 numbers retained for comparison rather than quietly replaced.
+
+---
+
+## Citation
+
+```bibtex
+@techreport{ty2026schemaforge,
+  title  = {SchemaForge: Distilling Ultra-Large Foundation Models into Edge SLMs
+            for Real-Time Enterprise JSON Extraction --
+            A Comparative Study of Gemma-4 Teachers and MiniCPM5-1B},
+  author = {Ty, Arjhine A.},
+  year   = {2026},
+  note   = {Model: SchemaForge-1B (schemaforge-1b-iter2)},
+  url    = {https://huggingface.co/arrochi112/SchemaForge-1B-JSON-Extractor}
+}
+```
+
+## Acknowledgements
+
+Teachers: `google/gemma-4-31B`, `google/gemma-4-E4B-it`. Student architecture: `openbmb/MiniCPM5-1B`. Compute: Nebius AI Cloud. Serving: vLLM. Constrained decoding: Outlines.
+
+**License:** Apache 2.0 — subject to the upstream licenses of the base and teacher models.

SchemaForge_Whitepaper.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f4e2c4d4a82f0d9f0e06644cb570b80e17d1406eb2471a9288ed3326167a490
+size 1017064