Update model card

README.md

@@ -9,6 +9,7 @@ tags:
   - query-complexity
   - knowledge-distillation
   - research-computing
+  - hpc
 pipeline_tag: text-classification
 ---
 
@@ -16,7 +17,9 @@ pipeline_tag: text-classification
 
 Fine-tuned [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) (149M parameters) for three-class query complexity classification: **LOW**, **MEDIUM**, or **HIGH**.
 
-Built for the [STREAM](https://github.com/uicacer/STREAM) project (Smart Tiered Routing Engine for AI Models) to route queries automatically to the most cost-effective inference tier — local CPU, HPC GPU, or cloud API — at ~15ms per query with no API dependency.
+Built for the [STREAM](https://github.com/uicacer/STREAM) project (Smart Tiered Routing Engine for AI Models) to route queries automatically to the most cost-effective inference tier — local CPU, HPC GPU, or cloud API — at ~32 ms per query (CPU p50) with no API dependency.
+
+Covers **10 domains** representing the full breadth of a research university population: hpc, mathematics, statistics_ml, physics_chemistry, engineering, life_sciences, cs_software, philosophy_ethics, social_sciences, and history_culture.
 
 ## What It Does
 
@@ -51,9 +54,11 @@ complexity = max(result[0], key=lambda x: x["score"])["label"]
 
 ## Training
 
-**Knowledge distillation approach**: Claude Sonnet 4.6 (with extended thinking) labeled 6,912 queries across 6 domains and 3 complexity classes. ModernBERT-base was then fine-tuned on those labels. This is LLM-supervised fine-tuning — Claude generates hard labels; ModernBERT learns from them. The result runs at ~15ms per query with no API dependency.
+**Knowledge distillation approach**: Claude Sonnet 4.6 labeled 6,000 queries using a reasoning-depth rubric. ModernBERT-base was fine-tuned on those labels. The result runs at ~32 ms per query (CPU p50) with no API dependency — a 5× latency reduction vs. the LLM judge baseline.
+
+**Training dataset**: [anasnassar/llm-query-complexity-benchmark](https://huggingface.co/datasets/anasnassar/llm-query-complexity-benchmark) — 6,000 doubly balanced queries across 10 domains × 3 complexity classes (200/domain/class hard cap; 4,800 train / 1,200 test, 80/20 stratified split, seed=42).
 
-**Training dataset**: [anasnassar/llm-query-complexity-benchmark](https://huggingface.co/datasets/anasnassar/llm-query-complexity-benchmark) — 6,912 queries, 6 domains, balanced across complexity classes.
+**Sources**: Derived from [sentence-transformers/stackexchange-duplicates](https://huggingface.co/datasets/sentence-transformers/stackexchange-duplicates) (Apache 2.0), [cais/mmlu](https://huggingface.co/datasets/cais/mmlu) (MIT), [TIGER-Lab/MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (MIT), and [qiaojin/PubMedQA](https://huggingface.co/datasets/qiaojin/PubMedQA) (MIT).
 
 **Hyperparameters**:
 
@@ -70,26 +75,41 @@ complexity = max(result[0], key=lambda x: x["score"])["label"]
 
 ## Evaluation
 
-Three evaluation strategies are used to address data leakage from LLM-generated near-duplicates:
+Evaluated on a fixed 750-query held-out test set (250/class), stratified split, seed=42.
+
+| Metric | Value |
+|--------|-------|
+| Accuracy | 64.2% |
+| Macro-F1 | 0.640 |
+| FREE-tier retention | 85.4% |
+| Latency p50 (CPU) | 32 ms |
 
-| Strategy | Description |
-|----------|-------------|
-| **Domain-held-out 6-fold CV** | Train on 5 domains, test on 6th. Primary reported metric. |
-| **Similarity-aware split** | Near-duplicate queries (cosine sim > 0.90) kept on same side of split. |
-| **Real-world (LMSYS Arena)** | Evaluated on real user prompts from Chatbot Arena — fully out-of-distribution. |
+**Per-class recall (Wilson 95% CI):**
 
-*Note: Random train/test split on LLM-generated data yields inflated accuracy (~99%) due to near-duplicate phrasings. Domain-held-out and real-world numbers are the rigorous metrics.*
+| Class | Recall | 95% CI |
+|-------|--------|--------|
+| LOW   | 70.8%  | [66.1%, 75.0%] |
+| MEDIUM | 49.3% | [44.4%, 54.1%] |
+| HIGH  | 72.5%  | [67.9%, 76.7%] |
 
-Full evaluation code: [scripts/eval/](https://github.com/uicacer/STREAM/tree/main/scripts/eval)
+## Judge Comparison
 
-## Performance
+| Judge | Latency p50 | Accuracy | Macro-F1 | API dependency |
+|-------|-------------|----------|----------|----------------|
+| ModernBERT (this model) | 32 ms | 64.2% | 0.640 | None |
+| Llama 3.2 3B (LLM judge) | 164 ms | 49.0% | 0.436 | Ollama |
 
-| Judge | Latency (p50) | Notes |
-|-------|--------------|-------|
-| ModernBERT (this model) | ~15ms | CPU, no API dependency |
-| Llama 3.2 3B (LLM judge) | ~390ms | Requires Ollama |
+## Threshold-Tunable Routing
 
-26× latency reduction vs. the LLM judge baseline.
+Rather than a fixed argmax decision, STREAM exposes a tunable threshold θ ∈ [0,1]. A query is routed to cloud when `P(HIGH) ≥ θ`; otherwise to HPC or local. As θ increases, cloud spend drops but HIGH recall decreases — a continuous precision-recall-cost tradeoff.
+
+**Budget-aware adaptive routing** automatically raises θ as cloud spend approaches the monthly budget cap:
+
+```
+θ_eff(t) = max(θ_base, S(t)/B)
+```
+
+where S(t) is cumulative spend and B is the monthly budget.
 
 ## Integration in STREAM
 
@@ -108,7 +128,48 @@ result = judge_complexity("Explain quantum entanglement", strategy="modernbert")
   title     = {{STREAM}: Multi-Tier {LLM} Inference Middleware with Dual-Channel {HPC} Token Streaming},
   author    = {Nassar, Anas and Mohr, Steve and Apanasevich, Leonard and Sharma, Himanshu},
   booktitle = {Practice and Experience in Advanced Research Computing (PEARC '26)},
-  year      = {2026}
+  year      = {2026},
+  doi       = {10.1145/3785462.3815847}
+}
+
+@misc{nassar2026benchmark,
+  author    = {Nassar, Anas},
+  title     = {{LLM} Query Complexity Benchmark},
+  year      = {2026},
+  publisher = {Hugging Face},
+  url       = {https://huggingface.co/datasets/anasnassar/llm-query-complexity-benchmark}
+}
+
+% Original source datasets
+@article{hendrycks2021mmlu,
+  author  = {Dan Hendrycks and others},
+  title   = {Measuring Massive Multitask Language Understanding},
+  journal = {ICLR},
+  year    = {2021},
+  url     = {https://huggingface.co/datasets/cais/mmlu}
+}
+
+@article{wang2024mmlupro,
+  author  = {Yubo Wang and others},
+  title   = {{MMLU-Pro}: A More Robust and Challenging Multi-Task Language Understanding Benchmark},
+  journal = {arXiv:2406.01574},
+  year    = {2024},
+  url     = {https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro}
+}
+
+@inproceedings{jin2019pubmedqa,
+  author    = {Qiao Jin and others},
+  title     = {{PubMedQA}: A Biomedical Research Question Answering Dataset},
+  booktitle = {EMNLP},
+  year      = {2019},
+  url       = {https://huggingface.co/datasets/qiaojin/PubMedQA}
+}
+
+@misc{stackexchange_dataset,
+  author = {Reimers, Nils and Gurevych, Iryna},
+  title  = {{StackExchange} Duplicate Questions Dataset},
+  year   = {2019},
+  url    = {https://huggingface.co/datasets/sentence-transformers/stackexchange-duplicates}
 }
 ```