|
|
--- |
|
|
library_name: transformers |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-large |
|
|
pipeline_tag: text-classification |
|
|
author: Shreyan C (@thethinkmachine) |
|
|
datasets: |
|
|
- BhabhaAI/DEITA-Complexity |
|
|
--- |
|
|
|
|
|
|
|
|
 |
|
|
|
|
|
# Maxwell Instruction Complexity Estimator (MICE) |
|
|
|
|
|
[](https://huggingface.co/thethinkmachine/Maxwell-Task-Complexity-Scorer-v0.2) [](LICENSE) [](#) |
|
|
|
|
|
A fast, efficient, and accurate instruction complexity scorer powered by ModernBERT-Large. MICE predicts normalized task difficulty scores (0–1) for English instructions, with an easy option to rescale to custom ranges. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 Features |
|
|
|
|
|
* **Lightweight & Fast**: Leverages a compact backbone (ModernBERT-Large + LoRA) with only 14.4M trainable parameters. |
|
|
* **Data-Driven**: Trained on 66.5K English instruction–score pairs from the DEITA-Complexity dataset. |
|
|
* **High Fidelity**: Matches the performance of models 34× larger on standard complexity benchmarks. |
|
|
* **Flexible Scoring**: Outputs normalized scores (0–1) by default, with optional denormalization to any range (e.g., \[1–6], \[0–100]). |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔧 Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification |
|
|
import torch |
|
|
|
|
|
model_name = "thethinkmachine/Maxwell-Task-Complexity-Scorer-v0.2" |
|
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
model = AutoModelForSequenceClassification.from_pretrained(model_name) |
|
|
|
|
|
# 1. Get normalized complexity (0–1) |
|
|
def get_normalized_score(text: str) -> float: |
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
with torch.no_grad(): |
|
|
logits = model(**inputs).logits.squeeze() |
|
|
return float(logits) |
|
|
|
|
|
# 2. Denormalize to [min_score, max_score] |
|
|
def get_denormalized_score(text: str, min_score: float = 1, max_score: float = 6) -> float: |
|
|
norm = get_normalized_score(text) |
|
|
raw = norm * (max_score - min_score) + min_score |
|
|
return float(round(raw, 2)) |
|
|
|
|
|
# Example |
|
|
query = "Is learning equivalent to decreasing local entropy?" |
|
|
print("Normalized:", get_normalized_score(query)) |
|
|
print("Evol-Complexity [1–6]:", get_denormalized_score(query)) |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📖 Model Details |
|
|
|
|
|
* **Architecture:** ModernBERT-Large backbone with LoRA adapters (rank 32, alpha 64, dropout 0.1). |
|
|
* **Task:** Sequence Classification. |
|
|
* **Languages:** English. |
|
|
* **Training Data:** 66,500 instruction–score pairs from \[BhabhaAI/DEITA-Complexity]. |
|
|
* **Normalization:** Min–max scaled to \[0,1]; denormalization recommended via `score * (max - min) + min`. |
|
|
|
|
|
### Data Distribution |
|
|
|
|
|
| Original Score | Count | % | |
|
|
| -------------- | ------ | ----- | |
|
|
| 1 | 8,729 | 13.3% | |
|
|
| 2 | 5,399 | 8.2% | |
|
|
| 3 | 10,937 | 16.7% | |
|
|
| 4 | 9,801 | 15.0% | |
|
|
| 5 | 24,485 | 37.4% | |
|
|
| 6 | 6,123 | 9.3% | |
|
|
|
|
|
Outliers (0,7–9) were pruned (<1% of data). |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚙️ Training Configuration |
|
|
|
|
|
* **Optimizer:** AdamW (lr=5e-5, weight decay=0.01) |
|
|
* **Batch Size:** 8 |
|
|
* **Epochs:** 3 |
|
|
* **Max Seq. Length:** 512 |
|
|
* **Warmup:** 10% of total steps |
|
|
* **Compute:** 50.3M tokens, TTP ratio ≈3.5 |
|
|
|
|
|
--- |
|
|
|
|
|
## 🌱 Environmental Impact |
|
|
|
|
|
* **Compute Used:** 16h on 1× NVIDIA L4 GPU (72W TDP) in GCP asia-south1. |
|
|
* **CO₂ Emissions:** 0.87 kg CO₂eq (fully offset). |
|
|
* **Estimator:** ML CO₂ Impact Calculator. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🔍 Bias & Limitations |
|
|
|
|
|
* **Domain Bias:** Trained primarily on general English; may underperform on technical/coding/math instructions. |
|
|
* **Language:** English-only. |
|
|
* **Scaling Caution:** Denormalization preserves ordering but absolute values depend on chosen range. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📚 Citation |
|
|
|
|
|
If you use MICE in your research, please cite: |
|
|
|
|
|
> Chaubey, S. (2024). Maxwell Instruction Complexity Estimator (MICE). https://huggingface.co/thethinkmachine/MICE |
|
|
|
|
|
--- |
|
|
|
|
|
## 🙋♂️ Author & Contact |
|
|
|
|
|
**Shreyan C** ([thethinkmachine](https://huggingface.co/thethinkmachine)) |
|
|
Email: [shreyan.chaubey@gmail.com](mailto:shreyan.chaubey@gmail.com) |
|
|
|
|
|
*This project is licensed under the Apache 2.0 License.* |
