File size: 4,315 Bytes
562442d 596d5b5 e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 596d5b5 e111026 562442d e111026 562442d e111026 562442d e111026 562442d 596d5b5 f3f8abb e111026 f3f8abb 99258c6 596d5b5 e111026 99258c6 e111026 99258c6 e111026 99258c6 e111026 596d5b5 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 596d5b5 e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 562442d e111026 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 |
---
library_name: transformers
license: apache-2.0
language:
- en
base_model:
- answerdotai/ModernBERT-large
pipeline_tag: text-classification
author: Shreyan C (@thethinkmachine)
datasets:
- BhabhaAI/DEITA-Complexity
---
# Maxwell Instruction Complexity Estimator (MICE)
[](https://huggingface.co/thethinkmachine/Maxwell-Task-Complexity-Scorer-v0.2) [](LICENSE) [](#)
A fast, efficient, and accurate instruction complexity scorer powered by ModernBERT-Large. MICE predicts normalized task difficulty scores (0–1) for English instructions, with an easy option to rescale to custom ranges.
---
## 🚀 Features
* **Lightweight & Fast**: Leverages a compact backbone (ModernBERT-Large + LoRA) with only 14.4M trainable parameters.
* **Data-Driven**: Trained on 66.5K English instruction–score pairs from the DEITA-Complexity dataset.
* **High Fidelity**: Matches the performance of models 34× larger on standard complexity benchmarks.
* **Flexible Scoring**: Outputs normalized scores (0–1) by default, with optional denormalization to any range (e.g., \[1–6], \[0–100]).
---
## 🔧 Usage
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
model_name = "thethinkmachine/Maxwell-Task-Complexity-Scorer-v0.2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# 1. Get normalized complexity (0–1)
def get_normalized_score(text: str) -> float:
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits.squeeze()
return float(logits)
# 2. Denormalize to [min_score, max_score]
def get_denormalized_score(text: str, min_score: float = 1, max_score: float = 6) -> float:
norm = get_normalized_score(text)
raw = norm * (max_score - min_score) + min_score
return float(round(raw, 2))
# Example
query = "Is learning equivalent to decreasing local entropy?"
print("Normalized:", get_normalized_score(query))
print("Evol-Complexity [1–6]:", get_denormalized_score(query))
```
---
## 📖 Model Details
* **Architecture:** ModernBERT-Large backbone with LoRA adapters (rank 32, alpha 64, dropout 0.1).
* **Task:** Sequence Classification.
* **Languages:** English.
* **Training Data:** 66,500 instruction–score pairs from \[BhabhaAI/DEITA-Complexity].
* **Normalization:** Min–max scaled to \[0,1]; denormalization recommended via `score * (max - min) + min`.
### Data Distribution
| Original Score | Count | % |
| -------------- | ------ | ----- |
| 1 | 8,729 | 13.3% |
| 2 | 5,399 | 8.2% |
| 3 | 10,937 | 16.7% |
| 4 | 9,801 | 15.0% |
| 5 | 24,485 | 37.4% |
| 6 | 6,123 | 9.3% |
Outliers (0,7–9) were pruned (<1% of data).
---
## ⚙️ Training Configuration
* **Optimizer:** AdamW (lr=5e-5, weight decay=0.01)
* **Batch Size:** 8
* **Epochs:** 3
* **Max Seq. Length:** 512
* **Warmup:** 10% of total steps
* **Compute:** 50.3M tokens, TTP ratio ≈3.5
---
## 🌱 Environmental Impact
* **Compute Used:** 16h on 1× NVIDIA L4 GPU (72W TDP) in GCP asia-south1.
* **CO₂ Emissions:** 0.87 kg CO₂eq (fully offset).
* **Estimator:** ML CO₂ Impact Calculator.
---
## 🔍 Bias & Limitations
* **Domain Bias:** Trained primarily on general English; may underperform on technical/coding/math instructions.
* **Language:** English-only.
* **Scaling Caution:** Denormalization preserves ordering but absolute values depend on chosen range.
---
## 📚 Citation
If you use MICE in your research, please cite:
> Chaubey, S. (2024). Maxwell Instruction Complexity Estimator (MICE). https://huggingface.co/thethinkmachine/MICE
---
## 🙋♂️ Author & Contact
**Shreyan C** ([thethinkmachine](https://huggingface.co/thethinkmachine))
Email: [shreyan.chaubey@gmail.com](mailto:shreyan.chaubey@gmail.com)
*This project is licensed under the Apache 2.0 License.* |