README.md

---
license: apache-2.0
base_model: sentence-transformers/all-MiniLM-L6-v2
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- text-classification
- semantic-routing
- llm-d
- loss:BatchAllTripletLoss
---

# llm-d-sc-complexity

A 384-dimensional sentence embedding model fine-tuned to separate **prompt complexity tiers**,
for use as a routing signal by [llm-d-sc](https://github.com/llm-d-incubation/llm-d-semantic-classifier),
the semantic classification runtime for llm-d.

The model does not emit a class directly. It produces an embedding that is ranked against a set of
labelled **anchors** (`anchors.json`, shipped with this repository). This keeps the taxonomy as data
rather than as a frozen classification head: anchors can be replaced or extended without retraining.

## Taxonomy

| Tier | Meaning |
|---|---|
| `SIMPLE` | Single-fact lookup or a one-step instruction |
| `MEDIUM` | One substantive task: a function, an explanation with an example, a short guide |
| `COMPLEX` | Multi-component design or build with several interacting concerns |
| `REASONING` | Proof, derivation, or formal analysis |

## Intended use

Selecting a serving tier per request. A `SIMPLE` prompt does not need a frontier model; a `REASONING`
prompt usually does. The classifier emits ranked evidence only. Routing, endpoint selection, and
session affinity remain the caller's responsibility.

## Evaluation

Evaluated by llm-d-sc on a **held-out set of 80 prompts authored independently of the training
corpus**, 20 per tier, deliberately drawn from domains outside the training pipeline's domain-transfer
list (sailing, agriculture, transit, broadcast, museums, orchestras). 20 of the 80 are boundary cases.

Classification method: cosine similarity against the anchors, mean of the top 3 per tier, argmax.

| Model | Accuracy | Macro F1 | Boundary cases |
|---|---:|---:|---:|
| **llm-d-sc-complexity (this model)** | **0.9750** | **0.9749** | **0.9500** |
| `all-MiniLM-L6-v2` (base, same anchors) | 0.6250 | 0.6234 | 0.6500 |

Per tier:

| Tier | Precision | Recall | F1 | Support |
|---|---:|---:|---:|---:|
| SIMPLE | 1.000 | 1.000 | 1.000 | 20 |
| MEDIUM | 0.909 | 1.000 | 0.952 | 20 |
| COMPLEX | 1.000 | 0.900 | 0.947 | 20 |
| REASONING | 1.000 | 1.000 | 1.000 | 20 |

Both errors are `COMPLEX` predicted as `MEDIUM`, the same boundary the training-time report
identified as the model's only remaining confusion. The base model's confidence is near-uniform
(0.25-0.27 across four tiers), which is the expected signature of an embedding space that carries
no complexity structure at all.

Latency on CPU (single thread, Apple M-series, embed plus rank): **p50 9.2 ms, p99 19.0 ms**.

These numbers were produced on a homelab and have not been independently reproduced.

## Training

Fine-tuned from `sentence-transformers/all-MiniLM-L6-v2` with `BatchAllTripletLoss` and
`group_by_label` batch sampling on 871 synthetic examples generated and cross-verified by two
separate LLMs. Pipeline: https://github.com/cnuland/hello-chris-sr-finetuned

## Usage

```python
from sentence_transformers import SentenceTransformer
import json, numpy as np

model = SentenceTransformer("cnuland/llm-d-sc-complexity")
anchors = json.load(open("anchors.json"))["anchors"]

def classify(text, top_k=3):
    q = model.encode(text, normalize_embeddings=True)
    scores = {}
    for tier, examples in anchors.items():
        sims = model.encode(examples, normalize_embeddings=True) @ q
        scores[tier] = float(np.sort(sims)[-top_k:].mean())
    return max(scores, key=scores.get), scores

print(classify("Prove that the square root of 3 is irrational."))
# ('REASONING', {...})
```

## Limitations

- English only.
- Trained on synthetic data; no human-labelled validation set exists.
- The `MEDIUM` / `COMPLEX` boundary is genuinely ambiguous and is where residual error concentrates.
- Anchor quality directly determines accuracy. Replacing `anchors.json` changes behaviour without retraining.

## License

Apache-2.0.