README.md

---
language:
  - en
license: mit
library_name: transformers
base_model: answerdotai/ModernBERT-base
tags:
  - promptforge
  - prompt-engineering
  - prompt-quality
  - modernbert
  - regression
  - text-classification
  - llm
pipeline_tag: text-classification
---

# PromptForge-Quality

Multi-dimension **prompt quality scorer**. Given an LLM prompt, returns an overall quality score plus per-dimension scores, inferred issues, and missing information.

Part of [PromptForge](https://github.com/arjun988/promptModel) — local-first prompt scoring and optimization.

## Model Details

### Model Description

PromptForge-Quality is a fine-tuned [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) encoder with regression heads that predict prompt quality on a **0–100** scale across seven dimensions.

- **Developed by:** PromptForge contributors  
- **Model type:** Encoder + multi-output regression (`promptforge_quality`)  
- **Language(s):** English  
- **License:** MIT  
- **Finetuned from:** [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) (~150M parameters)

### Dimensions scored

| Dimension | What it measures |
|-----------|------------------|
| `clarity` | How clear and unambiguous the prompt is |
| `specificity` | Level of concrete detail |
| `context` | Background / situation provided |
| `goal_definition` | How well the objective is defined |
| `constraints` | Limits, requirements, must/must-not rules |
| `completeness` | Whether enough information is present |
| `actionability` | How easy it is for an LLM to act on |
| `quality_score` | Aggregate overall score |

The model also surfaces **issues** (e.g. `too_vague`, `missing_context`) and **missing_information** hints.

### Model Sources

- **Repository:** https://github.com/arjun988/promptModel  
- **Companion model:** PromptForge-Optimizer (Qwen2.5-1.5B LoRA prompt rewriter)  
- **Demo:** Gradio app in the PromptForge repo (`demo/app.py`)

## Uses

### Direct Use

- Score prompts before sending them to an LLM  
- Diagnose weak prompts (what’s missing / unclear)  
- Measure before/after quality when rewriting prompts  
- Local / offline tooling via PromptForge CLI and Python API  

### Downstream Use

- Prompt linters in IDEs and agent frameworks  
- Dataset filtering / ranking for synthetic prompt corpora  
- Paired with **PromptForge-Optimizer** for score → optimize → re-score pipelines  

### Out-of-Scope Use

- Not a content moderator or safety classifier  
- Not a judge of factual correctness of LLM *answers*  
- Scores are calibrated on synthetic prompt quality labels — treat them as a useful proxy, not ground truth for every domain  

## Bias, Risks, and Limitations

- Trained largely on **synthetic** prompts with heuristic quality labels  
- May reward **length / structure** more than true semantic quality  
- English-centric; behavior on other languages is unverified  
- Very domain-specific jargon may score inconsistently  

### Recommendations

- Use scores comparatively (before vs after) rather than as absolute grades  
- Combine with human review for high-stakes prompt design  
- For custom domains, retrain with your own labeled prompts  

## How to Get Started with the Model

### Install & use with [`tuneprompt`](https://pypi.org/project/tuneprompt/) (recommended)

```bash
pip install tuneprompt

python -m promptforge download \
  --quality-repo ArjunShukla/PromptForge-Quality \
  --optimizer-repo ArjunShukla/PromptForge-Optimizer

python -m promptforge analyze "Build me a website"
# or: tuneprompt analyze "Build me a website"
```

```python
from promptforge import PromptForge

pf = PromptForge(quality_model_path="ArjunShukla/PromptForge-Quality")
print(pf.analyze("Make an app."))
# → quality_score, dimensions, issues, missing_information
```

> **Package:** [`tuneprompt`](https://pypi.org/project/tuneprompt/1.0.0/) on PyPI · **Import:** `promptforge` · **CLI:** `tuneprompt` / `promptforge` · **Code:** https://github.com/arjun988/promptModel

### Full pipeline (score + optimize)

```python
from promptforge import PromptForge

pf = PromptForge(
    quality_model_path="ArjunShukla/PromptForge-Quality",
    optimizer_model_path="ArjunShukla/PromptForge-Optimizer",
)
result = pf.run("Make an app about social media like facebook and stuff")
print(result["before"]["quality_score"], "→", result["after"]["quality_score"])
print(result["optimized_prompt"])
```

## Training Details

### Training Data

- **~25,000** synthetic prompts across coding, writing, research, data, and creative tasks  
- Quality levels from vague one-liners to fully specified prompts  
- Labels: overall `quality_score` + seven dimension scores (0–100)  

### Training Procedure

| Setting | Value |
|---------|-------|
| Base model | `answerdotai/ModernBERT-base` |
| Task | Multi-dimension regression |
| Epochs | 3 |
| Max length | 512 |
| Precision | fp16 |
| Hardware | NVIDIA RTX 5060 Laptop (8 GB) |
| Wall time | ~33 minutes |

Config: `configs/quality_scorer.yaml`

## Evaluation

Held-out results (local training run):

| Split | MAE | Pearson |
|-------|----:|--------:|
| Validation | **2.73** | **0.993** |
| Test (overall) | **0.96** | **0.999** |

Test Spearman (overall): **0.959**

### Summary

Strong correlation with synthetic quality labels on held-out data. Real-world prompts should still be sanity-checked — the scorer is best used for ranking and diagnosing structure gaps.

## Environmental Impact

- **Hardware Type:** NVIDIA RTX 5060 Laptop (8 GB)  
- **Hours used:** ~0.5 h for this checkpoint  
- **Cloud Provider:** N/A (local)  
- **Carbon Emitted:** Not measured  

## Technical Specifications

### Model Architecture and Objective

ModernBERT encoder with dual / multi regression heads predicting continuous quality scores (0–100).

### Compute Infrastructure

- **Hardware:** RTX 5060 Laptop GPU, 8 GB VRAM  
- **Software:** PyTorch (CUDA), Transformers, PromptForge training scripts  

### Artifact size

- **On-disk checkpoint:** ~574 MB  

## Citation

```bibtex
@software{promptforge_quality,
  title = {PromptForge-Quality},
  author = {PromptForge Contributors},
  year = {2026},
  url = {https://huggingface.co/ArjunShukla/PromptForge-Quality}
}
```

## Model Card Contact

Open an issue on the PromptForge GitHub repository.