Improve developer-focused model card

README.md

@@ -10,214 +10,254 @@ tags:
   - code-t5
   - constraint-dsl
 base_model: Salesforce/codet5p-220m
-pipeline_tag: text-generation
+pipeline_tag: text2text-generation
 ---
 
-# StructFix — CodeT5+ 220M (ConstraintDSL)
+# StructFix
 
-Experimental schema-aware structured output recovery model for small language models.
+Schema-aware structured output recovery for LLMs and agent workflows.
 
-A 220M CodeT5+ model fine-tuned to repair broken structured outputs from agents, tool calls, and LLM workflows using **ConstraintDSL** instead of raw JSON Schema.
+- Recovers invalid structured outputs
+- Repairs missing required fields
+- Fixes enum violations
+- Validates and repairs tool-call payloads
+- Handles markdown-wrapped or text-wrapped JSON
+- Lightweight: 220M parameters
 
-## Key result
+**91.9% schema success on unseen schemas with randomized field names.**
 
-Replacing raw JSON Schema with ConstraintDSL improved unseen-schema recovery from **55.0% to 96.3%** — same model, same data, same training procedure. Only the schema representation changed.
+StructFix is a CodeT5+ 220M model fine-tuned to repair broken structured outputs using **ConstraintDSL**, a compact schema representation designed for small language models.
 
-With fully randomized field names (`f_8f31a7`, `f_2d88b1`, ...), performance holds at **91.9%**, demonstrating strong structural reasoning with a residual dependency on field name semantics.
+## Problem
 
-## Performance
+LLM and agent outputs often look almost correct but fail validation.
 
-| Method | Schema Success (unseen) | Params | Latency |
-|--------|:----------------------:|:------:|:-------:|
-| json-repair | 65.2% | — | 0.13ms |
-| CodeT5+ + JSON Schema | 55.0% | 220M | ~600ms |
-| **StructFix (ConstraintDSL)** | **96.3%** | 220M | ~690ms |
-| **StructFix (random fields)** | **91.9%** | 220M | ~690ms |
+Input:
 
-## Per-corruption performance (unseen schemas, random hex fields)
-
-| Corruption | StructFix | json-repair |
-|-----------|:---------:|:-----------:|
-| `invalid_enum` | 96.4% | 0% |
-| `missing_required` | 92.2% | 0% |
-| `null_required` | 97.9% | 2.9% |
-| `wrong_type` | 92.0% | 0% |
-| `tool_call_partial_args` | 90.9% | 0% |
-| `tool_call_python_syntax` | 90.0% | 0% |
-| `tool_call_wrong_param` | 93.8% | 51.2% |
-| `agent_chain` | 87.2% | 40.5% |
+```json
+{
+  "priority": "urgent"
+}
+```
 
-## Validated examples
+Constraint:
 
-The following examples pass 100/100 inferences with 100% schema validity:
+```text
+priority must be one of: low | medium | high
+```
 
-### Missing required field
+Output:
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "tool": {"type": "string"},
-        "status": {"type": "string", "enum": ["success", "error", "pending"]},
-        "result": {"type": "string"},
-    },
-    "required": ["tool", "status", "result"],
+```json
+{
+  "priority": "high"
 }
-broken = '{"tool":"search","result":"Found 3 items"}'
-# Output: {"tool":"search","status":"success","result":"Found 3 items"}
 ```
 
-### Wrong type from legacy system
+## Quick Start
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "pipeline_id": {"type": "string"},
-        "rows_affected": {"type": "integer"},
-        "success": {"type": "boolean"},
-    },
-    "required": ["pipeline_id", "rows_affected", "success"],
-}
-broken = '{"pipeline_id":"ETL-42","rows_affected":"1250","success":"yes"}'
-# Output: {"pipeline_id":"ETL-42","rows_affected":1250,"success":true}
+Install:
+
+```bash
+pip install transformers torch
 ```
 
-### Markdown-wrapped response
+Run inference:
 
 ```python
-schema = {
-    "type": "object",
-    "properties": {
-        "user_id": {"type": "integer"},
-        "username": {"type": "string"},
-        "email": {"type": "string"},
-    },
-    "required": ["user_id", "username", "email"],
-}
-broken = '```json\n{"user_id":42,"username":"jdoe","email":"jdoe@example.com"}\n```'
-# Output: {"user_id":42,"username":"jdoe","email":"jdoe@example.com"}
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+model_id = "ottema/structfix-codet5p-220m"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
+
+dsl = """FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes"""
+
+broken_output = """{
+  "priority": "urgent"
+}"""
+
+prompt = f"""TASK repair_structured_output
+
+SPEC
+{dsl}
+
+BROKEN_OUTPUT
+{broken_output}"""
+
+inputs = tokenizer(prompt, return_tensors="pt", max_length=512, truncation=True)
+outputs = model.generate(
+    **inputs,
+    max_length=256,
+    num_beams=1,
+    do_sample=False,
+)
+
+repaired = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(repaired)
 ```
 
-### Single quotes
+Example output:
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "title": {"type": "string"},
-        "year": {"type": "integer"},
-        "category": {"type": "string", "enum": ["paper", "book", "article"]},
-    },
-    "required": ["title", "category"],
-}
-broken = "{'title':'Attention Is All You Need','year':2017,'category':'paper'}"
-# Output: {"title":"Attention Is All You Need","year":2017,"category":"paper"}
+```json
+{"priority":"high","description":""}
 ```
 
-### Truncated object
+## What It Repairs
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "product": {"type": "string"},
-        "price": {"type": "number"},
-        "currency": {"type": "string", "enum": ["USD", "EUR", "BRL"]},
-    },
-    "required": ["product", "price", "currency"],
-}
-broken = '{"product":"Widget","price":29.99,"currency":"USD"'
-# Output: {"product":"Widget","price":29.99,"currency":"USD"}
+| Category | Support |
+| --- | :---: |
+| Missing required fields | Yes |
+| Invalid enums | Yes |
+| Wrong types | Yes |
+| Partial tool calls | Yes |
+| Markdown-wrapped JSON | Yes |
+| Extra text before or after JSON | Yes |
+| Truncated objects and arrays | Yes |
+| Python-like tool calls | Yes |
+
+## When To Use It
+
+Use StructFix when you have a schema or tool definition and need to recover a structured payload from an LLM, agent, ETL, or integration workflow.
+
+Good fits:
+
+- Agent tool-call argument repair
+- JSON payload recovery before validation
+- Enum and required-field correction
+- Recovering JSON from assistant responses with prose or markdown
+- Lightweight local repair before retrying an expensive model call
+
+Not a good fit:
+
+- Arbitrary data cleaning without a schema
+- High-stakes financial, medical, legal, or regulatory corrections without human validation
+- Inputs longer than the model context window
+- Tasks where preserving every original field name is mandatory without post-validation
+
+## ConstraintDSL
+
+StructFix does not use raw JSON Schema directly at inference time. It expects a compact line-oriented schema format called ConstraintDSL.
+
+Example:
+
+```text
+FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes
+FIELD customer_id TYPE integer REQUIRED no
 ```
 
-### Thought process before JSON
+Tool-call example:
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "id": {"type": "integer"},
-        "name": {"type": "string"},
-        "role": {"type": "string", "enum": ["admin", "user", "viewer"]},
-    },
-    "required": ["id", "name", "role"],
-}
-broken = 'I need to look up the user.\nLet me query the database.\n{"id":1,"name":"Alice","role":"user"}'
-# Output: {"id":1,"name":"Alice","role":"user"}
+```text
+TOOL create_ticket
+ARG priority TYPE string VALUES low|medium|high REQUIRED yes
+ARG description TYPE string REQUIRED yes
+ARG customer_id TYPE integer REQUIRED no
 ```
 
-### Extra text before JSON
+Model input format:
 
-```python
-schema = {
-    "type": "object",
-    "properties": {
-        "status_code": {"type": "integer"},
-        "message": {"type": "string"},
-    },
-    "required": ["status_code", "message"],
-}
-broken = 'Here is the API response:\n{"status_code":200,"message":"OK"}'
-# Output: {"status_code":200,"message":"OK"}
+```text
+TASK repair_structured_output
+
+SPEC
+FIELD priority TYPE string VALUES low|medium|high REQUIRED yes
+FIELD description TYPE string REQUIRED yes
+
+BROKEN_OUTPUT
+{"priority":"urgent"}
 ```
 
-## Schema ablation study
+ConstraintDSL exists because raw JSON Schema generalized poorly in this setup. With the same base model, data, and training procedure, ConstraintDSL improved unseen-schema schema success from **55.0%** to **96.3%**.
+
+See [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl) for the DSL specification and compiler references.
+
+## Results
+
+### Main benchmark
+
+| Method | Schema Success |
+| --- | :---: |
+| json-repair | 65.2% |
+| CodeT5+ + raw JSON Schema | 55.0% |
+| **StructFix + ConstraintDSL** | **96.3%** |
+| **StructFix + randomized fields** | **91.9%** |
 
-| DSL variant | Schema Success |
-|------------|:--------------:|
-| Full ConstraintDSL | 96.3% |
-| No DSL at all | 78.4% (-17.9pp) |
-| Dummy DSL | 75.0% (-21.3pp) |
-| Conflicting DSL | 88.3% (-8.0pp) |
-| Shuffled field names | 73.4% (-22.9pp) |
-| Shuffled enum values | 89.8% (-6.5pp) |
-| Shuffled required flags | 93.2% (-3.1pp) |
-| Minimal DSL | 82.5% (-13.8pp) |
-| Field names only | 72.2% (-24.1pp) |
+### Schema representation ablation
 
-## Known limitations
+| Test | Schema Success |
+| --- | :---: |
+| Raw JSON Schema | 55.0% |
+| ConstraintDSL | 96.3% |
+| Randomized field names | 91.9% |
 
-### Field name substitution
+### Per-corruption performance
 
-The model exhibits residual dependency on field name semantics. When processing schemas with field names that overlap with its training vocabulary, it may substitute field names with semantically similar alternatives:
+Unseen schemas with random hex field names:
+
+| Corruption | StructFix | json-repair |
+| --- | :---: | :---: |
+| `invalid_enum` | 96.4% | 0% |
+| `missing_required` | 92.2% | 0% |
+| `null_required` | 97.9% | 2.9% |
+| `wrong_type` | 92.0% | 0% |
+| `tool_call_partial_args` | 90.9% | 0% |
+| `tool_call_python_syntax` | 90.0% | 0% |
+| `tool_call_wrong_param` | 93.8% | 51.2% |
+| `agent_chain` | 87.2% | 40.5% |
+
+Latency in the benchmark was about **690 ms/example** for StructFix and **0.13 ms/example** for json-repair.
+
+## Known Limitations
+
+- Field names unseen during training may be substituted by semantically similar names.
+- Synonym enum repair depends on lexical similarity and field-name semantics.
+- The model is English-oriented in the current version.
+- Maximum input length is 512 tokens.
+- Always validate the output against your schema after inference.
+- Not recommended for financial, medical, legal, or regulatory corrections without human review.
+
+Example field-name substitutions observed in showcase validation:
 
 | DSL field name | Model output |
-|---------------|-------------|
+| --- | --- |
 | `action` | `active` |
 | `records_processed` | `items_processed` |
 | `contract_id` | `consign_id` |
 | `to` | `strand` |
 
-This was discovered through a showcase validation suite (100 inferences per example). Of 25 hand-crafted real-world examples, 7 passed at 100% schema validity and 18 exhibited field name substitution.
-
-**Root cause**: The model learned strong structural reasoning from the DSL, but retains a lexical bias toward field names seen during training. When a real-world field name like `action` partially matches a common training field like `active`, the model may override the DSL spec.
-
-**Impact**: Benchmark performance (96.3% on generated test sets) is reliable. Real-world performance depends on how much the target field names overlap with training vocabulary.
+## Research Findings
 
-### Other limitations
+- Raw JSON Schema generalized poorly for this 220M model: **55.0%** schema success.
+- ConstraintDSL improved unseen-schema performance to **96.3%**.
+- Randomized field names still achieved **91.9%**, suggesting the model uses explicit constraints rather than only memorized field semantics.
+- Field names remain the most important DSL component in ablations.
 
-- `synonym_enum` repair: 0% with random fields — requires semantic field name understanding
-- Latency: ~690ms per example (vs 0.13ms for json-repair)
-- English-only in current version
-- Max input length: 512 tokens
+Full benchmark details are available in [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench).
 
-## Training details
+## Training Details
 
-- **Base model**: Salesforce/codet5p-220m (220M parameters)
-- **Training data**: 200K synthetic examples (StructFix-Bench)
-- **Format**: ConstraintDSL (not raw JSON Schema)
-- **Epochs**: 3
-- **Batch size**: 8 × 4 gradient accumulation = 32 effective
-- **Learning rate**: 2e-4
-- **Final eval loss**: 0.056
-- **Field name shuffling**: 50% of training examples use shuffled field names
-- **Synthetic enums**: 50% of training examples use synthetic enum values
+| Item | Value |
+| --- | --- |
+| Base model | `Salesforce/codet5p-220m` |
+| Parameters | 220M |
+| Training data | 200K synthetic examples |
+| Format | ConstraintDSL |
+| Epochs | 3 |
+| Effective batch size | 32 |
+| Learning rate | 2e-4 |
+| Final eval loss | 0.056 |
+| Field-name shuffling | 50% of training examples |
+| Synthetic enums | 50% of training examples |
 
-## Related
+## Related Repositories
 
-- [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench) — dataset and benchmark
-- [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl) — DSL specification and compilers
+- [StructFix model](https://huggingface.co/ottema/structfix-codet5p-220m): this model card, focused on usage.
+- [StructFix-Bench](https://huggingface.co/datasets/ottema/structfix-bench): dataset, benchmark splits, and ablations.
+- [ConstraintDSL](https://huggingface.co/datasets/ottema/constraint-dsl): DSL specification and compiler references.
 
 ## Citation
 
@@ -232,4 +272,4 @@ This was discovered through a showcase validation suite (100 inferences per exam
 
 ## License
 
-Apache-2.0 (check Salesforce CodeT5+ base model license for compatibility)
+Apache-2.0. Check the Salesforce CodeT5+ base model license for compatibility with your use case.