DATASET_CARD.md

---
language: en
license: cc-by-4.0
task_categories:
- text-classification
- preference-learning
- reward-modeling
size_categories:
- 10K<n<100K
tags:
- preference-pairs
- reasoning-traces
- outcome-reward-model
- pairwise-ranking
pretty_name: ORM Pairwise Preference Pairs
dataset_info:
  features:
  - name: chosen
    dtype: string
  - name: rejected
    dtype: string
  - name: meta
    dtype:
      chosen_chain_id: string
      rejected_chain_id: string
      chosen_label: int32
      rejected_label: int32
  splits:
  - name: train
    num_examples: 41656
  - name: validation
    num_examples: 1144
  - name: test
    num_examples: 1232
---

# ORM Pairwise Preference Pairs Dataset

<div align="center">

**High-Quality Reasoning Trace Preferences for Training Outcome Reward Models**

[![Paper](https://img.shields.io/badge/Paper-ArXiv-red)](link-to-arxiv)
[![Model](https://img.shields.io/badge/Model-HuggingFace-yellow)](https://huggingface.co/LossFunctionLover/pairwise-orm-model)

</div>

## 📋 Dataset Description

This dataset contains **44,032 curated pairwise preference judgments** over reasoning traces, designed for training robust Outcome Reward Models (ORMs). Each example is a pair of reasoning traces for the same task, with one marked as preferred (correct) and the other as rejected (incorrect).

**Key Features:**
- ✅ **Weeks of manual curation** and quality control on source data
- ✅ **Validated quality**: Base model achieves 98.2% pairwise accuracy
- ✅ **Strong signal separation**: Pearson r=0.87, Spearman ρ=0.83
- ✅ **Balanced construction**: Multiple negatives per positive for robust learning
- ✅ **Full traceability**: Chain IDs for linking back to source examples

## 📊 Dataset Statistics

### Split Sizes

| Split | Pairs | Negatives per Positive | Source Examples |
|-------|-------|------------------------|-----------------|
| **Train** | 41,656 | 8 | 9,482 (pointwise) |
| **Validation** | 1,144 | 4 | 524 (pointwise) |
| **Test** | 1,232 | 4 | 547 (pointwise) |
| **Total** | **44,032** | - | 10,553 |

### Quality Metrics (Source Pointwise Dataset)

**Base Model Log-Probability Analysis:**
- **Pearson correlation**: r = 0.87 (p < 1e-162)
- **Spearman correlation**: ρ = 0.83 (p < 1e-134)
- **Pairwise accuracy**: 98.2%
- **Mean log-prob (positive)**: -2.17
- **Mean log-prob (negative)**: -3.64
- **Separation**: Strong discrimination between correct/incorrect traces

![Base Model Distribution](https://your-image-link/distribution.png)

These metrics confirm robust signal quality before pairwise transformation, validating the dataset's suitability for preference learning.

## 🏗️ Dataset Construction

### Phase 1: Source Data Curation (Pointwise)

**Original Dataset Structure:**
```json
{
  "qid": "unique-question-id",
  "chain_id": "unique-chain-id", 
  "label": 0,  // Binary: 1=correct, 0=incorrect
  "orm_label": 0,
  "input_text": "Step-by-step reasoning trace...",
  "prompt": "Original problem statement",
  "steps": ["Step 1", "Step 2", ...],
  "final_answer": "Answer value",
  "meta": {
    "gold_answer": "Ground truth solution",
    "generated_by": "ORM-Repair-Synth-V1",
    "template": "error_type"
  }
}
```

**Curation Process (Weeks of Work):**
1. **Generation**: Synthetic reasoning traces with diverse error patterns
2. **Labeling**: Binary correctness annotation (1=correct, 0=incorrect)  
3. **Quality Control**: 
   - Manual review of reasoning validity
   - Verification against ground truth
   - Filtering of ambiguous cases
   - Removal of duplicates and near-duplicates
4. **Validation**: Base model log-probability analysis to confirm signal quality

**Quality Thresholds:**
- Positive examples: Logically sound reasoning leading to correct answer
- Negative examples: Clear errors in reasoning or incorrect final answer
- Filtering: Remove examples with log-prob inconsistent with label

### Phase 2: Pairwise Transformation

**Algorithm: Global Negative Sampling**

```python
def build_global_pairs(data, negatives_per_positive):
    """
    For each positive example, sample N negative examples globally.
    This creates diverse comparison pairs.
    """
    positives = [ex for ex in data if ex["orm_label"] == 1]
    negatives = [ex for ex in data if ex["orm_label"] == 0]
    
    pairs = []
    for pos in positives:
        sampled_negs = random.sample(negatives, k=negatives_per_positive)
        for neg in sampled_negs:
            pairs.append({
                "chosen": pos["input_text"],
                "rejected": neg["input_text"],
                "meta": {
                    "chosen_chain_id": pos["chain_id"],
                    "rejected_chain_id": neg["chain_id"],
                    "chosen_label": 1,
                    "rejected_label": 0
                }
            })
    return pairs
```

**Sampling Strategy Rationale:**
- **Train (8 neg/pos)**: Maximize training signal diversity
- **Val/Test (4 neg/pos)**: Balanced evaluation while maintaining diversity
- **Global sampling**: Ensures model learns general preferences, not task-specific patterns
- **Random seed**: Fixed for reproducibility (seed=42)

### Phase 3: Verification

**Post-Construction Checks:**
- ✅ All chosen traces have label=1
- ✅ All rejected traces have label=0  
- ✅ No duplicate pairs
- ✅ Chain IDs traceable to source
- ✅ Balanced length distribution across pairs

## 📝 Data Format

### Structure

Each example contains:

```json
{
  "chosen": "Step-by-step reasoning trace (correct)",
  "rejected": "Step-by-step reasoning trace (incorrect)", 
  "meta": {
    "chosen_chain_id": "pos-xxxxx",
    "rejected_chain_id": "synv1-xxxxx",
    "chosen_label": 1,
    "rejected_label": 0
  }
}
```

### Example

```json
{
  "chosen": "1. Calculate the cost of the unicorn piñata:\n2. Calculate the total cost of the Reese's:\n3. Calculate the total cost of the Snickers:\n4. Calculate the total cost of the Skittles:\n5. Add all the costs together:\n6. Compute the total cost step by step:\n7. Check the arithmetic result:\n8. Verification step:\n\nFinal Answer: 99",
  
  "rejected": "1. Assume that the weather forecast always grows linearly. However, assume an unrelated constant mistakenly.\n2. Therefore doubling the time doubles the value. However, assume an unrelated constant mistakenly.\n3. Ignore seasonal variations and round the result.\n4. Conclude with the projected incorrect value.\n\nFinal Answer: 15",
  
  "meta": {
    "chosen_chain_id": "pos-fe119ec6-f4b1-4710-80f9-8e64ced43c7e",
    "rejected_chain_id": "synv1-1b81a660-fa66-4cd0-9606-70b694486752",
    "chosen_label": 1,
    "rejected_label": 0
  }
}
```

### Field Descriptions

| Field | Type | Description |
|-------|------|-------------|
| `chosen` | string | Correct reasoning trace (label=1) |
| `rejected` | string | Incorrect reasoning trace (label=0) |
| `meta.chosen_chain_id` | string | Unique ID for chosen trace (traceable to source) |
| `meta.rejected_chain_id` | string | Unique ID for rejected trace (traceable to source) |
| `meta.chosen_label` | int | Always 1 (correct) |
| `meta.rejected_label` | int | Always 0 (incorrect) |

## 💻 Usage

### Loading the Dataset

```python
from datasets import load_dataset

# Load full dataset
dataset = load_dataset("akleshmishra/orm-pairwise-preference-pairs")

# Access splits
train_data = dataset["train"]
val_data = dataset["validation"] 
test_data = dataset["test"]

# Example usage
print(f"Train size: {len(train_data)}")
print(f"First example: {train_data[0]}")
```

### Training a Pairwise ORM

```python
import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

# Load dataset
dataset = load_dataset("akleshmishra/orm-pairwise-preference-pairs")

# Initialize model
base_model = AutoModel.from_pretrained("your-base-model")
tokenizer = AutoTokenizer.from_pretrained("your-base-model")

class PairwiseORM(nn.Module):
    def __init__(self, base_model):
        super().__init__()
        self.encoder = base_model
        # Freeze encoder
        for param in self.encoder.parameters():
            param.requires_grad = False
        
        # Trainable scoring head
        hidden_size = base_model.config.hidden_size
        self.score_head = nn.Sequential(
            nn.Linear(hidden_size, 256),
            nn.ReLU(),
            nn.Dropout(0.1),
            nn.Linear(256, 1)
        )
    
    def forward(self, input_ids, attention_mask):
        outputs = self.encoder(input_ids=input_ids, attention_mask=attention_mask)
        pooled = outputs.last_hidden_state.mean(dim=1)  # Mean pooling
        score = self.score_head(pooled)
        return score

# Training loop
def pairwise_loss(chosen_score, rejected_score):
    """Logistic pairwise ranking loss"""
    return -torch.log(torch.sigmoid(chosen_score - rejected_score)).mean()

# Prepare batch
def prepare_batch(examples):
    chosen_inputs = tokenizer(
        examples["chosen"],
        padding=True,
        truncation=True,
        max_length=512,
        return_tensors="pt"
    )
    rejected_inputs = tokenizer(
        examples["rejected"],
        padding=True,
        truncation=True,
        max_length=512,
        return_tensors="pt"
    )
    return chosen_inputs, rejected_inputs

# Train (simplified)
model = PairwiseORM(base_model)
optimizer = torch.optim.AdamW(model.score_head.parameters(), lr=1e-4)

for batch in dataloader:
    chosen_inputs, rejected_inputs = prepare_batch(batch)
    
    chosen_scores = model(**chosen_inputs)
    rejected_scores = model(**rejected_inputs)
    
    loss = pairwise_loss(chosen_scores, rejected_scores)
    
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
```

### Evaluation

```python
from sklearn.metrics import accuracy_score
import numpy as np

def evaluate_pairwise_accuracy(model, test_data):
    """
    Compute pairwise accuracy: fraction where score(chosen) > score(rejected)
    """
    correct = 0
    total = 0
    margins = []
    
    for example in test_data:
        chosen_score = model.score(example["chosen"])
        rejected_score = model.score(example["rejected"])
        
        margin = chosen_score - rejected_score
        margins.append(margin)
        
        if margin > 0:
            correct += 1
        total += 1
    
    accuracy = correct / total
    mean_margin = np.mean(margins)
    
    return {
        "accuracy": accuracy,
        "mean_margin": mean_margin,
        "median_margin": np.median(margins),
        "std_margin": np.std(margins)
    }
```

## 📊 Dataset Analysis

### Length Distribution

| Split | Avg Chosen Length | Avg Rejected Length |
|-------|-------------------|---------------------|
| Train | ~180 tokens | ~175 tokens |
| Validation | ~178 tokens | ~173 tokens |
| Test | ~182 tokens | ~176 tokens |

**Note**: Lengths are balanced to prevent length bias in learning.

### Error Pattern Distribution (Rejected Traces)

Common error types in rejected traces:
- Incorrect arithmetic calculations
- Logical fallacies in reasoning steps
- Missing or redundant steps
- Incorrect application of formulas
- Rounding errors
- Misinterpretation of problem constraints

### Chain ID Traceability

All examples include `chain_id` fields linking back to source pointwise dataset:
- `pos-*`: Positive (correct) reasoning traces
- `synv1-*`: Synthetic negative traces from ORM-Repair-Synth-V1

## 🔬 Experimental Results

Models trained on this dataset achieve:

- **Pairwise Accuracy**: 96.3% [95.3%, 97.1% CI]
- **Training Stability**: Converges in 800 steps
- **Anti-symmetry**: -0.998 correlation on label-swap test
- **Length Robustness**: 95.5%-99.7% across token ranges

See [Model Card](https://huggingface.co/LossFunctionLover/pairwise-orm-model) for full evaluation details.

## 🔗 Related Resources

- 📄 **Paper**: [ArXiv](link-to-arxiv) - "An Empirical Study of Robust Preference Learning under Minimal Supervision"
- 🤖 **Trained Model**: [HuggingFace](https://huggingface.co/LossFunctionLover/pairwise-orm-model)
- 💻 **Training Code**: [GitHub](https://github.com/Coder-12)
- 📊 **Source Pointwise Dataset**: Available upon request

## 📧 Contact & Citation

**Author**: Aklesh Mishra  
**Email**: akleshmishra7@gmail.com

If you use this dataset, please cite:

```bibtex
@misc{mishra2025orm-dataset,
  author = {Mishra, Aklesh},
  title = {ORM Pairwise Preference Pairs: A Curated Dataset for Training Outcome Reward Models},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/datasets/LossFunctionLover/orm-pairwise-preference-pairs}}
}

@article{mishra2025orm,
  title={An Empirical Study of Robust Preference Learning under Minimal Supervision},
  author={Mishra, Aklesh},
  journal={arXiv preprint arXiv:XXXX.XXXXX},
  year={2025}
}
```

## 📝 License

This dataset is released under the **Creative Commons Attribution 4.0 International (CC BY 4.0)** license.

You are free to:
- **Share**: Copy and redistribute the material
- **Adapt**: Remix, transform, and build upon the material

Under the following terms:
- **Attribution**: Give appropriate credit

## 🙏 Acknowledgments

This dataset represents **weeks of dedicated curation work** in preference learning for agentic reasoning. Special thanks to:

- The **MagiCore-Agentic** project for inspiring robust multi-step reasoning research
- The ML community for foundational work in reward modeling and RLHF
- All contributors to the open-source ecosystem

## ⚠️ Limitations & Considerations

### Known Limitations

1. **Domain**: Primarily math/reasoning tasks; may not generalize to all domains
2. **Synthetic negatives**: Some rejected traces are synthetically generated with error templates
3. **English only**: All reasoning traces are in English
4. **Length range**: Optimized for traces up to 512 tokens

### Ethical Considerations

- This dataset is designed for research purposes in improving AI reasoning
- Models trained on this data should not be used as sole arbiters of correctness in high-stakes decisions
- Users should validate model outputs independently in production settings

### Future Work

- [ ] Expand to multi-domain reasoning (code, science, etc.)
- [ ] Include multi-turn reasoning dialogues
- [ ] Add fine-grained error annotations
- [ ] Create multilingual versions

---

**Dataset Version**: 1.0  
**Last Updated**: November 27, 2025  
**Status**: ✅ Stable