README.md

---
language:
- multilingual
license: apache-2.0
license_name: kwaipilot-license
license_link: LICENSE
library_name: transformers
---
<div align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/61ee40a269351366e29972ad/KIYEa1c_WJEWPpeS0L_k1.png" width="100%" alt="Kwaipilot" />
</div>

<hr>


# SWE-Compass: Unified Evaluation Benchmark for Agentic Coding


---

## 🧭 Overview

**SWE-Compass** is a unified benchmark and dataset designed to evaluate **Agentic Coding** capabilities of large language models (LLMs) in realistic, multi-step software engineering workflows.

It bridges the gap between conventional static code-generation benchmarks and real-world, tool-driven development processes.  
Each instance corresponds to a reproducible issue-fixing or feature-implementation task that can be executed end-to-end: cloning a repository, applying patches, running tests, and verifying solutions.

### Key Features

- **≈ 2,000 curated tasks** from real GitHub issues and pull requests.  
- **8 task types × 8 scenarios × 10 programming languages** for comprehensive coverage.  
- **Fully reproducible pipeline** including setup scripts, environment dependencies, and test suites.  
- **Multi-dimensional evaluation** of correctness, reasoning trace, and agentic efficiency.

---

## 📁 Dataset Structure

```
SWE-Compass/
├─ data/
│  ├─ test.jsonl              # main evaluation set (~2,000 instances)
│  ├─ dev.jsonl               # optional validation split
│  └─ train.jsonl             # optional training data
├─ scripts/
│  ├─ setup_env.sh            # environment setup (dependency installation)
│  ├─ run_instance.py         # run one instance end-to-end
│  └─ eval_aggregate.py       # aggregate evaluation metrics
└─ README.md
```


```json
{
  "instance_id": "compass_01234",
  "task_type": "bug_fixing",
  "scenario": "mono_repo_ci",
  "language": "python",
  "difficulty": "medium",
  "source": {
    "repo": "owner/project",
    "commit": "abcdef123456",
    "issue_or_pr": "PR#13091",
    "gh_url": "https://github.com/owner/project/pull/13091"
  },
  "instruction": "Fix failing test in module X caused by Y...",
  "context_files": ["path/to/file1.py", "path/to/file2.py"],
  "tools_available": ["git", "pytest", "bash"],
  "evaluation": {
    "setup_cmds": ["pip install -e .", "pytest -q"],
    "test_cmd": "pytest -q",
    "timeout_sec": 1800
  },
  "reference_patch": "diff --git a/... b/...",
  "verified": true
}
```

# 🧪 Usage
## Load via 🤗 datasets

```python
from datasets import load_dataset

dataset = load_dataset("Kwaipilot/SWE-Compass", split="test")
print(len(dataset), dataset[0].keys())
Run Local Evaluation
```

## Run Local Evaluation

```bash
bash scripts/setup_env.sh
python scripts/run_instance.py --data data/test.jsonl --instance_id compass_01234
python scripts/eval_aggregate.py --data data/test.jsonl --runs ./runs/your_model_outputs
```

# 📊 Metrics

| Category | Metric | Description |
|-----------|---------|-------------|
| **Main** | Solved@1 / Solved@k | Fraction of tasks solved within k attempts |
| **Process** | Tool Calls / Latency | Efficiency and reasoning stability |
| **Failure Types** | Build Error / Test Fail / Timeout | Root-cause classification |

---


# 🧩 Citation

```bibtex
@article{xu2025swecompass,
  title   = {SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models},
  author  = {Jingxuan Xu and others},
  journal = {arXiv preprint arXiv:2511.05459},
  year    = {2025}
}
```

# 🤝 Contributing
We welcome community contributions — new verified instances, environment fixes, or evaluation scripts.
Please open a pull request or issue on this repository.