README.md

---
base_model: allenai/OLMo-2-0425-1B-Instruct
base_model_relation: finetune
library_name: peft
license: mit
tags:
  - lora
  - peft
  - transformers
  - audio-question-answering
  - correctness-assessment
  - orca
language:
  - en
pipeline_tag: text-classification
---

# ORCA — OLMo-2-1B-Instruct (Multinomial, seed 99)

ORCA (**O**pen-ended **R**esponse **C**orrectness **A**ssessment) scores the correctness of open-ended audio QA responses. Given a question, reference answer, candidate answer, and an LLM-generated rationale, it outputs a correctness score in [0, 1] and an uncertainty estimate.

**Paper:** [ORCA: Open-ended Response Correctness Assessment for Audio Question Answering](https://arxiv.org/abs/2512.09066) — accepted to *TACL 2026*  
**Code & usage:** [github.com/BUTSpeechFIT/ORCA](https://github.com/BUTSpeechFIT/ORCA)  
**Training data:** [BUT-FIT/orca-audio-qa-annotations](https://huggingface.co/datasets/BUT-FIT/orca-audio-qa-annotations)

## Model details

| Property | Value |
|---|---|
| Base model | `allenai/OLMo-2-0425-1B-Instruct` |
| LoRA rank / alpha | 128 / 128 |
| Loss function | Multinomial log-likelihood (5-class Likert) |
| Training seed | 99 |
| Training curriculum | Stage 1 (synthetic) → Stage 2 (LLM-judge) → Stage 3 (human) |
| Precision | bfloat16 |

## Quick start

```bash
pip install git+https://github.com/BUTSpeechFIT/ORCA.git
hf download BUT-FIT/orca-olmo-2-1b-multinomial --local-dir orca-olmo-1b
orca-infer --model_path orca-olmo-1b/model --data_jsonl your_data.jsonl --output_dir results/
```

See the [repository](https://github.com/BUTSpeechFIT/ORCA) for full usage, evaluation scripts, and the `download_and_infer.py` convenience script.

## Citation

```bibtex
@article{sedlacek-etal-2026-orca,
  title={ORCA: Open-ended Response Correctness Assessment for Audio Question Answering},
  author={Sedl\'{a}\v{c}ek, \v{S}imon and Barahona, Sara and Bola\~{n}os, Cecilia and
          Herrera-Alarc\'{o}n, Laura and Udupa, Sathvik and L\'{o}pez, Fernando and
          Ferner, Allison and Lozano-Diez, Alicia and Yusuf, Bolaji and Kesiraju, Santosh and
          Duraiswami, Ramani and \v{C}ernock\'{y}, Jan},
  howpublished={Accepted to Transactions of the Association for Computational Linguistics},
  year={2026},
  url={https://arxiv.org/abs/2512.09066}
}
```

## License

MIT License. See the [repository LICENSE](https://github.com/BUTSpeechFIT/ORCA/blob/main/LICENSE) for details.