File size: 2,393 Bytes
681710f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
title: Opus Research
---
# Opus Research
Independent research on small language models — pretraining from scratch,
post-training, and the practical engineering around getting models to run on
constrained hardware.
We publish weights, datasets, and methodology, including the parts that did not
work.
## Models
**[opus-1.5](https://huggingface.co/opus-research/opus-1.5)** — 0.88B parameter
language model pretrained from scratch on 2× RTX 4090. Trained in 42 hours;
energy use documented in the model card.
**[opus-2.0](https://huggingface.co/opus-research/opus-2.0)** — successor model,
targeting a larger parameter count. **Training is currently paused: we do not
have the compute budget to continue.** The architecture and data pipeline are
ready; the run is blocked on GPU access rather than on research.
**[gemma-2-2b-thinking](https://huggingface.co/opus-research/gemma-2-2b-thinking)**
— experiment in adding explicit chain-of-thought to a base model that lacks it.
LoRA on Gemma 2 2B, trained on our own reasoning dataset. Training loss 3.56 →
0.69 over 566 steps. Exploratory; not benchmarked.
**[bernard-gpt-oss-20b-lora](https://huggingface.co/opus-research/bernard-gpt-oss-20b-lora)**
— style-transfer LoRA for gpt-oss-20b, a 21B mixture-of-experts model. Notable
mainly for the training notes: a four-run ablation showing that the checkpoint
with the *worst* validation loss was the correct one to ship, plus a set of
gpt-oss-specific engineering gotchas we could not find documented elsewhere.
## Datasets
**[opus-thinking-10k](https://huggingface.co/datasets/opus-research/opus-thinking-10k)**
— ~9,000 chain-of-thought examples, used to train `gemma-2-2b-thinking`.
**[opus-thinking](https://huggingface.co/datasets/opus-research/opus-thinking)**
— earlier iteration of the same dataset.
## What we work on
- Pretraining small models end-to-end on consumer hardware
- Post-training: LoRA, style transfer, capability injection
- Dataset construction and filtering methodology
- Quantization and CPU inference for models that would otherwise need a
datacenter
Our constraint is compute, not ideas. Everything above was produced on consumer
GPUs, and the ceiling that imposes is the main thing limiting what we publish
next.
## Contact
Open to collaboration, compute partnerships, and grant programs. Reach us
through the organization page.
|