File size: 2,448 Bytes
2306875 3834424 2306875 3834424 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | ---
license: apache-2.0
language:
- en
tags:
- code
- swe-bench
- agentic
- sft
library_name: transformers
pipeline_tag: text-generation
datasets:
- ricdomolm/mini-coder-trajs-400k
base_model:
- talkie-lm/talkie-web-13b-base
---
# talkie-web-coder
13B model fine-tuned on agentic software-engineering trajectories from
[SWE-smith](https://github.com/SWE-bench/SWE-smith), starting from the
`talkie-web` base (same architecture as `talkie-1930` but pre-trained on
web-style data). Tuned for the
[mini-swe-agent](https://github.com/SWE-bench/mini-swe-agent) interaction
format.
## SWE-bench-Verified-Working-Harbor pass@1
| metric | value |
|---|---|
| **pass@1** (n=3 independent eval runs) | **5.75% ± 1.04 pp** |
| per-run resolved (out of 446) | 31, 23, 23 |
Eval pipeline: vLLM (`--model-impl transformers --max-model-len 32768
--dtype bfloat16`) → mini-swe-agent (`mini-extra swebench`, temperature 0.7,
`max_tokens=4096`), graded with the swebench harness against
`ricdomolm/SWE-bench_Verified-Working-Harbor`.
## Training recipe
| | |
|---|---|
| Base model | `talkie-web-13b-base` (chat-token reinitialised) |
| Dataset | `talkie-web-swe-100k-64k` (100k SWE-smith trajectories, packed at 64k) |
| Trainer | TRL `SFTTrainer` via `accelerate` (8× A100) |
| Optimizer | `adamw_torch_fused`, β=(0.9, 0.95), ε=1e-8 |
| LR | 2e-5, `cosine_with_min_lr`, warmup 3% |
| Precision | bf16 |
| Weight decay | 0.1 |
| Max grad norm | 30 |
| Max length | 65,536 |
| Packing | `bfd` + padding-free |
| Loss | `completion_only_loss=1` (loss only on assistant tokens) |
| Steps | 2,016 (this is ckpt-2000) |
## Usage
This model uses custom modeling code (`modeling_talkie.py`,
`configuration_talkie.py`). Load with `trust_remote_code=True`:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"ricdomolm/talkie-web-coder",
trust_remote_code=True,
torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("ricdomolm/talkie-web-coder")
```
For agentic eval, serve with vLLM and drive with mini-swe-agent:
```bash
vllm serve ricdomolm/talkie-web-coder \
--model-impl transformers --max-model-len 32768 --dtype bfloat16
```
## Companion model
[`ricdomolm/talkie-1930-coder`](https://huggingface.co/ricdomolm/talkie-1930-coder)
— same recipe, same SFT data, but starting from a different base model.
Reaches 4.48% ± 0.69 pp on the same eval (n=5). |