Hugging Face's logo

BLANK
/
slm-rl-freeway

Text Generation
PEFT
Safetensors
Transformers
lora
reinforcement-learning
atari
slm-rl
freeway
Model card Files
xet
 Community
slm-rl-freeway
File size: 3,226 Bytes
305448e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eb4c5a7
305448e
 
 
eb4c5a7
305448e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eb4c5a7
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
---
library_name: peft
base_model: LiquidAI/LFM2.5-350M
pipeline_tag: text-generation
tags:
  - lora
  - peft
  - transformers
  - reinforcement-learning
  - atari
  - slm-rl
  - freeway
license: apache-2.0
---
# BLANK/slm-rl-freeway

PEFT LoRA adapter that warm-starts **Freeway** play for
[LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M)
in the [SLM-RL](https://github.com/CraftsMan-Labs/SLM-RL) workshop.

| | |
|---|---|
| **Game** | `freeway` |
| **Base model** | `LiquidAI/LFM2.5-350M` |
| **Adapter layout** | `adapter/` (PEFT `adapter_config.json` + weights) |
| **Training** | `reject_sft` on DQN teacher demos |
| **Champion generation** | 1 |
| **Promoted** | True (baked pack / SFT adopted as RL initialization) |
| **Dataset pack** | [BLANK/slm-rl-freeway](https://huggingface.co/datasets/BLANK/slm-rl-freeway) |
| **DQN teacher** | [BLANK/slm-rl-freeway-dqn](https://huggingface.co/BLANK/slm-rl-freeway-dqn) |

Paste `BLANK/slm-rl-freeway` as the playground **adapter URL** (and usually the same id
as the **dataset URL**).

## Install

```bash
pip install "transformers>=4.46" peft accelerate torch
```

## Load with transformers + PEFT

Weights live under the `adapter/` subfolder — pass `subfolder="adapter"`.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "LiquidAI/LFM2.5-350M"
ADAPTER = "BLANK/slm-rl-freeway"  # this repo

device = (
    "cuda" if torch.cuda.is_available()
    else "mps" if torch.backends.mps.is_available()
    else "cpu"
)
dtype = torch.bfloat16 if device != "cpu" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(BASE)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=dtype)
model = PeftModel.from_pretrained(model, ADAPTER, subfolder="adapter")
model.to(device).eval()

messages = [
    {"role": "system", "content": "You play Freeway. Reply with ACTION: <id>."},
    {"role": "user", "content": "Legal actions: 1) NOOP 2) UP\nChoose."},
]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=24, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

### Download only the adapter files

```python
from huggingface_hub import snapshot_download

path = snapshot_download("BLANK/slm-rl-freeway", allow_patterns="adapter/*")
# then: PeftModel.from_pretrained(base_model, f"{path}/adapter")
```

## Workshop / SLM-RL CLI

```bash
slm-rl evolve --game freeway \
  --dataset-url BLANK/slm-rl-freeway \
  --adapter-url BLANK/slm-rl-freeway \
  --dqn-url BLANK/slm-rl-freeway-dqn \
  --generations 2
```

## Train metrics (if recorded)

```json
{
  "eval": {
    "skipped": true
  },
  "gate": {
    "promoted": true,
    "reason": "baked pack / SFT adopted as RL initialization"
  },
  "train": {
    "loss": 0.093,
    "mean_token_accuracy": 1.0,
    "num_pairs": 20357
  }
}
```

Trained with [SLM-RL](https://github.com/CraftsMan-Labs/SLM-RL).