---
license: other
library_name: transformers
tags: [continued-pretraining, cpt, merged-lora, multilingual, cross-lingual-transfer]
language: [ky, kk, pl]
---

# CPT merged full models — run `t3pilot` (t3 cross-lingual experiment)

Standalone full models = base (meta-llama/Llama-3.1-8B) with the trained LoRA
adapter merged in (r=64, lr=5e-5, 30% English mixed stream, 2 epochs, frozen
embeddings/lm_head). Load directly with `AutoModelForCausalLM.from_pretrained`,
no PEFT. Per-language eval losses are in `manifest.json`.

## Load

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
mid = "the-cramer-project/cpt-models-t3"
sub = "Llama-3.1-8B/FT-KY"
model = AutoModelForCausalLM.from_pretrained(mid, subfolder=sub, torch_dtype="bfloat16")
tok   = AutoTokenizer.from_pretrained(mid, subfolder=sub)
```

## Models

| Subfolder | Base | Language | LoRA r | LR | Target eval loss |
|---|---|---|---|---|---|
| `Llama-3.1-8B/FT-KY` | meta-llama/Llama-3.1-8B | Kyrgyz | 64 | 5e-05 | 1.021923542022705 |
| `Llama-3.1-8B/FT-KZ` | meta-llama/Llama-3.1-8B | Kazakh | 64 | 5e-05 | 1.0028022527694702 |