---
license: apache-2.0
language:
- en
tags:
- iko
- gpt2-medium
- conversational
- reddit
- qlora
- ties-merge
pipeline_tag: text-generation
base_model: gpt2-medium
datasets:
- dolma
- fineweb
---

# iko-2 (355M)

**iko-2** is the second model in the iko series — a GPT-2 Medium (355M parameters) language model that combines:

1. **iko-1 knowledge** (GPT-2 124M fine-tuned on 700K FineWeb documents) via distillation
2. **Reddit conversational style** from the Dolma v1.6 Reddit corpus

## Training Details

### Architecture
- **Base model:** GPT-2 Medium (355M parameters)
- **Training method:** 4-bit QLoRA with gradient checkpointing
- **LoRA config:** r=32, alpha=64, targets: ['c_attn', 'c_proj', 'c_fc']
- **Merge strategy:** TIES (TrIm, Elect Sign, and merge) with 80% density

### Training Data
- **Reddit Dolma v1.6** (~10000 examples, 85% of training mix)
- **iko-1 distillation corpus** (~1800 synthetic examples, 15% replay)
- **SuRe (Synthetic Replay)** for catastrophic forgetting prevention

### Hyperparameters
- Learning rate: 4e-05 with cosine schedule
- Layer-wise LR: embeddings 0.1×, bottom 0.3×, middle 1.0×, top 0.8×
- Warmup: 80 steps
- Effective batch size: 16
- Sequence length: 512
- Optimizer: 8-bit AdamW
- Training time: 15 minutes on T4 GPU

### Knowledge Transfer Pipeline
```
GPT-2 (124M) → [FineWeb fine-tune] → iko-1
                                         ↓ distillation
GPT-2 Medium (355M) → [QLoRA + Reddit + Replay] → [TIES merge] → iko-2
```

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("iko-01/iko-002")
tokenizer = AutoTokenizer.from_pretrained("iko-01/iko-002")

input_text = "The best thing about learning is"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Model Series
| Model | Parameters | Training Data | Method |
|-------|-----------|---------------|--------|
| iko-1 | 124M | FineWeb (700K docs) | QLoRA on GPT-2 |
| **iko-2** | **355M** | **Reddit + iko-1 distillation** | **QLoRA + TIES merge on GPT-2 Medium** |

## Limitations
- This model inherits biases present in Reddit data and GPT-2's pretraining corpus
- Not suitable for production use without additional safety fine-tuning
- Generated text may contain informal language reflecting Reddit's conversational style

## License
Apache 2.0