iko-004 (iko-4)

Continued Pre-training of GPT-2 Medium (355M) with SuRe-Interleave protection against catastrophic forgetting

• Base model lineage: gpt2-medium → iko-002 → iko-003 → iko-004
• Latest training data: Dolma Reddit shard reddit-v5-dedupe-pii-nsfw-toxic-0074.json.gz
• Technique: Parameter-efficient continued pre-training using LoRA + SuRe-Interleave recipe
  (Merge-and-Grow + surprise-driven micro-replay from previous shard + lower LR schedule)
• Hugging Face: iko-01/iko-004
• Family: iko (independent continued-pretraining series on filtered Reddit data)

Model Description

iko-004 is the fourth model in the iko series — a chain of continued pre-training runs starting from the original GPT-2 Medium checkpoint.

Each step takes one new Reddit shard from the Dolma corpus (deduplicated, PII/NSFW/toxic filtered) and continues unsupervised language modeling while trying to preserve as much previous knowledge as possible.

iko-004 applies a lightweight version of the SuRe-Interleave recipe described in the associated research note/paper draft:

• Merge-and-Grow: previous model (iko-003) is used as the new backbone; a fresh LoRA adapter is trained
• Micro-replay buffer: ~4k high-quality / diverse examples streamed from the previous shard (0053) are interleaved during training
• Lower learning rate and cosine schedule to reduce interference
• 4-bit quantization (QLoRA-style) + packing + gradient checkpointing to fit on a single T4 GPU

Goal: achieve longer, more stable continued pre-training without rapid degradation of earlier Reddit styles/domains.

Intended Use & Limitations

Primary intended use

• Research on continued pre-training / domain adaptation of small LMs
• Studying forgetting in sequential unsupervised adaptation on social media text
• Cheap & reproducible mid-training experiments on consumer hardware

Not intended for

• Direct chat / instruction use (no SFT or RLHF applied)
• Production deployment without heavy safety filtering
• Tasks that require strong reasoning, factuality or up-to-date knowledge

Known limitations

• Still inherits GPT-2 Medium's original weaknesses (small context 1024→512 used here, repetition, poor arithmetic…)
• Reddit data → can generate toxic, sarcastic, low-quality or harmful text
• No alignment / moderation applied
• Forgetting still happens over very long training chains (SuRe helps but does not eliminate it completely)
• Only ~80–90 minutes of wall-clock training on T4 → modest perplexity improvement

Training Details

Field	Value
Base model	iko-003 (itself continued from iko-002)
Architecture	GPT-2 Medium (355M)
Adapter	LoRA (r=32, α=64, dropout=0.05)
Target modules	c_attn, c_proj, c_fc
Quantization	4-bit NF4 + double quant
Sequence length	512
Effective batch size	16 (2 × 8 gradient accumulation)
Learning rate	2.2 × 10⁻⁵
Optimizer	AdamW 8-bit
Scheduler	cosine decay + warmup
Training steps	~250–270 (≈ 82 min on Colab T4)
Replay buffer	~4000 examples from previous shard (0053)
New data consumed	~62k documents from shard 0074
Hardware	Google Colab Tesla T4 (single GPU)
Framework	transformers + peft + trl + bitsandbytes
Date	March 2026

Evaluation (self-reported – preliminary)

No formal zero-shot / few-shot leaderboard evaluation yet.

Preliminary manual checks show:

• Maintains coherent Reddit-like style from iko-003
• Slightly better fluency on longer text than naive continued training without replay
• Still produces occasional toxic / unsafe completions (as expected)

You are very welcome to run perplexity on held-out Reddit text or downstream tasks and open a discussion / PR with results.

How to Use

from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

model_id = "iko-01/iko-004"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=180,
    do_sample=True,
    temperature=0.85,
    top_p=0.92,
    repetition_penalty=1.05
)

prompt = """Write a sarcastic reddit comment about someone overusing AI emojis:"""

out = pipe(prompt)[0]["generated_text"]
print(out)

Bias, Risks & Ethical Notes

• Trained on filtered Reddit → still contains sarcasm, dark humor, controversial opinions, slurs (even after NSFW/toxic filtering)
• No safety fine-tuning → model can generate offensive, illegal or dangerous content when prompted adversarially
• Use with responsibility: do not deploy in open-ended chat interfaces without strong moderation

Citation / How to reference

If you use iko-004 in academic work or public experiments, feel free to cite:

@misc{iko-series-2026,
  author       = {Younes (machkour)},
  title        = {iko — Continued Pre-training Series on Filtered Reddit Shards},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/iko-01/iko-004}}
}

Acknowledgments

• Hugging Face (transformers, peft, accelerate, datasets, hub)
• bitsandbytes & Unsloth community (fast QLoRA training)
• AllenAI Dolma team — reddit-v5-dedupe-pii-nsfw-toxic shards
• Previous iko steps built with help from open-source Colab patterns