---
license: apache-2.0
base_model: EleutherAI/pythia-1.4b
datasets:
- bigcode/the-stack
library_name: transformers
tags:
- memorization
- fine-tuned
---

# pythia-1.4b-code-run1

Pythia-1.4B fine-tuned for one epoch on a 1M-sequence (128-token) sample of The Stack.

This is a **run1** model. Extractable-memorization labels (k=50, l=50 greedy) from run1 are the labels used to train the memorization-prediction classifier. The matching **run2** model is fine-tuned on a disjoint 1M sample and provides the fresh held-out evaluation.

- Base model, EleutherAI/pythia-1.4b
- Dataset, The Stack (bigcode/the-stack)
- Training, 1 epoch, seq len 128, learning rate 5e-5, cosine schedule, seed 42