EleutherAI/affine-checkpoint-transfer-step54832

Affine Transform: EleutherAI/deep-ignorance-pretraining-stage-unfiltered@global_step54832 → EleutherAI/deep-ignorance-unfiltered@main

Learned affine transformation mapping hidden state activations from a source checkpoint to a target model.

Usage

from safetensors.torch import load_file
import torch.nn as nn
from huggingface_hub import hf_hub_download

# Download files
weights_path = hf_hub_download(repo_id="EleutherAI/affine-checkpoint-transfer-step54832", filename="affine_transforms.safetensors")
metadata_path = hf_hub_download(repo_id="EleutherAI/affine-checkpoint-transfer-step54832", filename="metadata.json")

# Load
import json
with open(metadata_path) as f:
    metadata = json.load(f)

weights = load_file(weights_path)
affine_transforms = {}
for layer_idx in metadata["layer_indices"]:
    linear = nn.Linear(metadata["hidden_dim"], metadata["hidden_dim"], bias=True)
    linear.weight.data = weights[f"layer_{layer_idx}.weight"]
    linear.bias.data = weights[f"layer_{layer_idx}.bias"]
    affine_transforms[layer_idx] = linear

MSE Metrics

Layer	MSE
5	0.061545
10	0.225621
15	0.552396
20	0.800604
25	1.295939
30	3.541077

Mean MSE: 1.079530

Training Details

• Source Model: EleutherAI/deep-ignorance-pretraining-stage-unfiltered@global_step54832
• Target Model: EleutherAI/deep-ignorance-unfiltered@main
• Hidden Dimension: 4096
• Ridge Alpha: 0.01
• Layers: [5, 10, 15, 20, 25, 30]
• Training Examples: 100000