anonx3247/llm-trm-compressor

LLM-TRM Dimension Compressor

Sparse MLA-inspired Dimensional Encoder-Decoder For LLM-TRM Architecture

Model Details

• Architecture: Linear compression with weight-tied decompression
• Input dimension: 2048
• Compressed dimension: 256
• Compression ratio: 8.0x

Training Metrics

Metric	Value
MSE Loss	0.149304
Cosine Similarity	0.8816
Relative Error	0.4587
Variance Ratio	0.5532

Usage

import torch
from huggingface_hub import hf_hub_download
from src.models.compression import DimensionCompressor

# Download and load
checkpoint_path = hf_hub_download(repo_id="anonx3247/llm-trm-compressor", filename="compressor.pt")
checkpoint = torch.load(checkpoint_path, map_location="cpu")

# Initialize compressor
compressor = DimensionCompressor(
    d_model=2048,
    d_compressed=256,
)
compressor.load_state_dict(checkpoint["compressor"])

# Use
hidden_states = ...  # [B, L, 2048]
compressed = compressor(hidden_states)  # [B, L, 256]
reconstructed = compressor.decompress(compressed)  # [B, L, 2048]

Part of LLM-TRM

This compressor is part of the LLM-TRM project for integrating Tiny Recursive Models with language models.