anonx3247/llm-trm-compressor-512

LLM-TRM Dimension Compressor

Sparse MLA-inspired Dimensional Encoder-Decoder For LLM-TRM Architecture

Model Details

• Architecture: Linear compression with weight-tied decompression
• Input dimension: 2048
• Compressed dimension: 512
• Compression ratio: 4.0x

Training Metrics

Metric	Value
MSE Loss	0.105578
Cosine Similarity	0.9185
Relative Error	0.3856
Variance Ratio	0.6832

Usage

import torch
from huggingface_hub import hf_hub_download
from src.models.compression import DimensionCompressor

# Download and load
checkpoint_path = hf_hub_download(repo_id="anonx3247/llm-trm-compressor-512", filename="compressor.pt")
checkpoint = torch.load(checkpoint_path, map_location="cpu")

# Initialize compressor
compressor = DimensionCompressor(
    d_model=2048,
    d_compressed=512,
)
compressor.load_state_dict(checkpoint["compressor"])

# Use
hidden_states = ...  # [B, L, 2048]
compressed = compressor(hidden_states)  # [B, L, 512]
reconstructed = compressor.decompress(compressed)  # [B, L, 2048]

Part of LLM-TRM

This compressor is part of the LLM-TRM project for integrating Tiny Recursive Models with language models.