base_model: unsloth/Llama-3.2-1B-Instruct
language:
- en
license: llama3.2
library_name: colar
pipeline_tag: text-generation
tags:
- latent-reasoning
- colar
- reasoning
- research
CoLaR — Coding / LiveCodeBench, Llama-3.2-1B
A CoLaR (Compressed Latent Reasoning) checkpoint for Coding (LiveCodeBench-style), fine-tuned from unsloth/Llama-3.2-1B-Instruct. The model reasons in compressed continuous latent embeddings rather than explicit chain-of-thought tokens.
Model
- Base model:
unsloth/Llama-3.2-1B-Instruct - Framework: CoLaR — Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains (arXiv:2505.16552).
- Domain: Coding (LiveCodeBench-style)
- Warm-start: colar-coding-l1b
- Files:
colar_coding_lcb.ckpt,hparams.yaml
Training procedure
Trained with the CoLaR supervised fine-tuning (SFT) recipe: the frozen base LLM is adapted with q/v LoRA (rank 128, alpha 32) plus a trainable Latent Head (3-layer MLP) and an embedding-compression module. The objective is next-token cross-entropy on the answer plus an embed_modeling_loss (MSE) that reconstructs the compressed reasoning-step embeddings (each latent token summarizes about compression_factor chain-of-thought tokens). Reinforcement learning (GRPO) is disabled — this is an SFT-only checkpoint.
- This checkpoint: CoLaR SFT, compression factor 5, warm-started from colar-coding-l1b; SFT only.
Datasets
- LiveCodeBench-style coding problems
How to load
This is a PyTorch-Lightning checkpoint (weights under the top-level key state_dict) that fits the CoLaR scaffold — it is not directly AutoModel-loadable. Load the base model, splice this state_dict in with strict=False, and use the CoLaR runtime settings:
COLAR_EMB_STD=0.018 COLAR_COMPRESS=<compression_factor> sep_token=###
TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1
See the CoLaR repository (github.com/xiaomi-research/colar) for the exact loader. The shared GSM8K warm-start ancestor is the official AlbertTan/CoLaR release.
Research artifact for latent-reasoning study (small 1–1.5B model).