--- base_model: unsloth/Llama-3.2-1B-Instruct language: - en license: llama3.2 library_name: colar pipeline_tag: text-generation tags: - latent-reasoning - colar - reasoning - research --- # CoLaR — Coding / LiveCodeBench, Llama-3.2-1B A **CoLaR** (Compressed Latent Reasoning) checkpoint for **Coding (LiveCodeBench-style)**, fine-tuned from `unsloth/Llama-3.2-1B-Instruct`. The model reasons in *compressed continuous latent embeddings* rather than explicit chain-of-thought tokens. ## Model - **Base model:** `unsloth/Llama-3.2-1B-Instruct` - **Framework:** CoLaR — *Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains* (arXiv:2505.16552). - **Domain:** Coding (LiveCodeBench-style) - **Warm-start:** colar-coding-l1b - **Files:** `colar_coding_lcb.ckpt`, `hparams.yaml` ## Training procedure Trained with the CoLaR supervised fine-tuning (SFT) recipe: the frozen base LLM is adapted with **q/v LoRA (rank 128, alpha 32)** plus a trainable **Latent Head** (3-layer MLP) and an **embedding-compression** module. The objective is next-token **cross-entropy** on the answer plus an **`embed_modeling_loss` (MSE)** that reconstructs the compressed reasoning-step embeddings (each latent token summarizes about `compression_factor` chain-of-thought tokens). Reinforcement learning (GRPO) is **disabled** — this is an SFT-only checkpoint. - **This checkpoint:** CoLaR SFT, compression factor 5, warm-started from colar-coding-l1b; SFT only. ## Datasets - LiveCodeBench-style coding problems ## How to load This is a **PyTorch-Lightning checkpoint** (weights under the top-level key `state_dict`) that fits the CoLaR scaffold — it is **not** directly `AutoModel`-loadable. Load the base model, splice this `state_dict` in with `strict=False`, and use the CoLaR runtime settings: ``` COLAR_EMB_STD=0.018 COLAR_COMPRESS= sep_token=### TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 ``` See the CoLaR repository (github.com/xiaomi-research/colar) for the exact loader. The shared GSM8K warm-start ancestor is the official `AlbertTan/CoLaR` release. *Research artifact for latent-reasoning study (small 1–1.5B model).*