--- base_model: unsloth/Llama-3.2-1B-Instruct language: - en license: llama3.2 library_name: colar pipeline_tag: text-generation tags: - latent-reasoning - colar - reasoning - research --- # CoLaR — Coding (symbolic-execution reasoning), Llama-3.2-1B A **CoLaR** (Compressed Latent Reasoning) checkpoint for **Coding (symbolic-execution style reasoning)**, fine-tuned from `unsloth/Llama-3.2-1B-Instruct`. The model reasons in *compressed continuous latent embeddings* rather than explicit chain-of-thought tokens. ## Model - **Base model:** `unsloth/Llama-3.2-1B-Instruct` - **Framework:** CoLaR — *Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains* (arXiv:2505.16552). - **Domain:** Coding (symbolic-execution style reasoning) - **Warm-start:** colar-gsm (GSM8K CoLaR base) - **Files:** `colar_coding.ckpt`, `hparams.yaml` ## Training procedure Trained with the CoLaR supervised fine-tuning (SFT) recipe: the frozen base LLM is adapted with **q/v LoRA (rank 128, alpha 32)** plus a trainable **Latent Head** (3-layer MLP) and an **embedding-compression** module. The objective is next-token **cross-entropy** on the answer plus an **`embed_modeling_loss` (MSE)** that reconstructs the compressed reasoning-step embeddings (each latent token summarizes about `compression_factor` chain-of-thought tokens). Reinforcement learning (GRPO) is **disabled** — this is an SFT-only checkpoint. - **This checkpoint:** CoLaR SFT, compression factor 5, warm-started from colar-gsm; SFT only (RL off). ## Datasets - Coding reasoning mix (symbolic-execution style) ## How to load This is a **PyTorch-Lightning checkpoint** (weights under the top-level key `state_dict`) that fits the CoLaR scaffold — it is **not** directly `AutoModel`-loadable. Load the base model, splice this `state_dict` in with `strict=False`, and use the CoLaR runtime settings: ``` COLAR_EMB_STD=0.018 COLAR_COMPRESS= sep_token=### TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 ``` See the CoLaR repository (github.com/xiaomi-research/colar) for the exact loader. The shared GSM8K warm-start ancestor is the official `AlbertTan/CoLaR` release. *Research artifact for latent-reasoning study (small 1–1.5B model).*