| --- |
| base_model: unsloth/Llama-3.2-1B-Instruct |
| language: |
| - en |
| license: llama3.2 |
| library_name: colar |
| pipeline_tag: text-generation |
| tags: |
| - latent-reasoning |
| - colar |
| - reasoning |
| - research |
| --- |
| # CoLaR — Coding (symbolic-execution reasoning), Llama-3.2-1B |
|
|
| A **CoLaR** (Compressed Latent Reasoning) checkpoint for **Coding (symbolic-execution style reasoning)**, fine-tuned from `unsloth/Llama-3.2-1B-Instruct`. The model reasons in *compressed continuous latent embeddings* rather than explicit chain-of-thought tokens. |
|
|
| ## Model |
| - **Base model:** `unsloth/Llama-3.2-1B-Instruct` |
| - **Framework:** CoLaR — *Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains* (arXiv:2505.16552). |
| - **Domain:** Coding (symbolic-execution style reasoning) |
| - **Warm-start:** colar-gsm (GSM8K CoLaR base) |
| - **Files:** `colar_coding.ckpt`, `hparams.yaml` |
|
|
| ## Training procedure |
| Trained with the CoLaR supervised fine-tuning (SFT) recipe: the frozen base LLM is adapted with **q/v LoRA (rank 128, alpha 32)** plus a trainable **Latent Head** (3-layer MLP) and an **embedding-compression** module. The objective is next-token **cross-entropy** on the answer plus an **`embed_modeling_loss` (MSE)** that reconstructs the compressed reasoning-step embeddings (each latent token summarizes about `compression_factor` chain-of-thought tokens). Reinforcement learning (GRPO) is **disabled** — this is an SFT-only checkpoint. |
| - **This checkpoint:** CoLaR SFT, compression factor 5, warm-started from colar-gsm; SFT only (RL off). |
|
|
| ## Datasets |
| - Coding reasoning mix (symbolic-execution style) |
|
|
| ## How to load |
| This is a **PyTorch-Lightning checkpoint** (weights under the top-level key `state_dict`) that fits the CoLaR scaffold — it is **not** directly `AutoModel`-loadable. Load the base model, splice this `state_dict` in with `strict=False`, and use the CoLaR runtime settings: |
| ``` |
| COLAR_EMB_STD=0.018 COLAR_COMPRESS=<compression_factor> sep_token=### |
| TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 |
| ``` |
| See the CoLaR repository (github.com/xiaomi-research/colar) for the exact loader. The shared GSM8K warm-start ancestor is the official `AlbertTan/CoLaR` release. |
|
|
| *Research artifact for latent-reasoning study (small 1–1.5B model).* |
|
|
