README.md

---
base_model: Qwen/Qwen2.5-VL-3B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
license: other
language:
  - en
tags:
  - color-grading
  - lut
  - cube-lut
  - image-editing
  - instruction-following
  - qlora
  - lora
  - peft
  - qwen2-vl
  - vlm
---

# LUT-SLM — Stage-2 Generator Adapters (QLoRA over Qwen2.5-VL-3B)

QLoRA adapters for the **Stage-2 generator** of the LUT-SLM project: a small vision-language model
that turns *(source image + natural-language photo-editing instruction)* into a single global color
Look-Up Table (LUT). Given *"make it warmer and lift the shadows"* the model emits the tokens of a
17³ `.cube` LUT that bakes in exactly that look; given a request a single global LUT physically
**cannot** satisfy (e.g. *"remove the person on the left"*) it emits `<unsupported>` and refuses.

These adapters are trained on the companion dataset
**[`ericrcwu/LUT_SLM`](https://huggingface.co/datasets/ericrcwu/LUT_SLM)** (see that card for the full
data story). The Stage-1 request router lives in
**[`ericrcwu/LUT_SLM_interpreter`](https://huggingface.co/ericrcwu/LUT_SLM_interpreter)**.

> **Status — research artifacts, work in progress.** These are smoke-scale / bilevel-search run
> outputs, not a finalized release. Treat them as reproducible checkpoints from the collapse-fix and
> two-stage experiments.

## What's in this repo

Each subfolder is a self-contained PEFT adapter (adapter weights + tokenizer + chat template +
`adapter_manifest.json`), **except** `distill_r1_distilled_corpus/`, which holds a distilled data
corpus rather than weights.

| Subfolder | What it is |
|---|---|
| `p6_twostage_d0f9c744_smokefull/` | **Deployed generator.** P6 two-stage run adapter used by the webapp / Modal deploy (`deploy/modal_app.py`). mean train loss ≈ 1.677, 182 steps, lr 2e-4. |
| `bl_63cd1bf7_smokefull/` | One-stage full-run winner from the bilevel-over-SFT search. mean train loss ≈ 1.747, 162 steps, lr 3e-4. |
| `bl_a0ccbcff_smokefull/` | Bilevel baseline adapter (full smoke run). |
| `bl_a0ccbcff_smoke600/` | Bilevel baseline adapter (600-example smoke run). |
| `distill_r1_smokefull/` | Distillation round-1 adapter. |
| `distill_r1_distilled_corpus/` | Distilled corpus (`active_rows.jsonl`, `active_manifest.json`, `harvest_cache.jsonl`) — **data, not weights.** |

## Shared architecture & training recipe

All adapters share the same shape (per `adapter_manifest.json`):

- **Base model:** `Qwen/Qwen2.5-VL-3B-Instruct`, with the output embedding **resized to 151,924
  tokens** — the base vocab plus 259 special LUT tokens (`<lut_bos>`, `<lut_eos>`, `<unsupported>`,
  `<lut_000>`…`<lut_255>`). Embeddings are **tied**.
- **LoRA:** `r = 16`, `alpha = 32`, `dropout = 0.05`, targets
  `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`.
- **Quantization:** 4-bit QLoRA — `nf4`, double-quant, bf16 compute dtype.
- **Optim:** effective batch size 32 (per-device 1 × grad-accum 32), cosine schedule, 3% warmup,
  grad-checkpointing, 2 epochs, seed 0.
- **Frozen VQ tokenizer:** the 64 LUT code tokens decode via
  `tokenizer_version = vq_v2_srgbres_17to4_cb256_t64…` (encoder 17³ → 4³ latent → 64 codes over a
  256-entry codebook; decoder → a residual LUT added to the sRGB identity grid → `.cube`). The
  tokenizer artifacts themselves ship with the [`LUT_SLM`](https://huggingface.co/datasets/ericrcwu/LUT_SLM)
  corpus shards.

**Output grammar:** supported → `<lut_bos> <lut_###> ×64 <lut_eos>`; unsupported → `<unsupported>`.

## How to load

The adapter targets a **vocab-resized** base, so you must resize the base embeddings to 151,924 and
add the 259 special tokens *before* attaching the adapter (the `adapter_config.json` `base_model`
field points at a local `models/base_resized`, i.e. the resized base — not a Hub repo).

```python
from huggingface_hub import snapshot_download
d = snapshot_download("ericrcwu/LUT_SLM_sft_adapters",
                      allow_patterns=["p6_twostage_d0f9c744_smokefull/*"])
# 1) load Qwen/Qwen2.5-VL-3B-Instruct, 2) add the 259 special tokens + resize embeddings to 151924,
# 3) PeftModel.from_pretrained(base, f"{d}/p6_twostage_d0f9c744_smokefull").
# See notebooks/colab_lut_slm_inference.ipynb in the source repo for a runnable end-to-end example
# (vocab is reconstructed in memory, LUT codes decoded with the frozen tokenizer, image rendered).
```

## Licensing & provenance

`license: other`. The base model is governed by its own Qwen license; these adapters are derived from
the mixed-provenance **[`LUT_SLM`](https://huggingface.co/datasets/ericrcwu/LUT_SLM)** corpus, several
sources of which are **personal-use / non-redistribution** (see that dataset's licensing section).
This repository makes **no license claim** over the underlying LUTs or images. Research use;
verify each source family's original terms before any redistribution or commercial use.