Update README model card (Kelix technique report)

README.md

@@ -3,95 +3,127 @@ language:
   - en
 library_name: "muse"
 tags:
+  - unified-multimodal
+  - discrete-token
+  - image-understanding
   - text-to-image
   - image-generation
   - diffusion
   - diffusion-transformer
   - DiT
   - flow-matching
-  - unified-multimodal
-  - discrete-token
-  - instruction-following
+  - vision-language-model
+  - VLM
+  - next-block-prediction
   - kelix
   - sft
-base_model: "OpenOneRec/Kelix-DiT"
-pipeline_tag: text-to-image
-arxiv: "https://arxiv.org/pdf/2602.09843"
+base_model: "Qwen/Qwen3-8B"
+pipeline_tag: any-to-any
 ---
 
 # Kelix-SFT
 
-**Kelix-SFT** is the **supervised-fine-tuning (SFT) stage** checkpoint of the diffusion-based image de-tokenizer of **Kelix**, a fully discrete autoregressive unified multimodal model proposed by the OneRec Team. Built on top of [`OpenOneRec/Kelix-DiT`](https://huggingface.co/OpenOneRec/Kelix-DiT), it is fine-tuned on high-quality, rejection-sampled text-to-image data to improve **instruction-following** and **fine-grained control** while still rendering **1024×1024** images.
+[Paper](https://arxiv.org/pdf/2602.09843) | [Citation](#citation)
 
-> 📄 **Technical report**: <https://arxiv.org/pdf/2602.09843>
+<p align="center">
+  <img src="assert/fig3.png" alt="Kelix training pipeline: Kelix-Tok, Unified LLM, and Image DiT" width="90%">
+</p>
 
-## Background
+<p align="center"><b>Figure 1:</b> The auto-regressive training workflow of Kelix, including the Kelix Tokenizer, the Unified LLM, and the Image DiT de-tokenizer.</p>
 
-Most vision–language models (VLMs) rely on a hybrid interface — discrete text tokens paired with continuous ViT features — and are biased toward understanding. Fully autoregressive unified models that use **discrete** visual tokens, on the other hand, have historically suffered from an information bottleneck: a single discrete code carries far less information than the continuous embedding it replaces, which degrades multimodal understanding (especially on text-rich tasks such as OCRBench).
+## Introduction
 
-Kelix addresses this with a **multi-token vision tokenizer** (Kelix-Tok) that decomposes each patch embedding into `N` parallel discrete codes, expanding the coding capacity exponentially while keeping the LLM context length unchanged via sum pooling on the encoder side. The unified LLM (Qwen3-8B) is trained with a **Next-Block Prediction (NBP)** paradigm, and a diffusion-based **image de-tokenizer (Kelix-DiT)** turns the LLM's hidden states into high-resolution images — forming a modular *Tokenizer → LLM → Detokenizer* pipeline that unifies understanding and generation under a single autoregressive objective.
+**Kelix-SFT** is the **complete end-to-end release** of **Kelix** at the supervised-fine-tuning (SFT) stage — a fully discrete autoregressive unified multimodal model. This release bundles **all three components** of the Kelix pipeline:
+
+1. **Kelix-Tok** — the multi-token discrete vision tokenizer (NaViT encoder + multi-token VQ codebooks),
+2. **Kelix-LLM** — the unified Qwen3-8B backbone trained with Next-Block Prediction (NBP), with the vocabulary expanded by 65,536 visual entries,
+3. **Kelix-DiT (SFT)** — the SFT-stage diffusion-based image de-tokenizer, fine-tuned from [`OpenOneRec/Kelix-DiT`](https://huggingface.co/OpenOneRec/Kelix-DiT).
+
+Together they form a modular *Tokenizer → LLM → Detokenizer* system that unifies **multimodal understanding** and **image generation** under a single autoregressive objective. Use this repo if you want the full Kelix pipeline for both understanding and generation; use [`OpenOneRec/Kelix-DiT`](https://huggingface.co/OpenOneRec/Kelix-DiT) if you only need the pretraining-stage image de-tokenizer component.
 
 Kelix achieves state-of-the-art results among comparable-scale unified models on both understanding and generation benchmarks; notably, it reaches **86.7 on OCRBench**, matching continuous-feature VLMs and surpassing the previous best discrete model by **+23%**.
 
-## Model Overview
+## What's in this release
+
+| Component | Description | Init / Base |
+|---|---|---|
+| **Kelix-Tok** | NaViT-style vision encoder + multi-token VQ (`N=8` independent sub-codebooks, total size `S=65,536`, sum-pooled on the encoder side). Encodes images into discrete visual codes aligned with the LLM text space. | NaViT initialized from Keye-VL 1.5; codebooks K-means initialized, trained with SimVQ |
+| **Kelix-LLM** | Unified autoregressive backbone with Block Encoder / Block Decoder and a Next-Block Prediction (NBP) objective (text block size 2, visual block size `N+1=9`). | Qwen3-8B, vocabulary expanded by 65,536 visual entries |
+| **Kelix-DiT (SFT)** | SFT-stage diffusion-based image de-tokenizer (flow-matching, SANA-DiT based), conditioned on the LLM's last hidden states. | Fine-tuned from `OpenOneRec/Kelix-DiT` |
+| *(frozen, not included)* | DC-AE-F32C32 latent VAE (32× spatial downsampling → 32×32 latent) | Used as-is from the original DC-AE release |
 
-| Item | Value |
-|---|---|
-| Role | Diffusion-based image de-tokenizer (SFT stage) |
-| Base architecture | SANA-DiT (customized), fine-tuned from `OpenOneRec/Kelix-DiT` |
-| Training objective | Flow-matching |
-| Latent VAE | DC-AE-F32C32 (32× spatial downsampling → 32×32 latent) |
-| Condition | Last hidden states from the Kelix LLM (between vision start/end tokens) |
-| Output resolution | **1024 × 1024** (strictly constrained) |
-| Training stage | **SFT** (Stage 2 of 2) |
+> ⚠️ The **DC-AE-F32C32** VAE used to map pixels ↔ latents for Kelix-DiT is **frozen and not retrained**, so it is not duplicated in this release. Please load it from its original repo when running the image-generation path.
 
 ## Training (SFT Stage)
 
-This checkpoint corresponds to the **SFT Stage** of Kelix-DiT, building directly on the pretraining-stage checkpoint:
+This release corresponds to the **Stage 4: Supervised Fine-Tuning** of the Kelix-LLM pipeline and the **SFT Stage** of the Kelix-DiT pipeline, building directly on the annealed Stage-3 checkpoints:
 
+**Kelix-LLM (Stage 4 — SFT):**
+- **Data**: replayed high-quality Stage-3 data, task-specific chain-of-thought (CoT) data, targeted image-generation data (attribute / subject / scene / style / perspective control), and task-specific enhancement data (complex OCR, math reasoning, STEM).
+- **Filtering**: rejection sampling is retained to filter noisy samples and mitigate hallucinations.
+- **Objective**: preserves the unified multimodal capability while boosting task-specific performance, keeping understanding and generation in balance.
+
+**Kelix-DiT (SFT Stage):**
 - **Data**: high-quality text-to-image data filtered via **rejection sampling** to remove low-quality pairs and mitigate hallucinations.
 - **Fine-grained control**: curated to cover key control dimensions — attribute control (color, size), subject control (quantity, type), scene generation (indoor/outdoor, time of day), artistic style (oil painting, sketch), and perspective (top-down, close-up).
 - **Target image**: strictly constrained to **1024×1024**.
 - **Optimization**: all DiT parameters are updated; the Kelix LLM, DC-AE encoder, and DC-AE decoder are frozen.
 
-This stage significantly improves the model's adherence to diverse and precise textual instructions, enabling high-fidelity generation that strictly complies with complex prompts.
+Together, these significantly improve the model's adherence to diverse and precise textual instructions, enabling high-fidelity generation that strictly complies with complex prompts while maintaining strong multimodal understanding.
 
 ## Usage
 
-Kelix-SFT is designed to be driven by the hidden states of the Kelix unified LLM. A typical generation pipeline is:
+This release is the full Kelix pipeline and can be used for **both understanding and generation** end-to-end (the only external dependency is the frozen DC-AE-F32C32 VAE for the generation path).
+
+**Image understanding (VQA / OCR / reasoning):**
+1. Encode the input image(s) with **Kelix-Tok** into `N`-way discrete visual codes, sum-pooled into a single composite token per patch on the encoder side.
+2. Feed the visual codes + text tokens into **Kelix-LLM** (Block Encoder → NBP backbone → Block Decoder).
+3. Decode the predicted text tokens via the standard text tokenizer.
 
-1. Feed a text prompt (and optional images) into the **Kelix LLM**, which autoregressively produces last hidden states `{h_*}` for the image blocks.
-2. Use `{h_*}` as the semantic condition (y-embedder input) for Kelix-SFT.
-3. Run flow-matching denoising in the DC-AE latent space (32×32) and decode with DC-AE to obtain a 1024×1024 image.
+**Image generation (text-to-image, 1024×1024):**
+1. Feed a text prompt (and optional images) into **Kelix-LLM**, which autoregressively produces last hidden states `{h_*}` for the image blocks.
+2. Use `{h_*}` as the semantic condition (y-embedder input) for **Kelix-DiT (SFT)**.
+3. Run flow-matching denoising in the DC-AE latent space (32×32) and decode with the frozen DC-AE to obtain a 1024×1024 image.
 
-> ⚠️ This checkpoint is a **component** of the Kelix pipeline, not a standalone text-to-image model. To generate images end-to-end you also need the Kelix unified LLM and the frozen DC-AE-F32C32 VAE. For broader scenario generalization, see the pretraining-stage checkpoint [`OpenOneRec/Kelix-DiT`](https://huggingface.co/OpenOneRec/Kelix-DiT).
+> 💡 For broad-scenario generalization on the generation side, the pretraining-stage de-tokenizer [`OpenOneRec/Kelix-DiT`](https://huggingface.co/OpenOneRec/Kelix-DiT) can be swapped in for Kelix-DiT (SFT); Kelix-Tok and Kelix-LLM in this release are compatible with both.
 
 ## Key Results
 
-Kelix (8B, with the SFT-stage de-tokenizer) image-generation results:
+Kelix (8B) results with this release:
+
+**Image understanding**
 
-| Benchmark | Score |
-|---|---|
-| GenEval (Overall) | **87.6** |
-| WISE (Overall) | **57.0** |
-| DPG-Bench (Overall) | **85.5** |
+| Benchmark | SEED-Bench | RealWorldQA | MMBench-EN | AI2D | MMMU | MathVista | ChartQA | TextVQA | OCRBench |
+|---|---|---|---|---|---|---|---|---|---|
+| Score | 76.0 | 72.1 | 80.2 | 82.4 | 54.1 | 76.5 | 83.0 | 81.4 | **86.7** |
+
+**Image generation**
+
+| Benchmark | GenEval (Overall) | WISE (Overall) | DPG-Bench (Overall) |
+|---|---|---|---|
+| Score | **87.6** | **57.0** | **85.5** |
 
 Highlights (see the technical report for full tables):
-- GenEval **87.6** — SOTA among discrete-tokenization unified models, **+0.6** over the 27B Qwen-Image.
-- WISE **57.0** — 2nd only to Nextflow (7B, 59.0), beating all continuous-tokenization unified models and larger dedicated T2I models (e.g., FLUX.1-dev 12B, 50.0).
-- DPG-Bench **85.5** — competitive with the SOTA X-Omni (7B, 87.7) without using reinforcement learning.
+- **OCRBench 86.7** — matching Qwen2.5-VL-7B (86.4), **+23%** over the previous best discrete unified model X-Omni (70.4). First discrete-token unified model to close the gap with continuous-feature VLMs on text-rich tasks.
+- **MathVista 76.5** — surpasses the 14B Bagel (73.1) and the 30B Manzano (73.3) despite a much smaller parameter count.
+- **GenEval 87.6** — SOTA among discrete-tokenization unified models, **+0.6** over the 27B Qwen-Image.
+- **WISE 57.0** — 2nd only to Nextflow (7B, 59.0), beating all continuous-tokenization unified models and larger dedicated T2I models (e.g., FLUX.1-dev 12B, 50.0).
+- **DPG-Bench 85.5** — competitive with the SOTA X-Omni (7B, 87.7) **without** using reinforcement learning.
 
 ## Citation
 
+If you find Kelix useful, please cite our technical report.
+
 ```bibtex
-@techreport{kelix2026,
+@article{kelix2026,
   title   = {Kelix Technique Report: Closing the Understanding Gap of Discrete Tokens in Unified Multimodal Models},
-  author  = {OneRec Team},
+  author  = {Kuaishou Technology},
+  journal = {arXiv preprint arXiv:2602.09843},
   year    = {2026},
-  url     = {https://arxiv.org/pdf/2602.09843}
+  url     = {https://arxiv.org/abs/2602.09843}
 }
 ```
 
 ## License
 
-Please contact the OneRec Team for the license of the Kelix series. The base SANA-DiT and DC-AE components are subject to their respective original licenses.
+Please contact the OneRec Team for the license of the Kelix series. The base components (Qwen3-8B, SANA-DiT, DC-AE, Keye-VL / NaViT) are subject to their respective original licenses.