README.md

---
license: mit
pipeline_tag: text-generation
library_name: transformers
---

<p align="center">
    <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/>
</p>
<p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>&nbsp;&nbsp; | &nbsp;&nbsp;🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope </a>&nbsp;&nbsp; | &nbsp;&nbsp; <a href="https://arxiv.org/abs/2606.15079">Tech Report </a>&nbsp;&nbsp; | &nbsp;&nbsp; 💻 <a href="https://github.com/inclusionAI/Ling-V2.5">GitHub</a></p>

# Ling-2.6-flash-base

Ling-2.6-flash-base is the base checkpoint behind the Ling-2.6-flash model. It is a flash-scale Mixture-of-Experts language model retrofitted from the Ling-2.0 base checkpoint with a hybrid linear attention design, continued pre-training, and long-context mid-training.

This release is intended for research, continued pre-training, distillation, and supervised or preference-based fine-tuning. It is not a chat-aligned assistant model. If you want an out-of-the-box instruction model, use the corresponding post-trained Ling-2.6-flash checkpoint instead.

## 1. Model Overview

Ling-2.6-flash-base is designed for efficient instant-response modeling with stronger long-context efficiency than the previous GQA-based Ling-2.0 generation. The core upgrade is a hybrid attention retrofit that combines Lightning Attention with MLA in a 7:1 ratio, together with a smooth migration pipeline from the original architecture.

Ling-2.6 base models are trained through approximately 9.6T tokens across migration pre-training, continued pre-training, and mid-training, with staged context extension from 4K to 256K. Ling-2.6-flash-base serves as the base checkpoint for the post-trained Ling-2.6-flash instant model.

## 2. Key Features

- Hybrid linear attention architecture combining Lightning Attention and MLA in a 7:1 ratio
- Flash-scale MoE backbone optimized for efficient serving and high token efficiency
- Long-context training pipeline extended to 256K context during mid-training
- Continued pre-training mixture covering agentic data, long-context data, knowledge-rich web data, math, code, and multilingual corpora
- Strong base-model quality across knowledge, math, code, reasoning, and long-context understanding benchmarks

## 3. Model Summary

| Item | Value |
| --- | --- |
| Architecture | Fine-grained MoE with hybrid linear attention |
| Parameter Scale | Total ~104B, Activated ~7.4B |
| Transformer layers | 32 |
| Routed experts per MoE layer | 256 |
| Shared experts per MoE layer | 1 |
| Active routed experts per token | 8 |
| Attention heads | 32 |
| Dense FFN layers | 1 |
| Hidden size | 4096 |
| Dense intermediate size | 9216 |
| Expert intermediate size | 1024 |
| KV LoRA rank | 512 |
| Q LoRA rank | 1536 |
| Layer group size | 8 |
| Positional encoding | Partial RoPE |
| Attention design | Lightning Attention + MLA, 7:1 ratio |
| Training recipe | Migration pre-training + continued pre-training + mid-training |
| Total training tokens | ~9.6T |
| Context training schedule | 4K -> 32K -> 256K |

## 4. Training Highlights

### Architecture Migration

The model is converted from the Ling-2.0 generation into the Ling-2.6-flash architecture through a multi-stage migration pipeline that includes:

1. Lightning Attention conversion
2. Linear warmup
3. MLA conversion
4. MLA warmup
5. Full continued pre-training

This retrofit is designed to preserve pre-trained capability while reducing long-context compute cost, KV-cache pressure, and decode latency.

### Data Mixture

The continued pre-training and mid-training stages include:

- Agentic corpus built from tool-use and coding environments
- Long-context corpus covering mathematics, web parsing, summarization, retrieval, and multi-hop reasoning
- General web knowledge data with targeted STEM and factual augmentation
- Math and code corpora
- Multilingual data spanning 21 languages


## 5. Base Model Evaluation

The following numbers are selected from the technical report and reflect base-model evaluation rather than chat-aligned or instruction-tuned performance.

| Benchmark | Ling-2.0-flash-base | Ling-2.6-flash-base |
| --- | ---: | ---: |
| MMLU | 82.98 | 84.13 |
| MMLU-Pro | 60.73 | 61.36 |
| GPQA | 35.35 | 37.88 |
| SimpleQA | 10.01 | 18.33 |
| C-SimpleQA | 49.43 | 63.53 |
| MMMLU | 62.76 | 64.76 |
| GSM8K | 90.60 | 91.89 |
| OmniMath | 28.30 | 29.90 |
| HumanEval-Plus | 83.54 | 81.10 |
| LiveCodeBench | 30.40 | 33.48 |
| BIRD-SQL | 38.69 | 38.40 |
| BBH | 84.82 | 85.06 |
| AutoLogic | 61.10 | 62.82 |
| LEval | 73.41 | 77.86 |
| LongBenchv2 | 33.40 | 34.19 |

Ling-2.6-flash-base shows broad gains over Ling-2.0-flash-base, especially on knowledge-oriented, reasoning-oriented, and long-context evaluations.



## 6. Intended Use

Recommended use cases:

- Continued pre-training
- Supervised fine-tuning for domain adaptation
- Preference optimization and RL post-training
- Distillation research
- Long-context and MoE systems research

Not recommended as-is for:

- Direct end-user chat deployment
- Safety-critical applications without additional alignment and evaluation
- Production use without post-training and task-specific validation


## 7. Limitations

- This is a base model and is not instruction-aligned.
- Outputs may be inaccurate, biased, incomplete, or unsafe without additional post-training.
- Long-context quality depends on the serving stack, positional scaling configuration, and prompt format used at inference time.
- The training mixture includes web-scale and synthetic data, so the model may reproduce factual errors or undesirable artifacts.
- Benchmark results in the technical report are collected under controlled internal evaluation settings and should not be treated as a guarantee of downstream production behavior.

## 8. Relationship to Other Releases

- [Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash): instruction and instant-response optimized model derived from this base.

If your goal is interactive assistant use rather than research on base checkpoints, the post-trained Ling-2.6-flash model is usually the better starting point.

## 9. Usage

This is a base checkpoint. The example below illustrates the loading pattern only.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "inclusionAI/Ling-2.6-flash-base"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
		model_name,
		torch_dtype=torch.bfloat16,
		trust_remote_code=True,
		device_map="auto",
)

prompt = "Summarize the benefits of hybrid linear attention."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
		**inputs,
		max_new_tokens=256,
		do_sample=False,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

For production inference, prefer serving stacks that support the released architecture and remote code path.

## 10. License

This model is released under the MIT License.