README_en.md

metadata

frameworks:
  - Pytorch
license: Apache License 2.0
tasks:
  - text-generation

AI-Flow-Ruyi (如意大模型)

中文   |   English
🤗 Hugging Face   |   🤖 ModelScope   |    📑  Paper

Long long ago...

Deep within the Loong King's palace lay a divine rod, capable of infinite transformations, shrinking and growing at will. One day, finding himself at leisure, the Loong King mused to the rod: "With such formidable power, if only you could aid our Loong tribe in new endeavors." Suddenly, the rod spoke in reply: "I have an idea. What if this transformative ability were used to help humankind solve their problems?" No sooner said than done. The rod shimmered and transformed into an immensely powerful model, its "capabilities" freely scaling to match the complexity of any challenge. Beholding this marvel, the Loong King exclaimed, "Why, this is a true 'Ruyi' treasure—a wish-fulfilling aid to resolve all manner of troubles!" Thus, he named it "Ruyi" and sent it forth into the human world to offer its assistance.

News

• 🎉🎉[2025/7/4]：AI-Flow-Ruyi-7B-Preview released!

Introduction

AI-Flow-Ruyi is a Familial Model developed by the AI Flow team under the leadership of Professor Li Xuelong, CTO, Chief Scientist of China Telecom, and President of the Institute of Artificial Intelligence (TeleAI), China Telecom. Designed for next-generation "Device-Edge-Cloud" model service architectures, its core innovation lies in shared familial parameters across large and small models. Leveraging an early-exit mechanism, the system dynamically routes queries to branch models of appropriate parameter sizes based on problem complexity. These branches operate independently while enabling information sharing and seamless transitions through their shared features. Combined with distributed Device-Edge-Cloud deployment, this facilitates collaborative inference within the model family, significantly enhancing distributed reasoning efficiency.

AI-Flow-Ruyi-7B-Preview

The AI-Flow-Ruyi-7B-Preview was released on July 4th. Its largest branch operates at 7B parameters and can differentiate into early-exit variants with equivalent capacities of 3B, 4B, 5B, and 6B parameters.

Key branch specializations:

• 3B/4B branches: Optimized for simple dialogue scenarios, delivering faster response times with minimal resource consumption
• 5B/6B branches: Targeting daily general-purpose tasks, striking a balance between capability and responsiveness
• 7B branch: Designed for complex problem-solving, exhibiting more well-rounded capabilities across multiple dimensions – though with moderately slower inference speeds and higher resource demands

Position No.	Early-Exit Layer	Equivalent Model Size	Branch Designation	Target Scenario
1	Layer 11	3B	AI-Flow-Ruyi-7B-E3B	Simple dialogue
2	Layer 15	4B	AI-Flow-Ruyi-7B-E4B	Simple dialogue
3	Layer 19	5B	AI-Flow-Ruyi-7B-E5B	Daily tasks
4	Layer 23	6B	AI-Flow-Ruyi-7B-E6B	Daily tasks
5	Layer 27	7B	AI-Flow-Ruyi-7B-E7B	Complex problems

Training process

Prior to training initiation, we initialized parameters for the 7B main branch using Qwen team's pre-trained Qwen2.5-7B (pre-trained on 18 trillion high-quality tokens). For early-exit branches, decoder layers were initialized with parameters from the subsequent layer of their respective early-exit positions.

Following initialization, we conducted multi-branch joint pre-training with approximately 400 billion tokens on proprietary high-quality datasets, resulting in the AI-Flow-Ruyi-7B-Base foundation model.

Subsequently, we performed multi-branch joint instruction-following fine-tuning across all branches using ~1.2 million high-quality instruction samples, yielding the AI-Flow-Ruyi-7B-Preview.

Performance review

We conduct a review based on OpenCompass and its official configuration files on multiple datasets in a 0-shot manner. The evaluation results show that the 7B master branch is basically equal to Qwen2.5-7B-Instruct in terms of general-purpose task performance.

Common tasks review

Model	MMLU	MMLU-Pro	CMMLU	ARC-c	BBH	Mean
Qwen3-8B(think)	74.78	66.02	76.33	63.39	60.68	68.24
Qwen2.5-7B-Instruct	70.88	56.33	75.71	86.44	51.51	68.17
Llama-3.1-8B-Instruct	53.16	45.36	51.65	83.73	72.47	61.27
AI-Flow-Ruyi-7B-E7B(ours)	87.19	59.78	48.14	69.83	74.47	67.88

Code tasks review

Model	MBPP	HumanEval	LiveCodeBench	Mean
Qwen3-8B(think)	78.60	84.76	63.10	75.49
Qwen2.5-7B-Instruct	70.82	84.15	34.55	63.17
Llama3.1-8B-Instruct	68.48	63.41	8.15	46.68
AI-Flow-Ruyi-7B-E7B(ours)	66.93	64.63	30.01	53.86

STEM tasks review

Model	Math	GPQA	GSM-8K	Mean
Qwen3-8B(think)	83.84	38.38	93.03	71.75
Qwen2.5-7B-Instruct	73.66	35.35	88.48	65.83
Llama3.1-8B-Instruct	49.22	25.25	85.82	53.43
AI-Flow-Ruyi-7B-E7B(ours)	44.94	24.75	81.65	50.45

At the same time, the performance of each early exit branch shows a monotonically increasing trend with the number of equivalent parameters.

Model	MMLU	MMLU-Pro	CMMLU	ARC-c	BBH	Mean
AI-Flow-Ruyi-7B-E3B(ours)	66.93	44.70	19.80	40.00	32.29	40.74
AI-Flow-Ruyi-7B-E4B(ours)	78.86	48.60	26.51	58.98	41.98	50.99
AI-Flow-Ruyi-7B-E5B(ours)	75.34	49.13	33.91	65.76	64.48	57.72
AI-Flow-Ruyi-7B-E6B(ours)	84.58	53.06	33.94	73.22	47.33	58.43
AI-Flow-Ruyi-7B-E7B(ours)	87.19	59.78	48.14	69.83	74.47	67.88

Usage

Step 1. Create and activate a virtual environment

conda create -n ruyi python=3.12
conda activate ruyi

Step 2. Clone this warehouse to local

git clone https://github.com/TeleAI-AI-Flow/AI-Flow-Ruyi.git
cd AI-Flow-Ruyi

Step 3. Installation from source (PS: flash_attn compilation and installation is slow, it is recommended to move to the official repository to download whl manual installation)

pip install -e .

Step 4. Download model weights

git clone https://www.modelscope.cn/TeleAI-AI-Flow/AI-Flow-Ruyi-7B-Preview0704.git models/AI-Flow-Ruyi-7B-Preview0704

Step 5. Run Demo

python demo.py

View demo code

import torch
from ruyi.global_var import set_global_val
from transformers import GenerationConfig
from transformers import AutoModelForCausalLM, AutoTokenizer


model_path = f"models/AI-Flow-Ruyi-7B-Preview0704"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True, attn_implementation='flash_attention_2', torch_dtype=torch.bfloat16).to('cuda')


generation_config = GenerationConfig(
    do_sample=True,                  
    top_k=30,                        
    top_p=0.95,                      
    temperature=0.6,                 
    repetition_penalty=1.2,          
    no_repeat_ngram_size=3,          
    max_new_tokens=8192
)

# input text
messages = [
    {"role": "user", "content": "Introduce yourself."},
]

# Apply chat_template template
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")

# Model Generation
with torch.no_grad():
    # Setting the early exit point
    # - 11: First early exit point corresponding to about 3B.
    # - 15: second early exit point, corresponding to approximately 4B.
    # - 19: third early exit point, corresponding to about 5B.
    # - 23: fourth early exit point, corresponding to approximately 6B.
    # - 27: fifth early exit point, corresponding to about 7B.
    set_global_val("early_exit_point", 11)  

    output = model.generate(
        inputs["input_ids"].to('cuda'),
        generation_config=generation_config
    )

# Decode and print results
generated_text = tokenizer.decode(output[0], skip_special_tokens=False)
print(generated_text)

Citation

@misc{an2025aiflowperspectivesscenarios,
      title={AI Flow: Perspectives, Scenarios, and Approaches}, 
      author={Hongjun An and and Wenhan Hu and Sida Huang and Siqi Huang and Ruanjun Li and Yuanzhi Liang and Jiawei Shao and Yiliang Song and Zihan Wang and Cheng Yuan and Chi Zhang and Hongyuan Zhang and Wenhao Zhuang and Xuelong Li},
      year={2025},
      eprint={2506.12479},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.12479}, 
}