Qwen3.5-9B-base-rednote-200K

A full-parameter fine-tuned version of Qwen3.5-9B-Base on the REDSearcher_SFT_10K dataset, with 200K context length support.

Unlike Qwen3.5-9B-rednote-200K which is fine-tuned from the instruct model, this variant starts from the base model (no prior instruction tuning), making it suitable for research purposes or further fine-tuning with custom instruction formats.

Model Details

• Base Model: Qwen/Qwen3.5-9B-Base
• Architecture: Qwen3_5ForConditionalGeneration (multimodal, vision + text)
• Training Type: Full fine-tune (all parameters)
• Dataset: Zchu/REDSearcher_SFT_10K
• Context Length: 200,000 tokens
• Precision: bfloat16
• Framework: Megatron-Swift v4.1.0.dev0

Training Configuration

Parameter	Value
Learning Rate	1e-5
LR Schedule	Cosine decay
Warmup Fraction	5%
Min LR	1e-6
Optimizer	Adam (β₁=0.9, β₂=0.95, ε=1e-8)
Weight Decay	0.1
Gradient Clipping	1.0
Global Batch Size	32
Micro Batch Size	1
Epochs	1
Max Sequence Length	200,000
Attention Backend	Flash Attention

Frozen Modules

• Vision encoder (visual.visual)
• Vision-language aligner (visual.visual.merger)

Gradient Checkpointing

• Granularity: full
• Method: uniform
• Recomputed modules: core_attn

Hardware & Parallelism

Configuration	Value
GPUs	8
Data Parallel Size	4
Tensor Parallel Size	2
Sequence Parallel	Enabled
Distributed Optimizer	Enabled
Optimizer CPU Offload	Enabled

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Estwld/Qwen3.5-9B-base-rednote-200K",
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Estwld/Qwen3.5-9B-base-rednote-200K")

inputs = tokenizer("Your prompt here", return_tensors="pt").to(model.device)
outputs = model.generate(inputs.input_ids, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

GPU Memory Requirements

• Inference (bf16): ~18 GB VRAM (single GPU)
• Fine-tuning (full): ~80+ GB VRAM per GPU (with full parameters, optimizer states, and activations at 200K context)

Limitations

• This is a research checkpoint at iteration 309. Further training steps may improve performance.
• Fine-tuned from the base model, so it lacks built-in instruction-following behavior. Consider applying chat templates or further instruction tuning for conversational use.

License

This model inherits the Apache 2.0 License from the base Qwen3.5-9B-Base model.