Steve-KJ/Minerva-CPT-v0.2

Minerva CPT v0.2

Continued pre-training of Qwen3-Coder-Next 80B MoE on Korean corpora.

Training details

• Base model: Qwen/Qwen3-Coder-Next (80B parameters, MoE)
• Training: 2385 steps (~5B Korean tokens)
• Hardware: 8× NVIDIA B300 SXM6 AC (275GB VRAM)
• Framework: PyTorch 2.12 nightly + DeepSpeed ZeRO-3 (no offload)
• Trainer: HuggingFace Trainer + custom CPT pipeline

Checkpoint contents

• model-0000{1-4}-of-00004.safetensors — model weights (bf16, ~159GB)
• tokenizer.json, tokenizer_config.json — tokenizer
• config.json, generation_config.json — model config
• global_step2385/, rng_state_*.pth, scheduler.pt, training_args.bin — full state for resume
• zero_to_fp32.py — DeepSpeed helper to merge fp32 weights if needed

Usage

Inference (requires trust_remote_code=True):

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Steve-KJ/Minerva-CPT-v0.2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Steve-KJ/Minerva-CPT-v0.2",
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True,
)

Status

This is a continued pre-training (CPT) checkpoint — not yet instruction-tuned. SFT (Supervised Fine-Tuning) is the next step for chat / instruction following.

Trained by Azwell AI, 2026.