WaylonJBrown/ling-3.0-tiny-sft-agentic

Ling-3.0-tiny LoRA (SFT-T + Nemotron SWE)

PEFT LoRA adapter on inclusionAI/Ling-3.0-tiny (7.9B hybrid MoE, 1.3B active). Trained for coding chat and agentic SWE tool use.

Training mix

Source	Count in mix
WlnJBrn09/SFT-1_10-1-26 (SFT-T, upsample 4×)	2,236
nvidia/Nemotron-SFT-SWE-v3.5	5,115
nvidia/Nemotron-RL-Agentic-SWE-Pivot-v1 (pass_rate >= 0.375, cap 8k)	8,000
Train total	15,351
SFT-T valid	36

SFT-T has no think traces (enable_thinking=False). Assistant-only labels, left-truncate to 4096 tokens.

Hyperparameters

• Hardware: 1× NVIDIA H100 80GB HBM3 (Runpod)
• LoRA r=16, alpha=32, dropout=0.05 on attention projections
• 1000 optimizer steps (~0.52 epoch), batch 2, grad accum 4, lr 2e-4 cosine, bf16
• Save every 250 steps

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "inclusionAI/Ling-3.0-tiny"
adapter = "WaylonJBrown/ling-3.0-tiny-sft-agentic"
tok = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base, trust_remote_code=True, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(model, adapter)