training/DATASETS.md

# Cortex A 0.5 — SFT dataset mix

Quality-filtered frontier distillations covering math, science, coding, agentic
tool-use, instruction following, long context, and roleplay. The r0b0tlab corpus
is the spine (native train/val/test). Complements fill Claude Opus 4.6–4.8
reasoning, DeepSeek V4 coding, and character roleplay (core mix is only ~1.1% dialogue).

**Loader note.** `r0b0tlab/...` README configs omit `data_files`, so
`load_dataset(id, "sft_balanced")` globs every parquet (including `glm47_native`)
and crashes with a schema CastError. We load `data_dir="data/sft_balanced"` only.

| Dataset | Domain | Teachers | Rows (approx) | Role |
|---|---|---|---|---|
| [r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation](https://huggingface.co/datasets/r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation) `sft_balanced` | mixed SOTA | Qwen3.8-Max, GLM-5.2, Kimi K3 | 52,205 train / 2,872 val / 2,860 test | **Required core.** Math 26%, code 25%, reasoning 18%, instruction 15%, agent/tools 10%, science, long-context, multilingual, dialogue |
| [Jackrong/DeepSeek-V4-Distill-8000x](https://huggingface.co/datasets/Jackrong/DeepSeek-V4-Distill-8000x) | coding / reasoning | DeepSeek V4 | ~7.7k | Coding + STEM traces |
| [Jackrong/Claude-opus-4.7-TraceInversion-5000x](https://huggingface.co/datasets/Jackrong/Claude-opus-4.7-TraceInversion-5000x) | reasoning | Claude Opus 4.7 | ~5k | High-quality inverted traces |
| [Jackrong/Claude-opus-4.6-TraceInversion-9000x](https://huggingface.co/datasets/Jackrong/Claude-opus-4.6-TraceInversion-9000x) | reasoning | Claude Opus 4.6 | ~9k | Broader Opus 4.6 coverage |
| [Roman1111111/claude-opus-4.6-10000x](https://huggingface.co/datasets/Roman1111111/claude-opus-4.6-10000x) | reasoning | Claude Opus 4.6 | ~9.6k | General reasoning transfer |
| [nohurry/Opus-4.6-Reasoning-3000x-filtered](https://huggingface.co/datasets/nohurry/Opus-4.6-Reasoning-3000x-filtered) | reasoning | Claude Opus 4.6 | ~2.3k | Filtered long reasoning |
| [ansulev/claude-opus-4.8-distill-5k](https://huggingface.co/datasets/ansulev/claude-opus-4.8-distill-5k) | reasoning | Claude Opus 4.8 | ~5k | Newest Opus line |
| [TeichAI/Claude-Opus-4.6-Reasoning-887x](https://huggingface.co/datasets/TeichAI/Claude-Opus-4.6-Reasoning-887x) | long reasoning | Claude Opus 4.6 | 887 | Dense high-effort traces (upweighted) |
| [lordx64/reasoning-distill-claude-opus-4-7-max](https://huggingface.co/datasets/lordx64/reasoning-distill-claude-opus-4-7-max) | reasoning | Claude Opus 4.7 Max | varies | Extra 4.7-max distill |
| [Jackrong/Qwen3.5-reasoning-700x](https://huggingface.co/datasets/Jackrong/Qwen3.5-reasoning-700x) | reasoning | Qwen3.5 | ~700 | In-family reasoning style |
| [beyoru/Aesir-Character-CoT-roleplay](https://huggingface.co/datasets/beyoru/Aesir-Character-CoT-roleplay) | roleplay | character CoT | ~2.0k | Persona + chain-of-thought RP (upweighted) |
| [ChengyuDu0123/HER-Dataset](https://huggingface.co/datasets/ChengyuDu0123/HER-Dataset) `sft_multi_turn` | roleplay | literary RP | 8k cap | Multi-turn character dialogue |

**Validation.** Held-out `validation` split from r0b0tlab only (2,872 traces, subsampled to 256 during the run for tok/s). Complements are train-only so val stays clean.

**QAT.** Unsloth `qat_scheme="int8-int4"`: 4-bit weights + 8-bit dynamic activation quantization.

**Vision.** Vision encoder is **frozen** this SFT phase so text tok/s stay high on T4s. Weights remain in the checkpoint (needed for the ~450MB 4-bit + mmproj target).