ryan-0608/MoS-DFlash-Qwen3-8B-Training-Archive

MoS DFlash Qwen3-8B Training Archive

This repository is the reusable checkpoint archive for the Qwen3-8B request-level MoS/DFlash experiments. These are draft-model checkpoints, not standalone causal language models.

Pinned dependencies

• Target model: Qwen/Qwen3-8B@b968826d9c46dd6066d109eabc6255188de91218
• D0 initializer: z-lab/Qwen3-8B-DFlash-b16@9b41424b7109f9c5413454f481b09a82b85333f4
• Target tokenizer, embedding, and LM head are runtime dependencies and are not redistributed here.

Archive layout

U1 contains the 8B Generalist plus the four matched joint/MLP-only MoS arms:

dflash_legacy.py
registry/
schema/
experiments/
  generalist/mixed-800k/
  joint-mos/d0-init/
  joint-mos/generalist-init/
  mlp-only-mos/d0-init/
  mlp-only-mos/generalist-init/
provenance/original-code/<sha256>/

Each checkpoint directory contains its config, complete safetensors shard set, and index where required. In U1, auto_map.AutoModel selects the repository root module:

• dflash_legacy.DFlashDraftModel for code SHA-256 08625e679c8a2f55a919db108dbec6591fbeb9bd4c012ee387757cd578b2ef8a.

The complete hashes are in registry/checksums.sha256 and the artifact manifest. The exact original checkpoint-local code is retained once under provenance/. Later specialist, attention-only, and integrated-router batches must pass their own manifests and verification gates before they are added.

Loading one checkpoint

Pin a verified release tag or commit:

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "ryan-0608/MoS-DFlash-Qwen3-8B-Training-Archive",
    subfolder="<manifest load_subfolder>",
    revision="<verified commit or tag>",
    trust_remote_code=True,
)

The target model must be loaded separately. Do not use these draft weights as a standalone text-generation model.

Resume state

U1 does not place FSDP optimizer state in this loadable model repository. Reviewed resume bundles are planned for a separate hash-addressed storage archive; its final location will be documented only after that archive exists and is verified.

• bitwise-exact is used only after an actual parity test with accumulation_steps=1.
• stateful-non-bitwise means fp32 masters, AdamW moments, and scheduler can be restored, but the original accumulation geometry prevents a bitwise claim.
• Raw source training_state.pt files are not published. Only the reviewed, path-scrubbed derivative is eligible.
• PyTorch .pt files are pickle-based. Load only from a pinned, verified release.

Data and evaluation boundary

Training/evaluation prompts, prepared datasets, hidden-state features, activations, and per-sample outputs are not included. Dataset-license review is not complete enough to redistribute those materials. Public result artifacts are limited to reviewed aggregate evidence.

License and attribution

The released draft weights and MoS code are provided under MIT. See LICENSE and THIRD_PARTY_NOTICES.md. Qwen3 is an Apache-2.0 runtime dependency and its weights are not included.