Vyuhu-280M-Base-3012m

This repository is an archived experimental checkpoint and architecture reference, not the final Vyuhu model.

The published weights correspond to checkpoint-00022982 at 3,012,296,704 input training tokens (~3.012B). They replace the earlier OrisTeam/Vyuhu-280M-Base-1704m checkpoint while preserving the same Vyuhu-280M architecture and Hugging Face interface.

This training run is no longer being developed as the final Vyuhu base model. Development continues toward Vyuhu 1.0.

Vyuhu-280M-Base-3012m is a primarily Polish causal base language model trained from scratch as a single dense supernetwork with four deterministic compute paths.

This is a base language model, not a conversational assistant.

It has not undergone instruction tuning, SFT, RLHF/RLAIF, DPO or comparable preference optimization, or a dedicated safety-alignment stage.

Generated text may be incorrect, inconsistent, repetitive, biased, offensive, or otherwise undesirable.

Why this experiment exists

Vyuhu started as a practical experiment in getting several fixed compute paths from a single training run.

Dynamic routing was not attractive for the small-model setting I was working with, so I tested a simpler alternative: deterministic paths through one shared network. The selected path is fixed before the forward pass and can reduce active compute without loading a different checkpoint.

The goal was not to show that Vyuhu is better than a conventional dense model at the same parameter or compute budget. The practical goal was to test a new architecture while obtaining several usable model paths of different effective sizes from one run for later experiments.

This checkpoint should therefore be read as an architecture and training validation artifact, not as a controlled claim of superiority over independently trained dense models.

The run was trained on a single NVIDIA RTX 5060 Ti 16 GB.

Compute profiles

Profile	Schedule	Active compute blocks	Role
Vasudeva	[4, 4, 4]	16 / 16	full path
Sankarshana	[1, 3, 1]	9 / 16	medium path
Pradyumna	[0, 2, 0]	6 / 16	compact path
Aniruddha	[0, 0, 0]	4 / 16	anchor-only path

All profiles share the same trained supernetwork. There is no token-level routing, MoE expert selection, or learned per-token compute controller.

The profile is selected externally and remains fixed for the request.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "OrisTeam/Vyuhu-280M-Base-3012m"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    device_map="auto",
)

Default profile:

sankarshana

Select a profile per request:

outputs = model(
    input_ids=input_ids,
    profile="pradyumna",
)

or during generation:

generated = model.generate(
    **inputs,
    profile="sankarshana",
    max_new_tokens=64,
    do_sample=False,
    use_cache=False,
)

Accepted names and aliases:

vasudeva      v
sankarshana   s
pradyumna     p
aniruddha     a

For concurrent APIs and servers, prefer request-local profile= instead of mutating model-wide state with set_profile().

Model at a glance

Property	Value
Parameters	282,681,792 (282.68M)
Vocabulary	32,000
Context length	1024
Hidden size	1152
Query heads / KV heads	18 / 6
Head dimension	64
FFN	3584
Mandatory GQA anchors	4
Optional elastic heavy blocks	12
Elastic mixer width	768
Elastic kernel	8
Activation	SwiGLU
Normalization	RMSNorm, eps 1e-6
Positional encoding	RoPE
Published checkpoint	checkpoint-00022982
Training input tokens	3,012,296,704 (~3.012B)
Initialization	From scratch
Language	Primarily Polish
License	Apache 2.0

Architecture

High-level flow:

input GQA anchor
    ↓
stage 0: optional heavy-prefix → L if compute skipped → H → GQA anchor
    ↓
stage 1: optional heavy-prefix → L if compute skipped → H → GQA anchor
    ↓
stage 2: optional heavy-prefix → L if compute skipped → H → GQA anchor
    ↓
final RMSNorm → tied LM head

Every profile shares the mandatory GQA anchors. Optional heavy compute is selected as a nested prefix inside each stage.

Elastic heavy blocks use:

RMSNorm
→ gated causal-convolution mixer
→ residual
→ RMSNorm
→ SwiGLU
→ residual

The L channel is a cheap low-rank transfer used when a heavy suffix is skipped.

The H controller is an always-on low-rank representation checkpoint conditioned on the selected global profile.

There is no learned profile selector.

What worked

The architecture behaved as intended during training: all four fixed paths remained usable and their behavior generally followed the expected compute hierarchy.

In the evaluations run during development, smaller profiles did not simply collapse as compute was removed. The second-largest path was often close to the full path, while lower-compute paths remained usable but more prone to loss of topic continuity or simplification.

Qualitative generation tests also showed that the profiles were not behaviorally identical. Despite sharing the same underlying network, they differed in wording, continuation strategy, and how well they maintained local coherence.

These observations support the claim that the shared-path training setup is functional. They do not establish that it is better than training independent dense models.

Export equivalence

The published Hugging Face checkpoint was verified against the original Vyuhu training implementation for all four profiles.

Before serialization:

source Vyuhu -> HF Vyuhu
all tested profiles max |logit delta| = 0.0

Tested sequence lengths:

8
17
64
127

After a complete Hugging Face round trip:

source Vyuhu
    ↓
save_pretrained()
    ↓
model.safetensors
    ↓
AutoModelForCausalLM.from_pretrained()
    ↓
all tested profiles max |logit delta| = 0.0

Named parameters and persistent buffers were also checked after reload and remained bit-identical in the tested release path.

Checkpoint history

Checkpoint	Approx. training tokens	Status
checkpoint-00013005 / 1704m	1,704,591,360	original public reference
checkpoint-00016000 / 2097m	2,097,152,000	intermediate evaluation
checkpoint-00022000 / 2884m	2,883,584,000	later training-progress checkpoint
checkpoint-00022982 / 3012m	3,012,296,704	currently published weights

The original OrisTeam/Vyuhu-280M-Base-1704m repository referred to the earlier public checkpoint and was later updated and renamed for the 3012m release.

Historical evaluation references

The numbers below are preserved as checkpoint references. They are not fresh measurements of the published 3012m checkpoint unless explicitly stated.

SpeakLeash Open PL — polish_mc, 0-shot

Profile	Snapshot	acc ↑	acc_norm ↑	F1 ↑
Vasudeva	2884m progress	0.45955	0.45840	0.04793
Sankarshana	2884m progress	0.45138	0.44973	0.05898
Sankarshana	1704m reference	0.43419	0.46037	0.08561
Vasudeva	1704m reference	0.41980	0.44941	0.16019
Pradyumna	1704m reference	0.43072	0.43967	0.04794
Aniruddha	1704m reference	0.42674	0.43080	0.04990

These values should not be interpreted as a universal model ranking. No 3012m benchmark result is claimed here without evaluation under the same frozen setup.

Frozen Polish Constitution likelihood

Model / profile	Snapshot	PPL / token ↓	BPC ↓	bits / UTF-8 byte ↓
Azurro/APT3-500M-Base	reference	5.254	0.45228	0.43104
Azurro/APT3-275M-Base	reference	6.282	0.50101	0.47748
Vyuhu / Vasudeva	2884m progress	11.782	0.72785	0.69367
sdadas/polish-gpt2-medium	reference	12.298	0.69308	0.66053
Vyuhu / Sankarshana	2884m progress	12.504	0.74538	0.71037
Vyuhu / Vasudeva	1704m reference	13.060	0.75823	0.72261
Vyuhu / Sankarshana	1704m reference	13.703	0.77241	0.73613
Vyuhu / Pradyumna	1704m reference	15.180	0.80261	0.76491
Vyuhu / Aniruddha	1704m reference	18.669	0.86367	0.82310
sdadas/polish-gpt2-small	reference	24.960	0.88858	0.84685

Raw PPL is tokenizer-dependent; bits / UTF-8 byte is generally more useful for cross-tokenizer comparison.

Standalone Sankarshana

A physically extracted earlier Sankarshana reference model is available as:

OrisTeam/Sankarshana-175M-Base-1704m

Parameter counts:

Vyuhu supernetwork:       282,681,792
Standalone Sankarshana:   177,324,816

The extraction was verified against the corresponding Vyuhu path:

source Vyuhu/Sankarshana -> physical Sankarshana
max |logit delta| = 0.0

physical Sankarshana -> Hugging Face reload
max |logit delta| = 0.0

This is useful as a reference for a possible workflow in which a jointly trained path is later extracted and developed independently.

Intended use

This checkpoint is mainly useful for:

• experimenting with deterministic multi-budget compute in one shared model,
• inspecting how fixed paths behave as active compute is reduced,
• extracting or continuing smaller paths in later experiments,
• compute-aware inference experiments,
• supervised fine-tuning or continued pre-training experiments,
• benchmarking compact Polish causal language models,
• reproducing the original Vyuhu architecture and export path.

Limitations

• Archived experimental checkpoint, not the final Vyuhu release.
• No controlled claim that the architecture outperforms independently trained dense models.
• Base model only; no instruction tuning or preference optimization.
• No dedicated safety-alignment stage.
• May generate false, incoherent, repetitive, biased, offensive, or otherwise undesirable content.
• Should not be relied upon for medical, legal, financial, safety-critical, or other high-stakes decisions without independent validation.
• Context length is 1024 tokens.
• Current release has no optimized KV-cache generation path.
• Primarily trained for Polish.
• Historical benchmark values refer to explicitly labeled earlier checkpoints unless otherwise stated.
• set_profile() mutates model-level state; use request-local profile= in concurrent servers.

Development status

3012m is retained as a comparison artifact and record of the original Vyuhu architecture line.

Maintenance may include documentation fixes, reproducibility checks, export fixes, and tests around the existing architecture. The checkpoint itself is not planned as the main target for further model development.

Work toward Vyuhu 1.0 is changing parts of the design, including the treatment of maximum-compute paths and anchors, larger model scale, and an optimized KV-cache path. Other experiments also include MTP and Engram-style components for smaller networks.

Citation

@misc{Vyuhu280MBase3012m,
  author       = {Aleksander Ogrodzki},
  title        = {Vyuhu-280M-Base-3012m},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/OrisTeam/Vyuhu-280M-Base-3012m},
  note         = {Original Vyuhu architecture-line reference checkpoint}
}

Model owner: Aleksander Ogrodzki / OrisTeam — architecture, training pipeline, data preparation, model training, release maintenance, and future Vyuhu development.

License

Released under the Apache License 2.0. See LICENSE for the full license text.