Sankarshana-175M-Base-1704m

Sankarshana-175M-Base-1704m is a compact Polish causal language model physically extracted from a jointly trained Vyuhu supernetwork.

The release contains only the fixed Sankarshana compute path. It is not a wrapper around the full Vyuhu model and it does not require the original 282.68M-parameter supernetwork at inference time.

Property	Value
Parameters	177,324,816 (177.32M)
Size class	175M
Vocabulary	32,000
Context length	1024
Hidden size	1152
Query heads / KV heads	18 / 6
Head dimension	64
FFN	3584
Training tokens at extraction	1,704,591,360
Initialization	From scratch
Language	Primarily Polish
License	Apache 2.0

This is a base language model. It has not undergone instruction tuning, SFT, RLHF/RLAIF, preference optimization, or safety alignment.

The model is not intended to behave like a conversational assistant. Generated text may be incorrect, inconsistent, repetitive, biased, offensive, or otherwise undesirable. Outputs should not be treated as factual, safe, or reliable without additional evaluation and task-specific safeguards.

Why Sankarshana

Vyuhu was trained as one dense supernetwork exposing four deterministic compute profiles:

• Vasudeva
• Sankarshana
• Pradyumna
• Aniruddha

The profiles share the same trained supernetwork but execute different fixed amounts of compute.

For this public standalone release, Sankarshana was selected because it occupies a useful gap in the Polish model landscape around the 175M parameter class and showed the strongest quality-throughput trade-off among the evaluated Vyuhu profiles at this checkpoint.

On the official SpeakLeash Open PL polish_mc benchmark, Sankarshana achieved the highest normalized accuracy among the evaluated Vyuhu profiles while using substantially less compute than the full Vasudeva path.

The full dynamic Vyuhu supernetwork is therefore not required to use this release. Sankarshana is distributed as a conventional fixed-path causal LM.

Physical extraction

The source checkpoint contains all four Vyuhu compute profiles. This repository contains only the trained Sankarshana path.

Sankarshana uses the fixed stage schedule:

[1, 3, 1]

The extracted graph retains:

• 4 mandatory GQA attention anchors,
• 5 elastic heavy blocks,
• the Sankarshana-specific H-controller state,
• the Sankarshana-specific L-channel state,
• final RMSNorm,
• tied token embedding / LM head.

Unused heavy blocks and profile-conditioning rows belonging to Vasudeva, Pradyumna, and Aniruddha are not included.

There is:

• no runtime profile selector,
• no token router,
• no MoE routing,
• no dynamic expert selection.

Extraction equivalence

The physical model was verified directly against the original Vyuhu checkpoint using the Sankarshana profile.

Source Vyuhu parameters:       282,681,792
Extracted Sankarshana params:  177,324,816

source -> extracted max |logit delta|: 0.0
extracted -> HF reload max |logit delta|: 0.0

The standalone Hugging Face model therefore reproduces the source Sankarshana function exactly for the tested deterministic equivalence probes.

Architecture

Sankarshana combines mandatory global GQA attention anchors with cheaper elastic causal-convolution blocks.

Property	Value
Hidden size	1152
Query heads	18
KV heads	6
Head dimension	64
Attention	GQA in mandatory anchors
Mandatory GQA anchors	4
Retained elastic heavy blocks	5
Stage schedule	[1, 3, 1]
Elastic mixer width	768
Elastic kernel size	8
Feed-forward	SwiGLU
FFN intermediate size	3584
Normalization	RMSNorm
Norm epsilon	1e-6
Positional encoding	RoPE
Dropout	0
Linear bias	No
Context length	1024

The optional heavy blocks use a gated causal-convolution mixer followed by SwiGLU. Global self-attention is concentrated in the mandatory GQA anchors.

Training snapshot

This release was extracted from:

checkpoint-00013005

at:

1,704,591,360 input training tokens

The model was trained from random initialization as part of the Vyuhu supernetwork.

The full training corpus for the run is larger than the amount consumed by this snapshot, so this repository should be understood as an early training-state release, not the final point of the full Vyuhu run.

Evaluation

SpeakLeash Open PL

Official SpeakLeash lm-evaluation-harness, branch polish4, task group polish_mc, 0-shot.

Metric	Sankarshana
Accuracy	0.43419
Normalized accuracy	0.46037
F1	0.08561

At this checkpoint, Sankarshana achieved the best acc_norm among the evaluated Vyuhu profiles.

For reference, the same evaluation produced:

Model / profile	acc_norm
Sankarshana ~177M	0.46037
Vasudeva 282.68M	0.44941
APT3-275M-Base	0.44204
Pradyumna ~125M	0.43967
Aniruddha ~100M	0.43080

These results are benchmark-specific and should not be interpreted as universal model rankings.

Frozen Polish Constitution likelihood

A separate likelihood evaluation used the same frozen Polish Constitution text for every model.

Metric	Sankarshana
PPL / token	13.703
BPC	0.77241
bits / UTF-8 byte	0.73613

Per-token perplexity is tokenizer-dependent. For comparisons across different tokenizers, BPC and especially bits per UTF-8 byte are more meaningful than raw PPL.

Loading

This repository contains custom Transformers code, so loading requires trust_remote_code=True.

Review the repository code before enabling remote code execution.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "OrisTeam/Sankarshana-175M-Base-1704m"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

Generation

inputs = tokenizer(
    "Polska jest",
    return_tensors="pt",
)

output = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
    use_cache=False,
)

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Example greedy health-check output from the exported checkpoint:

Polska jest w stanie zaoferować nam wiele możliwości.
W jaki sposób możemy

This example is only a functional sanity check, not a quality demonstration.

Intended use

Sankarshana is intended primarily for:

• research on compact Polish causal language models,
• continued pre-training,
• supervised fine-tuning,
• instruction tuning,
• classification or ranking through additional heads,
• experimentation with local inference and deployment,
• benchmarking compact Polish LMs.

Because this is a base model, downstream users should normally fine-tune or otherwise adapt it before conversational or application-facing use.

Limitations

• No SFT or instruction tuning.
• No RLHF, RLAIF, DPO, or comparable preference optimization.
• No dedicated safety alignment stage.
• May generate false or fabricated information.
• May produce incoherent, repetitive, biased, offensive, or otherwise undesirable text.
• Should not be relied upon for medical, legal, financial, safety-critical, or other high-stakes decisions without independent validation.
• Context length is 1024 tokens.
• The current release does not implement an optimized KV-cache path; generation recomputes the prefix.
• The model was trained primarily for Polish and should not be assumed to provide equivalent quality in other languages.
• Benchmark results refer to this specific early checkpoint and evaluation setup.

Access

The repository may require users to acknowledge the model terms or warning before accessing the files.

This is not a manual approval process by the model author. Access does not imply that generated outputs are reviewed, endorsed, safe, or suitable for a particular application.

@misc{Vyuhu280MBase1704m,
  author       = {Aleksander Ogrodzki},
  title        = {Vyuhu-280M-Base-1704m},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/OrisTeam/Vyuhu-280M-Base-1704m},
  note         = {Model architecture, training data pipeline, and training pipeline developed by the author}
}

• Aleksander Ogrodzki^OrisTeam - model owner, architecture and training pipeline development, data preparation, model training, and future development roadmap

License

Sankarshana-175M-Base-1704m is released under the Apache License 2.0.

See the repository LICENSE file for the complete license text.