Melih1234/ANKA-50M-RMW3

ANKA-50M RMW3

A 48.94M-parameter English causal language model trained from random initialization on 5 billion tokens using a single NVIDIA A100 80 GB GPU.

ANKA-50M RMW3 explores how much language modeling capability can be obtained under a strict 50-million-parameter budget. It combines a parameter-shared recurrent Transformer backbone with two learned components—Reflect and Morph—and was trained with a mixture that includes WorldPairs relational examples.

Model type: pretrained causal base model. This checkpoint is not instruction-tuned, chat-aligned, or safety-aligned. Its native behavior is text continuation.

ANKA-50M RMW3 GITHUB

link: https://github.com/mamalito345/ANKA-50M-RMW3

Highlights

• 48,944,657 unique trainable parameters, including the tied embedding/output matrix.
• 5,000,000,000 training tokens from random initialization; no pretrained initialization, model merging, or distillation.
• One NVIDIA A100-SXM4 80 GB, not a multi-GPU run.
• 117,899 tokens/second average measured training throughput.
• 11 h 46 m 49 s total training wall time, equal to approximately 11.78 GPU-hours.
• Approximately 424.4 million training tokens per GPU-hour.
• Exact PyTorch architecture, tokenizer, SafeTensors checkpoint, evaluation reports, and architecture ablations are included.

Training efficiency

Training efficiency was a first-class goal of this release. The complete 5B-token base run used BF16 precision and an in-memory data pipeline on a single accelerator.

Measurement	Value
Training tokens	5,000,000,000
Completed optimizer steps	76,294
Batch size	64
Sequence length	1,024
Tokens per step	65,536
Hardware	1 × NVIDIA A100-SXM4 80 GB
Precision	BF16
Average throughput	117,899 tokens/s
Training wall time	42,409 s
Human-readable wall time	11 h 46 m 49 s
Compute used	11.78 GPU-hours
Tokens per GPU-hour	424.4M
Best internal validation loss	2.334379

The throughput figure is the average recorded by the training loop for the full run. Corpus construction and tokenizer training are separate preprocessing stages and are not included in the 42,409-second model-training time.

Architecture

Component	Configuration
Parameters	48,944,657
Vocabulary	16,384 tokens
Hidden width	512
Physical Transformer blocks	13
Recurrent passes	2
Effective block applications	26
Attention	Grouped-query attention, 8 Q / 2 KV heads
Head dimension	64
Feed-forward network	SwiGLU, width 1,280
Position encoding	RoPE
Normalization	Pre-RMSNorm
Reflect rank	32
Morph stages / experts	4 / 4
Morph expert rank	368
Maximum context	1,024 tokens
Input/output weight tying	Enabled
Linear biases	Disabled

The 13 physical Transformer blocks are applied twice with shared parameters. This produces 26 effective block applications without duplicating the backbone weights.

Reflect

During the second recurrent pass, each Reflect unit compares the current hidden state with the corresponding state saved during the first pass. A gated rank-32 transformation turns that discrepancy into a learned correction.

Morph

Four sparse Morph stages are distributed across the effective depth. At each stage, a learned router assigns each token to one of four rank-368 experts. Only the selected expert is evaluated for that token, and its output is added through a learned scale.

WorldPairs

WorldPairs is a data-side component containing paired descriptions of controlled world-state transformations. It was included to expose the model to explicit relational changes rather than relying only on unpaired next-token examples.

Architecture ablations

The following results come from the included 4,096-token architecture audit. They are diagnostic ablations, not replacements for the official benchmarks.

Audit condition	Loss	Change from full model
Full RMW3	2.629027	—
Reflect disabled	2.896107	+0.267081
Morph disabled	2.796865	+0.167838
Morph router assignments shifted	2.655662	+0.026635

Additional observations:

• Reflect improved token-level NLL for 70.70% of the audited tokens.
• All 13/13 Reflect units had a positive individual ablation contribution.
• Morph improved token-level NLL for 65.33% of the audited tokens.
• All 4/4 Morph stages had a positive individual ablation contribution.
• The trained operating scale of 1.0 was the best point in both Reflect and Morph scale sweeps.

The complete machine-readable reports are available under results/architecture_audits/.

Zero-shot evaluation

The four multiple-choice tasks were evaluated zero-shot with lm-evaluation-harness==0.4.12, seed 3407, and no few-shot examples. WikiText-103 was evaluated on the raw test split with a 1,024-token context and a 512-token stride.

Task	Raw accuracy	Normalized accuracy	Standard error
HellaSwag	27.28%	28.46%	±0.45% normalized
ARC-Easy	45.79%	42.26%	±1.02% raw
PIQA	58.00%	56.80%	±1.15% raw
WinoGrande	51.46%	—	±1.40% raw

Language-model evaluation

Dataset	Loss	Perplexity	Scored tokens
WikiText-103 test	3.256828	25.967	322,587
Internal held-out validation	2.340759	10.389	131,072 audit tokens

Raw reports, generation samples, exact standard errors, environment metadata, and timing information are stored in results/.

Quick start

This release uses an exact custom PyTorch implementation. It is not presented as a native Transformers AutoModelForCausalLM checkpoint.

# Run these commands inside the downloaded model repository.
pip install -r requirements.txt
python verify_release.py

Generate a continuation:

python inference.py \
  --prompt "The future of artificial intelligence" \
  --max-new-tokens 128 \
  --temperature 0.8 \
  --top-p 0.95

Python usage:

from inference import generate, load_model

model, tokenizer = load_model(".")

text = generate(
    model=model,
    tokenizer=tokenizer,
    prompt="Science is important because",
    max_new_tokens=128,
    temperature=0.8,
    top_p=0.95,
    seed=3407,
)

print(text)

The implementation automatically uses CUDA BF16 when CUDA is available and FP32 on CPU. Autoregressive generation does not yet implement a KV cache, so GPU generation is recommended.

Release contents

File or directory	Purpose
model.safetensors	BF16 base-model weights
tokenizer.json	16,384-token English tokenizer
config.json	Exact RMW3 architecture configuration
generation_config.json	Default sampling settings
modeling_anka.py	Exact PyTorch model implementation
inference.py	Loading and generation utility
verify_release.py	Strict structure, parameter, tying, and integrity checks
training_summary.json	Training time, throughput, hardware, and validation summary
results/	Official evaluations, samples, and architecture audits
CRITICAL_CHECKSUMS.sha256	Checkpoints for the weights and tokenizer

Training data

The tokenizer and model were trained from scratch on an English mixture assembled from openly available sources. The mixture includes educational web text, mathematics and science material, academic and biomedical text, reference works, books, educational question-answer content, and WorldPairs.

Examples of upstream collections used during corpus construction include FineWeb-Edu, FineMath, peS2o, PubMed, arXiv abstracts, Cosmopedia, LibreTexts, Wikimedia, Project Gutenberg, and StackExchange. The full training corpus is not redistributed in this repository, and every upstream component remains subject to its original license and terms.

Intended use

Suitable uses include:

• research on language models under strict parameter and compute budgets;
• studying recurrent depth, low-rank correction, and sparse token routing;
• reproducible base-model evaluation;
• continued research training or downstream experimentation.

Limitations and safety

• This is a base model, not a conversational assistant.
• It may hallucinate facts, repeat text, or produce incoherent continuations.
• It can reproduce biases or unsafe patterns present in its training data.
• It has not been trained for safety refusal, instruction following, tool use, or retrieval.
• Its 1,024-token context and 48.94M-parameter capacity limit long-range reasoning and factual coverage.
• It must not be used as the sole basis for medical, legal, financial, or safety-critical decisions.

Reproducibility and integrity

The checkpoint is stored in SafeTensors format. For compatibility, both token_embeddings.weight and lm_head.weight are serialized; the runtime model ties them to the same parameter, so the unique trainable count remains 48,944,657.

Run:

python verify_release.py
sha256sum -c CRITICAL_CHECKSUMS.sha256

Expected critical hashes:

model.safetensors  91db1ccda91615523e21e57eb0a0e2d5cfba8a4f0ee5dae3eaef96f387a968d5
tokenizer.json     9502f89628bbe79d6adf98c0e199bf3ac4c74864d8175674da45ef039e38c44e

License

The model code and released weights are provided under the Apache License 2.0. Dataset components retain their upstream terms.