Mossez-100M-Base

Mossez-100M-Base is a compact decoder-only causal language model developed by Mossez Systems for research, local inference, and experimentation with Russian–English language modeling.

The released weights include the model's initial web pretraining followed by a quality-focused continued-pretraining stage. The continued-pretraining stage is part of the model's training history; the public model name remains Mossez-100M-Base.

This is a base model, not a chat or instruction-following model.

Model details

Property	Value
Parameters	100,092,672
Architecture	Llama-compatible decoder-only Transformer
Layers	12
Hidden size	768
Query attention heads	12
Key/value heads	4
Intermediate size	2,048
Context length	1,024 tokens
Vocabulary	32,000
Tokenizer	Byte-level BPE
Activation	SwiGLU / SiLU
Normalization	RMSNorm
Position encoding	RoPE
Embeddings	Tied input/output embeddings
Primary languages	Russian, English
Weight format	Safetensors, FP32

Intended use

The model is intended for:

• research on compact causal language models;
• local text completion;
• tokenizer, inference, and post-training experiments;
• use as a base checkpoint for continued pretraining or supervised fine-tuning;
• educational and reproducibility work.

The model is not intended to be used as a factual authority, production assistant, safety-critical system, or autonomous decision-maker.

Training

Initial pretraining

The initial model was trained on a Russian–English web corpus:

• approximately 85% Russian FineWeb2-HQ data;
• approximately 15% English FineWeb-Edu data;
• 2,000,027,648 training-token exposure;
• 30,518 optimizer steps;
• 65,536 effective tokens per optimizer step;
• sequence length 1,024.

Quality continued pretraining

The final public weights received an additional 100,007,936-token quality-focused continued-pretraining stage:

Source	Training tokens
Russian Wikipedia	60,004,352
English Wikipedia	25,001,984
Replay from the original corpus	15,001,600
Total	100,007,936

Wikipedia source configurations:

• wikimedia/wikipedia, 20231101.ru;
• wikimedia/wikipedia, 20231101.en;
• pinned dataset revision: b04c8d1ceb2f5cd4588862100d08de323dccfbaa.

The Wikipedia corpus was normalized, filtered, exactly deduplicated, near-deduplicated, and split at source-article level to prevent chunks from the same article crossing train, validation, and test sets.

Continued-pretraining configuration:

• 1,526 optimizer steps;
• effective batch: 65,536 tokens;
• maximum learning rate: 5e-5;
• 50 warmup steps;
• cosine decay;
• AdamW;
• gradient checkpointing;
• final selected checkpoint: step 1,526.

Total training exposure

The combined training exposure was:

2,100,035,584 tokens

This number is token exposure, not a claim of unique corpus size.

Evaluation

Perplexity was measured with sequence length 1,024. Lower is better.

Evaluation set	Before quality CPT	Mossez-100M-Base	Relative PPL improvement
Wikipedia validation	19.5363	13.6782	29.99%
Independent Russian test	18.0996	12.8478	29.02%
Independent English test	22.4920	16.0280	28.74%
Independent combined test	19.3179	13.7286	28.93%
Original web-domain validation	20.0903	20.7899	-3.48%

The quality-focused stage substantially improved performance on held-out encyclopedic text while causing a modest regression on the original web-domain validation set.

The independent combined test contained 2,048 blocks, or 2,097,152 tokens:

• Russian: 1,434 blocks;
• English: 614 blocks.

Usage

Install a recent version of transformers, torch, and safetensors.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mossez-systems/Mossez-100M-Base"

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
)
model.to(device)
model.eval()

prompt = "Искусственный интеллект — область информатики, которая"
inputs = tokenizer(
    prompt,
    return_tensors="pt",
    add_special_tokens=False,
).to(device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.08,
        max_new_tokens=96,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Prompting notes

This is a base completion model. It does not have a chat template and was not trained to follow system, user, and assistant roles. Sentence beginnings and document-style prefixes work better than chat-style prompts.

Greedy decoding is useful for diagnostics but is more likely to collapse into repetition. Sampling does not guarantee factuality.

Limitations

Mossez-100M-Base is a small research model and has significant limitations:

• it can hallucinate names, dates, places, quotations, and causal relationships;
• it may confuse entities or continue a prompt as if it referred to another subject;
• it can repeat words, phrases, headings, or entire semantic patterns;
• it may generate fluent but false encyclopedic-looking text;
• it is not instruction tuned and may ignore requests or formatting constraints;
• its 1,024-token context window is small;
• Russian performance is generally stronger than English performance;
• its knowledge is not current and should not be treated as a reliable snapshot of the world;
• it may reproduce biases, errors, personal information, or undesirable content present in public web data;
• its outputs have not been comprehensively evaluated for safety.

Do not rely on the model for medical, legal, financial, security, or other high-stakes decisions.

Training data and attribution

The model was trained on processed subsets derived from:

• FineWeb2, released under ODC-By 1.0 and subject to Common Crawl terms;
• FineWeb, released under ODC-By 1.0 and subject to Common Crawl terms;
• Wikimedia Wikipedia, whose source text is available under the applicable Wikimedia licensing terms, including CC BY-SA and GFDL.

The original datasets are not redistributed in this model repository. Dataset licenses and source-site terms remain applicable to the underlying data. See NOTICE.md for attribution information.

License

The model weights and original repository materials are released under the Apache License 2.0. Third-party datasets, source text, names, and trademarks remain subject to their own licenses and terms.

See LICENSE and NOTICE.md.

Citation

@software{mossez_100m_base_2026,
  author       = {{Mossez Systems}},
  title        = {Mossez-100M-Base},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/mossez-systems/Mossez-100M-Base}
}

Acknowledgements

This work uses resources made available by Hugging Face, Common Crawl, Wikimedia contributors, the PyTorch project, and the Transformers project. Their inclusion here does not imply endorsement of Mossez Systems or this model.