mungus-v1

A Russian-language text generation model with 27.8B parameters. It produces short, stylistically consistent answers in Russian and shows its reasoning trace before every reply.

This language model has its own style, which consists of 30% femboyism and 70% aggression.

Specification

Total parameters	27,781,427,952
Vocabulary size	248,320
Embedding dimensions	5,120
Hidden layers	64
Attention heads	24
Head dimension	256
Context length	262,144

Architecture

Qwen3_5ForConditionalGeneration — a hybrid vision-language backbone, used here for text generation. The checkpoint keeps the vision tower intact.

Component	Details
Text parameters	24,777,901,056
Attention	24 query heads × 256, 4 KV heads (GQA), output gate, 16 full-attention layers (every 4th layer)
Linear attention	48 Gated DeltaNet layers: 16 key / 48 value heads, 128 head dim, conv kernel 4
FFN	SwiGLU, intermediate size 17,408
Position encoding	partial RoPE (factor 0.25, theta 1e7), M-RoPE interleaved
Embeddings / lm_head	untied, 248,320 × 5,120 each (1,271,398,400 params each)
Vision tower	460,730,096 params (27 layers, hidden 1152, patch 16, spatial merge 2)
Precision	bfloat16
Checkpoint size	55.6 GB, 18 safetensors shards

Usage

bf16 (needs ~56 GB of VRAM, e.g. A100/H100 80 GB):

import torch
from transformers import AutoProcessor, AutoModelForCausalLM

repo = "RaspizdAI/mungus-v1"
proc = AutoProcessor.from_pretrained(repo)
tok = proc.tokenizer
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval()

messages = [{"role": "user", "content": "Объясни простыми словами, что такое Kubernetes."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=700, do_sample=True,
        temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.05,
        pad_token_id=tok.pad_token_id or tok.eos_token_id,
    )
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

4-bit, on a single consumer GPU (self-contained):

import torch
from transformers import AutoProcessor, AutoModelForCausalLM, BitsAndBytesConfig

repo = "RaspizdAI/mungus-v1"
proc = AutoProcessor.from_pretrained(repo)
tok = proc.tokenizer

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo, quantization_config=bnb, dtype=torch.bfloat16, device_map="auto"
).eval()

messages = [{"role": "user", "content": "Объясни простыми словами, что такое Kubernetes."}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs, max_new_tokens=700, do_sample=True,
        temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.05,
        pad_token_id=tok.pad_token_id or tok.eos_token_id,
    )
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The visible answer comes after the reasoning block, so max_new_tokens should be at least 512. Greedy decoding tends to produce long, repetitive traces; sampling in the 0.6–0.8 range works better.

Quad-Bench Results

Dataset

A custom Russian instruction/answer dataset. Answers are short and follow a consistent style; each sample carries an English reasoning trace. The model was trained on it with the reasoning block separated from the final answer.

License & Attribution

This model is licensed under CC-BY-Sa-4.0.

• You are free to share and adapt the material.
• You must give appropriate credit, provide a link to the license, and indicate if changes were made.
• You cannot remove the RaspizdAI copyright and attribution, nor can you close the source of derivative models.