---
language:
- ru
- en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- gpt2
- causal-lm
- experimental
- trained-from-scratch
- spacemodel
- undertrained
model-index:
- name: Galactic 2.5 Mini
  results: []
---

# Galactic 2.5 Mini
### SpaceModel · ~50M parameters · Experimental RU/EN checkpoint

> **Not a usable chat assistant.** These are genuine weights after **only 300 optimizer steps from random initialization** on a tiny Russian/English corpus. Test generations are largely incoherent, contain repeated newlines and mix languages. Russian and English describe the training data and tokenizer, not demonstrated proficiency. Do not advertise this checkpoint as a fully trained bilingual model.

## Overview

| Field | Value |
|---|---|
| Name | Galactic 2.5 Mini |
| Developer / brand | SpaceModel (user-specified designation) |
| Unique trainable parameters | **49,999,872** |
| Architecture | GPT-2-compatible decoder-only Transformer |
| Layers / hidden size / heads | 12 / 512 / 8 |
| Feed-forward size | 2,048 |
| Vocabulary | 23,258, newly trained byte-level BPE |
| Position embeddings | Learned absolute, 512 positions |
| Training sequence length | 128; longer contexts were not trained |
| Embeddings | Input/output weights tied |
| Initialization | Random; **no pretrained checkpoint used** |
| Training / export precision | FP32 / FP16 safetensors |
| Training task | Causal next-token prediction |
| Chat template / instruction tuning | None |
| Release status | Short-run technical proof of training only |

SpaceModel is the requested developer label for this project; corporate identity and ownership of a Hugging Face namespace were not verified. The code and local CPU training run were produced with AI assistance. “2.5” is a requested product name, not an assertion about prior releases.

## Actual training run

- Hardware available: 2 CPU cores, approximately 2 GiB system RAM; no GPU.
- Seed: 42.
- Optimizer: AdamW, peak LR 5e-4, betas (0.9, 0.95), weight decay 0.1, gradient norm clipping 1.0.
- LR schedule: 20-step warmup multiplied by cosine decay to approximately 10% of peak.
- Steps: **300**. Batch size: **2**. Sequence length: **128**.
- **76,800 input-token presentations**, corresponding to **76,200 shifted prediction targets**. These are not unique-token counts.
- One RU window and one EN window in every batch; random sampling with replacement.
- Training-loop time: **250.69 seconds**, excluding data preparation, initial/final validation, export and verification.
- PyTorch 2.5.1+cpu; Transformers 4.46.3.
- No optimizer state or RNG state is shipped; restarting from the weights uses a fresh optimizer.

### Small held-out loss check (not a benchmark)

The final 5% by characters of each work, split at a newline, was held out before tokenizer training. Eight fixed random batches of two 128-token windows per language were evaluated before and after training. Overlap between validation windows is possible. Metrics below are FP32 **before FP16 export**.

| Language | Initial cross-entropy (nats/token) | Final cross-entropy (nats/token) |
|---|---:|---:|
| Russian | 10.1326 | 7.2389 |
| English | 10.1002 | 6.2290 |

A falling loss shows that optimization ran; it **does not** demonstrate useful generation, reasoning, instruction following, language fluency or safety. No external benchmarks were run. Full logs are in `training_log.jsonl` and the run report is in `training_report.json`.

## Data and provenance

Five Project Gutenberg UTF-8 texts:

| Language | Work | Author / editor | Source |
|---|---|---|---|
| RU | Духовные оды | Gavriil Derzhavin | https://www.gutenberg.org/ebooks/14741 |
| RU | 1001 задача для умственного счета | Sergei Rachinskii | https://www.gutenberg.org/ebooks/16527 |
| RU | Московия в представлении иностранцев XVI-XVII в. | Pavel Apostol; editor Georgii Lukomskii | https://www.gutenberg.org/ebooks/30774 |
| EN | Alice’s Adventures in Wonderland | Lewis Carroll | https://www.gutenberg.org/ebooks/11 |
| EN | Pride and Prejudice | Jane Austen | https://www.gutenberg.org/ebooks/1342 |

Prepared token pools: RU training **82,978**, RU validation **5,546**; EN training **199,642**, EN validation **10,912**. The run sampled only a small fraction of the available windows, with replacement. The raw corpus is not balanced in size; training batches are balanced by language.

Only the exterior Gutenberg header/license footer was stripped. Editorial material and contents may remain. Russian material includes historical orthography and very different domains from modern dialogue. This is not a modern, curated bilingual pretraining corpus. Texts are listed by Gutenberg as public domain in the USA; check applicable local copyright and source terms. Raw texts are not bundled. Raw-file SHA-256 hashes and download URLs are recorded in `data_sources.json`.

## Load with Transformers

Install `torch`, `transformers==4.46.3`, `tokenizers==0.20.3`, and `safetensors`.

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Local folder containing config.json and model.safetensors,
# or replace this with YOUR_NAMESPACE/Galactic-2.5-Mini after upload.
model_id = "./model"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float32, trust_remote_code=False
).eval()

inputs = tokenizer("Однажды вечером", return_tensors="pt")
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=48, do_sample=True,
        temperature=0.8, top_p=0.9, repetition_penalty=1.1,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

This code demonstrates loading and generation, **not output quality**. No custom remote code is required. For shorter, actually seen positional ranges, keep prompt plus generated text within 128 tokens; the configured hard limit is 512.

## Verification

The exported `model.safetensors` was loaded offline through `AutoModelForCausalLM` and tested on one Russian and one English prompt. Parameter count, finite FP16 weights, finite FP32 logits, tokenizer round trips and generation execution passed. The resulting texts were incoherent. Exact unedited samples, file size and SHA-256 are in `verification.json`.

- Weight file: **100,014,488 bytes**.
- Weight SHA-256: `c5203f2624e8f39c502f643698ca8691d6a156a5ca4c7a30ad8e7be738795391`.

## Training code

`training/` contains the data-preparation, training, inference and upload scripts, requirements and a Russian usage guide. The original downloadable project uses `model/` as its export directory; the published Hub repository places the contents of that directory at the root.

From a clone of the Hub repository, to reproduce into a separate output directory:

```bash
pip install -r training/requirements.txt
cd training
python prepare_data.py
python train.py --steps 300 --output reproduced
```

To initialize further optimization from the Hub checkpoint after preparing the **same** tokenizer-compatible data:

```bash
# Inside training/; parent directory is the published model.
python train.py --resume .. --output continued --steps 1000
```

Review the guide before changing data. Never retrain the tokenizer on different texts and pair the new token IDs with these existing weights. For substantial training, expand and audit the corpus, add checkpointing, use appropriate GPU resources, and perform independent evaluations. Repeating this tiny corpus is not a substitute for pretraining.

## Limitations and intended use

For educational inspection, pipeline testing and experimentation with further pretraining only. Not suitable for deployed assistants, factual answers, medical/legal advice, translation, reasoning, arithmetic, moderation or safety-critical decisions. The checkpoint can emit biased, offensive or nonsensical content; it has no safety alignment. Historical source texts may contain obsolete or objectionable views. Training-data memorization has not been evaluated.

## License

The original project code, tokenizer, configuration and model weights are offered under MIT; see `LICENSE`. Source works and third-party software retain their own legal status and terms. No claim of rights over third-party source texts is made.

## Кратко по-русски

**Galactic 2.5 Mini, SpaceModel, 49 999 872 параметра.** Веса действительно обучались с нуля, но всего 300 шагов на CPU. Это сильно недообученный экспериментальный checkpoint, а не рабочая русско-английская чат-модель. Проверенные примеры генерации бессвязны. Для полезного качества нужны существенно больше данных и вычислений. Не скрывайте этот статус при публикации.