README.md

---
language:
- de
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- custom-code
- research
- safetensors
datasets:
- HuggingFaceFW/fineweb-2
- HuggingFaceFW/finewiki
- HuggingFaceFW/fineweb
license: mit
---

<h1 align="center"> HanseLM 78M Base</h1>


![HanseLM-78M-Base](https://cdn-uploads.huggingface.co/production/uploads/64bbaff0af05eb17c7b24b6f/KyIAArwsvU9zXBBrvzenB.png)

**HanseLM 78M Base is a small German-first causal language model and an
experimental research preview.** It was trained from scratch to study whether
a hybrid attention/convolution architecture can learn useful German text
representations under a roughly 1.5-billion-token compute budget.

This is not a frontier model, not an instruction-tuned assistant, and not
expected to be broadly useful without further post-training. It may produce
incorrect, repetitive, mixed-language, unsafe, or memorized text. The release
is primarily useful for architecture experiments, education, controlled
fine-tuning, and reproducibility work.

The release code and model weights are available under the MIT License. The
source datasets are not redistributed and retain their own terms; see
[`DATA_SOURCES.md`](DATA_SOURCES.md).

## Model details

| Property | Value |
|---|---|
| Model type | Decoder-only causal language model |
| Parameters | 78,430,848 |
| Vocabulary | 24,576-token byte-level BPE |
| Context trained/tested | 2,048 tokens |
| Hidden size | 640 |
| Layers | 14: 11 attention, 3 causal convolution |
| Attention | Grouped-query attention, 10 query heads, 2 KV heads |
| Head dimension | 64 |
| Feed-forward | SwiGLU, hidden size 1,792 |
| Position encoding | RoPE, theta 10,000 |
| Normalization | Pre-norm RMSNorm, epsilon 1e-6 |
| Precision | BF16 training/autocast; FP32 master weights |
| Embeddings | Input/output weights tied |
| Training stage | Base pretraining only |

### Architecture

Each block applies a pre-normalized token mixer and a pre-normalized SwiGLU
feed-forward network with residual connections. The layer pattern is:

```text
A A C A A A C A A A C A A A
```

`A` blocks use causal scaled-dot-product grouped-query attention. Ten query
heads share two key/value heads in five-head groups. Queries and keys receive
per-head RMS normalization before rotary position encoding. `C` blocks use a
gated, depthwise causal 1D convolution with kernel size 7. Convolution blocks
occur at layers 3, 7, and 11 (one-indexed). All projections are bias-free.

The hybrid design reduces the number of global-attention layers, but this
release does not claim an inference-speed advantage: the reference
implementation has no KV cache and currently recomputes the visible context
during autoregressive generation.

## Tokenizer

The tokenizer is a custom NFC-normalizing, byte-level BPE tokenizer with byte
fallback. It was trained alongside the project and contains:

```text
<|pad|> <|bos|> <|eos|> <|system|> <|user|>
<|assistant|> <|tool|> <|tool_result|>
```

The presence of role tokens does not make the base model chat-capable. No chat
template is defined for this repository.

## Training

The sampled pretraining mixture contained approximately 1.500 billion tokens:

| Source | Language | Approx. tokens | Share |
|---|---:|---:|---:|
| FineWeb2 (`deu_Latn`) | German | 1.200B | 80% |
| FineWiki (`de`) | German | 225M | 15% |
| FineWeb | English | 75M | 5% |

Training used two stages:

1. 1.350B tokens with 1,024-token sequences.
2. 150.012M additional tokens with 2,048-token sequences.

Phase 2 continued from the Phase 1 final checkpoint with AdamW
(`betas=(0.9, 0.95)`), a peak learning rate of `4e-5`, a minimum learning rate
of `4e-6`, and 100 warmup steps. Training ran locally with PyTorch/ROCm on an
AMD Radeon RX 9070 XT.

### Compute and hardware

The full base model was trained locally on a **single consumer GPU**, not a
multi-GPU server or cloud cluster:

| Item | Value |
|---|---|
| GPU | 1× AMD Radeon RX 9070 XT |
| Available VRAM | 15.81 GiB (marketed as 16 GB) |
| Software | PyTorch 2.12.0 + ROCm 7.14 under WSL2 |
| Precision | BF16 autocast, FP32 AdamW state |
| Peak allocated VRAM | 12.15 GiB in Phase 1; 11.83 GiB in Phase 2 |
| Typical observed throughput | approximately 43k–48k tokens/s |
| Phase 1 GPU-process time | approximately 8.38 GPU-hours |
| Phase 2 GPU-process time | 1.03 GPU-hours |
| Total pretraining compute | approximately **9.41 single-GPU hours** |

Of the total, 8.01 hours come directly from structured trainer timers and
approximately 1.4 hours are estimated for the initial unstructured segment.
The Phase 1 number includes replayed work after interrupted runs. Timers
include validation and checkpoint overhead while the trainer was active, but
exclude intentional pauses, dataset preparation, checkpoint selection, and
the later release evaluations.

This modest hardware footprint is part of the experiment: HanseLM tests what
can be trained end-to-end on one 16 GB desktop GPU. It should not be confused
with frontier-scale pretraining, and the low compute budget is also a major
reason for the model's limited factual and reasoning ability.

Documents were packed with EOS separators. The current attention mask does not
isolate documents inside a packed sequence, so tokens can attend across an EOS
boundary. This is a known training limitation.

## Evaluation

The selected final checkpoint beat the late Phase 2 checkpoints on three
independent mixed-validation samples. The margin over step 12,000 was small
but consistent.

### Held-out language-model loss

All rows below use 2,048-token sequences, batch size 6, 200 batches, and seed
1,234.

| Validation corpus | Loss | Perplexity |
|---|---:|---:|
| FineWiki German | 2.574947 | 13.131 |
| FineWeb English | 2.943911 | 18.990 |
| Mixed validation | 3.175663 | 23.943 |
| FineWeb2 German | 3.310921 | 27.410 |

Perplexity is specific to this tokenizer and each corpus. Values must not be
compared directly with models using another tokenizer, and these results are
not downstream knowledge, reasoning, safety, or instruction-following
benchmarks. Standardized German and English benchmarks have not yet been run.

Phase 2 improved the shared mixed-validation loss from 3.228526 (Phase 1) to
3.175663.

A fixed-seed generation check found no repeated token trigrams in the two
German and one English sample; a deliberately mixed prompt had a 0.0217
repeated-trigram fraction and continued in English. These four samples are too
small for a quality claim. They also exposed a clear factual error about
Hamburg, reinforcing that the model must not be used as a factual source. The
full samples and a successful 2,048-token finite-logit smoke test are recorded
under [`evaluation/`](evaluation/).

A broader 20-sample suite (10 prompts, two seeds) produced a mean repeated
token-trigram fraction of 0.0445 and a maximum of 0.1613. German and English
continuations were often locally grammatical, and the mixed prompt continued
in German. However, factual prompts failed consistently: the model called
Berlin the capital of southern France or Croatia, placed German federal
history in 1921/1998, and gave contradictory freezing temperatures. Recipe
and story continuations were recognizable in form but frequently incoherent.
The honest conclusion is that the checkpoint has learned language form and
domain-like continuation, but not dependable knowledge or instruction
following. Raw outputs are stored in
[`evaluation/generation-suite.json`](evaluation/generation-suite.json).

## Usage

This repository contains custom Transformers code, so loading requires
explicitly trusting the repository code:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Evicka/HanseLM-78M-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Die Hanse war", return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_k=50,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The reference implementation does not implement KV caching. Generation is
therefore slower than similarly sized cached Transformer models.

## Intended uses

- Research on small language models and hybrid token mixers.
- Reproducible continued pretraining or post-training experiments.
- Educational inspection of a complete from-scratch model pipeline.
- Low-stakes, human-reviewed German or English text continuation experiments.

It is not intended as a factual source, production assistant, autonomous
agent, safety-critical component, or replacement for a larger evaluated
model. High-impact decisions and unsupervised public text generation are out
of scope.

## Limitations and risks

- Only 78M parameters and about 1.5B training tokens.
- Mostly German web text; only 5% of the sampled mixture was English.
- No instruction tuning, preference optimization, or safety post-training.
- No systematic benchmark, bias, toxicity, privacy, or memorization audit.
- No guaranteed factuality, reasoning, arithmetic, code, or tool-use ability.
- Possible repetition, abrupt continuations, malformed text, and language
  switching.
- A maximum trained context of 2,048 tokens; longer inputs are rejected.
- Packed-document cross-attention as described above.
- Web and Wikipedia data can contain errors, stereotypes, personal data,
  copyrighted material, and other undesirable content.

## Reproducibility

The staged evaluation record is in
[`evaluation/internal-eval.json`](evaluation/internal-eval.json). The selected
weight source is the Phase 2 final checkpoint after 150,011,904 Phase 2 tokens.
The model contains 78,430,848 parameters.

The custom adapter was verified with Transformers 4.57.6 and PyTorch
2.12.0+rocm7.14.0. Loading through `AutoModelForCausalLM` produced no missing
or unexpected keys, and its logits matched the native implementation exactly
for the equivalence input (`max_abs_diff = 0`). Artifact checksums are recorded
in [`MANIFEST.sha256`](MANIFEST.sha256).

## License and attribution

The HanseLM release code and model weights are licensed under the
[MIT License](LICENSE). Training-source attribution and third-party terms are
documented separately in [`DATA_SOURCES.md`](DATA_SOURCES.md). In particular,
FineWeb and FineWeb2 identify ODC-By 1.0, while FineWiki identifies CC BY-SA
4.0 and GFDL for its Wikipedia-derived text. The MIT license does not
relicense those source datasets.

## Citation

No paper or archival report exists yet. For reproducible references, cite the
repository and an immutable release revision:

```bibtex
@software{hanse_lm_78m_base_2026,
  author = {EvickaStudio},
  title = {HanseLM 78M Base},
  year = {2026},
  url = {https://huggingface.co/Evicka/HanseLM-78M-Base},
  version = {v0.1}
}
```