Anadolu-40M

Anadolu is a 40M-parameter Turkish causal language model trained from scratch. It needs no GPU: it was pre-trained, fine-tuned and is served on a single Intel i7-7700 CPU. It ships with a NumPy-only inference engine, so the model can run on a machine with no PyTorch and no CUDA.

  • 40,178,176 parameters · 512-token context · 16,389-token byte-level BPE (no UNK)
  • Standard decoder-only Transformer, GPT-2 compatible weights → AutoModelForCausalLM
  • Chat format: <|bos|><|user|>…<|end|><|assistant|>…<|end|>
  • Hosted demo & free OpenAI-compatible API: https://anadoluai.duckdns.org:8443
  • Discord bot with per-user memory: Anadolu AI#4542

This is a small model. It is not a GPT-4 substitute — see Limitations.


Model details

Architecture Decoder-only Transformer (GPT-2 compatible)
Parameters 40,178,176
Layers / heads / d_model 10 / 8 / 512
MLP inner size 2048 (GELU, gelu_new / tanh approximation)
Context length 512 tokens
Vocabulary 16,389 (byte-level BPE, GPT-2 style — every byte is representable, no UNK)
Positional encoding Learned
Normalisation Pre-LayerNorm, eps = 1e-5
Embeddings Tied input/output
Precision float32
Runtime NumPy only (numpy-fp32 export)

Special tokens

Token ID
`< pad
`< user
`< assistant
`< end
`< bos

Training

Stage Detail
Data Turkish Wikipedia (wikimedia/wikipedia, 20231101.tr, CC BY-SA) + Bilkent Turkish Writings Dataset v2
Corpus size ~870.8M characters → ~227.6M BPE tokens available
Tokens seen 344,064,000 (as recorded in the export metadata)
Stage 1 From-scratch pre-training on CPU, then 512-token context adaptation
Stage 2 Supervised fine-tuning (SFT) for chat, ~500 steps (best-validation checkpoint of a 600-step run)
SFT data 1,399 hand-written Turkish chat examples + 208 pairs distilled from larger models
Source checkpoint checkpoints/anadolu40x512b/sft512c.pt
Export numpy-fp32, 2026-11-18

Evaluation

Metric Value
Language-modelling validation loss 3.8330
Perplexity 46.199
Pre-training validation loss 2.9253
SFT validation loss 0.4566
LM eval setup 128 sequences · 65,408 tokens · block 512 · seed 1234

Perplexity is measured on held-out Turkish text from the same distribution as training. It is not a comparison against other Turkish models — no apples-to-apples benchmark has been run yet.

Chat quality and raw language-modelling perplexity can move in opposite directions. This export was re-trained on a larger, cleaner SFT set, which raised the LM perplexity (26.1 → 46.2) while clearly improving the answers users see: on the 8-prompt probe set the defect rate fell from 0.38 to 0.00 per reply, all 8 answers were correct, and there were no repetition loops. Knowledge questions are answered from the service's knowledge base in production, so the perplexity change is not visible to API users.


Usage — transformers

pip install torch transformers
python example_transformers.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "USERNAME/anadolu-40m"          # ← replace with this repo's id
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

messages = [{"role": "user", "content": "Merhaba, nasılsın?"}]
ids = tok.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_tensors="pt"
)

with torch.no_grad():
    out = model.generate(
        ids,
        max_new_tokens=96,
        do_sample=False,          # önerilen: açgözlü üretim (bkz. aşağıda)
    )

print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The model is a plain GPT-2, so transformers pipeline("text-generation"), GPT2LMHeadModel, LoRA fine-tuning and GGUF/llama.cpp conversion are all possible without custom code.

Recommended generation settings

Use greedy decoding (do_sample=False) and leave no_repeat_ngram_size at 0. Both matter a lot on a 40M model:

  • Prefer greedy decoding. Sampled decoding (temperature=0.7, top_p=0.92) is usable with no_repeat_ngram_size=0, but greedy output is the most reliable on a 40M model.
  • no_repeat_ngram_size=3 hurts. Blocking repeated trigrams is a common anti-loop trick, but here it pushes the model onto rare sub-word pieces and it starts producing nonsense words. Measured on the 8-prompt probe set with greedy decoding and repetition_penalty=1.08 (values below are from the previous export; the principle is unchanged):
Prompt no_repeat_ngram_size=3 no_repeat_ngram_size=0
Yapay zekâ nedir? Yapay sinirlerin bütünüdür, emülsiyon ve emülsifiyim, sinirlerinizi ayır… Yapay zekâ, bilgisayarların öğrenme, akıl yürütme ve karar verme gibi insan zekâsına benze…
Kitap okumak neden faydalıdır? Kitap okumadan önce, bir sayfa okuyarak veyaetrafortraya göre değişir. Kitap okumak, bir problemi çözmek için bir yöntemdir.
Kahve içmek zararlı mı? Kahvenin, kahve içer, su ve su içer, içmek su içer ve su içerir. Kahve içmek zararlı mı?
Merhaba, nasılsın? Merhacı, bir gerdanyana eylemi'dir. Merhaba! İyiyim, teşekkürler. Sen nasılsın?

generation_config.json therefore ships with do_sample: false and no_repeat_ngram_size: 0, and repetition_penalty: 1.08.

Note that the public API at anadoluai.duckdns.org:8443 uses sampled decoding (temperature=0.7) with no_repeat_ngram_size=0, and wraps the model in a router with a knowledge-base layer and a garbled-output fallback, so its answers are not directly comparable to a bare model.generate() call.


Usage — NumPy only (no PyTorch)

The repo also contains the original NumPy inference engine under anadolu/. This is the code that actually serves the public API; it needs only numpy, safetensors and threadpoolctl — no PyTorch, no CUDA.

pip install numpy safetensors threadpoolctl
python example_numpy.py
from anadolu.chatml import build_prompt_ids
from anadolu.runtime import Engine, GenerationConfig

# `example_numpy.py` converts model.safetensors into the layout the engine
# expects and writes it to a temp dir; `work` is that directory.
engine = Engine(work, threads=4)

ids = build_prompt_ids(
    engine.weights.tokenizer,
    [{"role": "user", "content": "Merhaba, nasılsın?"}],
    max_tokens=engine.weights.block_size,
)
print(engine.generate(ids, GenerationConfig(max_new_tokens=96))["text"])

Files

File Description
model.safetensors Weights in GPT-2 layout (fp32)
config.json GPT-2 config (n_embd=512, n_positions=512, n_layer=10, n_head=8)
generation_config.json Default sampling parameters
bpe.json Original byte-level BPE vocabulary (used by the NumPy engine)
tokenizer.json, tokenizer_config.json, special_tokens_map.json HF tokenizer + chat template
anadolu/ NumPy-only inference engine (tokenizer, runtime, chat template)
example_transformers.py Run with transformers
example_numpy.py Run with NumPy only
meta.json Export metadata (steps, losses, evaluation)

Hosted API (free)

An OpenAI-compatible endpoint is available; anonymous use needs no key (10 requests/min, 100,000 tokens/day):

curl -sk https://anadoluai.duckdns.org:8443/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "anadolu-40m",
        "messages": [{"role": "user", "content": "Türkiye'"'"'nin başkenti neresi?"}]
      }'

The same endpoint is served by the NumPy engine in this repository, so a self-hosted instance behaves identically.


Limitations

  • Small model. 40M parameters cannot follow complex instructions, do long reasoning, or match large models on knowledge questions. Short Turkish chat, simple Q&A, rewriting and short summaries are the realistic use cases.
  • Repetition. Like other small models it can loop. A repetition penalty (repetition_penalty=1.08) is enabled by default for this reason; no_repeat_ngram_size is left at 0 because the measured effect of setting it to 3 was a severe quality loss (see Recommended generation settings).
  • Knowledge cutoff / facts. Trained on a Wikipedia dump and a student-writing corpus, so it is unreliable for current events and for factual recall outside its training distribution.
  • No safety tuning. The model was not RLHF'd or red-teamed; outputs should be reviewed before any user-facing deployment.
  • Turkish only. It is trained on Turkish data and is not a multilingual model.
  • Context is 512 tokens including the prompt.

Lisans / License

Code and weights are published for research and evaluation use.

⚠️ The training data is not fully permissive. It includes Turkish Wikipedia (CC BY-SA) and the Bilkent Turkish Writings Dataset v2, which is distributed under an Academic Use License (non-commercial, attribution required). Please review those terms before any commercial use of the weights. A final license for this repository has not been fixed yet.


Türkçe özet

Anadolu-40M, sıfırdan eğitilmiş 40 milyon parametreli bir Türkçe dil modelidir. GPU gerektirmez; eğitimi de sunumu da tek bir Intel i7-7700 işlemcisinde yapılır. Ağırlıklar GPT-2 uyumlu olduğu için transformers ile doğrudan kullanılabilir; ayrıca repoda yalnızca NumPy ile çalışan özgün çıkarım motoru bulunur.

Küçük bir modeldir: kısa Türkçe sohbet, basit soru-cevap, yeniden yazma ve kısa özet işleri için uygundur. Uzun akıl yürütme, güncel bilgi ve karmaşık talimat takibi beklenmemelidir. Eğitim verisi Türkçe Vikipedi (CC BY-SA) ile Bilkent Türkçe Yazılar Veri Kümesi v2'yi (akademik, ticari olmayan) içerir.

Downloads last month
298
Safetensors
Model size
40.2M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train mete72722/anadolu-40m