Sovythos-V2-Instruct
Egyptian-built instruction-tuned language model
A 303M-parameter decoder-only Transformer, pretrained from randomly initialized weights and then fine-tuned to follow instructions in Egyptian Arabic, Modern Standard Arabic, and English.
A note on what this actually is
This card is written the way I'd want to read it if I were deciding whether to use this model: plainly, with real numbers, and without inflating what a 303-million-parameter model trained on a single GPU can realistically do.
Sovythos-V2-Instruct is not a competitor to GPT-4, Gemini, or even most 7B open models. It is a small, fully from-scratch model โ architecture, tokenizer, pretraining, and instruction tuning all built and trained independently, without initializing from any existing foundation model's weights. At this parameter count, that comes with real trade-offs: limited world knowledge, occasional factual errors, and a ceiling on how complex a task it can reliably handle. What it does represent is the first checkpoint in this project that was actually trained to hold a conversation, rather than just to predict the next token in raw text.
If you're looking for state-of-the-art performance, this isn't it, and it isn't trying to be. If you're interested in what a small, transparently-documented, independently-trained Arabic/Egyptian model looks like โ including its rough edges โ this is that.
Overview
Sovythos-V2-Instruct starts from Sovythos-V2-Base, a 303M-parameter Llama-style decoder pretrained from scratch (RMSNorm, RoPE, grouped-query attention, SwiGLU), and applies supervised fine-tuning (SFT) on top of it: masked-loss instruction tuning where the training signal comes only from the assistant's responses, not from the prompt tokens.
The instruction data mixes three things:
- General-purpose instruction data in English and Arabic, drawn from established public datasets (see Training Data below for exact sources).
- Egyptian Arabic conversational data, since dialectal Egyptian is underrepresented in almost every open instruction dataset, and it's the dialect this project cares about most.
- A small, hand-written identity set โ a few dozen examples establishing who built the model, what it's called, and why, repeated a modest number of times during training so the model has a consistent answer when asked about itself, without that identity data dominating the rest of the mixture.
Everything below โ the loss curves, the hyperparameters, the data sources โ reflects what was actually run, not a idealized description of an eventual final version.
Model Specifications
| Property | Value |
|---|---|
| Base architecture | Sovythos-V2-Base (decoder-only, Llama-style) |
| Parameters | ~303M (270.6M non-embedding) |
| Hidden size | 1,024 |
| Layers | 24 |
| Attention heads | 16 query / 4 key-value (GQA) |
| Vocabulary size | 32,000 |
| Training context (SFT) | 1,024 tokens |
| Fine-tuning method | Supervised fine-tuning with per-token loss masking |
| Precision | FP16 mixed precision, FP32 master weights |
| Framework | PyTorch, from-scratch implementation (no external training frameworks) |
The architecture itself is unchanged from the base model; nothing here is novel about the network design. What changed is purely the training objective and the data.
Training Data
Instruction data was assembled from a mix of public datasets and one hand-written set, with automatic fallbacks configured for each category in case a specific dataset became unavailable. What was actually used in the run behind this checkpoint:
| Category | Dataset used | Fallback configured | Examples pulled |
|---|---|---|---|
| Egyptian Arabic conversational | ahmedsamirio/oasst2-9k-translation |
ISLAM-PO/documents-Egyptian-Arabic, HeshamHaroon/ArzEn-MultiGenre |
~9,400 |
| Arabic (MSA) instructions | adlbh/alpaca_clean_arabic |
Yasbok/Alpaca_arabic_instruct |
~20,000 |
| English instructions (general) | tatsu-lab/alpaca |
yahma/alpaca-cleaned |
~20,000 |
| English instructions (higher quality, human-written) | HuggingFaceH4/no_robots |
databricks/databricks-dolly-15k |
~9,400 |
| Model identity | Hand-written, this project | โ | ~130 (oversampled 6ร) |
That's roughly 59,000 instruction examples total, split into train and validation sets before training. This is a modest dataset by modern instruction-tuning standards โ large SFT runs for comparable open models often use several hundred thousand to a few million examples. The smaller scale here is a direct consequence of available compute (see Training Infrastructure) and is one of the clearest places where quality will improve as the project's resources grow.
Each example was formatted into the model's native chat template (<|system|>, <|user|>, <|assistant|> special tokens) with a roughly 50% chance of including a system prompt, to encourage the model to behave reasonably whether or not one is present at inference time.
Training Recipe
| Hyperparameter | Value |
|---|---|
| Optimizer | AdamW (ฮฒโ=0.9, ฮฒโ=0.95, ฮต=1e-8) |
| Peak learning rate | 2e-5 |
| Minimum learning rate | 2e-6 |
| LR schedule | Cosine decay with linear warmup |
| Warmup ratio | 3% of total steps |
| Weight decay | 0.01 (2D+ tensors only) |
| Batch size | 8 (per step) |
| Gradient accumulation | 4 |
| Effective batch size | 32 sequences (~32,768 tokens/step) |
| Max gradient norm | 1.0 |
| Loss | Cross-entropy, masked to assistant tokens only |
| Epochs (planned) | 3 |
Loss trajectory
Training was monitored on a held-out validation split. Approximate validation loss at selected checkpoints from the run behind this model:
| Step | Val. loss | Approx. perplexity |
|---|---|---|
| 0 | 10.50 | ~36,300 (near-random, vocab size is 32,000) |
| 125 | 10.09 | ~24,100 |
| 450 | 7.76 | ~2,340 |
| 1,035 | 4.86 | ~129 |
For context: a loss around 4.9 corresponds to the model still being meaningfully uncertain between roughly 130 plausible next tokens on average. Coherent, chat-quality generation on models of this type generally starts to emerge somewhere in the 2.5โ3.0 loss range. This means the checkpoint currently published is an intermediate point in training, not a converged one โ treat any example outputs from this stage as a demonstration of the training process working correctly, not as representative final quality.
This card will be updated with the final loss curve and revised expectations once a full training pass completes.
Training Infrastructure
This model was trained on a single consumer/free-tier GPU instance (Google Colab), which comes with real constraints worth being upfront about:
- Session limits. Free-tier Colab sessions can disconnect after a fixed number of hours or after a period of inactivity, interrupting training runs mid-epoch.
- No guaranteed hardware. GPU allocation varies session to session; throughput is not constant.
- A full 3-epoch pass over ~57,800 training examples at this batch size takes roughly a day of continuous compute โ longer than a single free Colab session typically allows, which means this training run has been (and likely will continue to be) resumed across multiple sessions from saved checkpoints rather than completed in one sitting.
This is mentioned not as an excuse but because it directly explains why this is being released as a checkpoint mid-training rather than a single finished artifact: the infrastructure available for this project is modest, and the honest path is to publish progress transparently rather than wait for ideal conditions that may not arrive.
Intended Uses
- Evaluating instruction-following behavior in a small, from-scratch Arabic/Egyptian model
- Research on low-resource dialectal Arabic fine-tuning
- Continued fine-tuning or further SFT on top of this checkpoint
- Educational reference for building an instruction-tuning pipeline from a custom pretrained base
- Community feedback on architecture and data choices before scaling up
Not intended for: production deployment as a customer-facing assistant, use cases requiring reliable factual accuracy, or any application where hallucinated or incoherent output carries real consequences. At this parameter count and training stage, none of those are safe assumptions to make about this model's behavior.
Limitations
Stated directly, without hedging:
- Small parameter count. 303M parameters is small by current standards; the model's world knowledge, reasoning depth, and long-context coherence are all correspondingly limited compared to larger open or closed models.
- Intermediate training checkpoint. As shown above, the validation loss at the time of this checkpoint indicates the model is not yet at the quality level a fully-trained instruct model of this size could reach.
- Modest instruction dataset. ~59K examples is workable but small; expect inconsistency on less common instruction phrasings or task types not well represented in the mixture.
- Egyptian dialect coverage is a work in progress. Dialectal data is the smallest of the four categories used; MSA and English are currently better represented than Egyptian Arabic in absolute example count, even though Egyptian coverage was a primary goal.
- Hallucination. Like any model at this scale, it will confidently state incorrect facts, especially about specific names, dates, or technical details outside its training data.
- Not safety-tuned. No RLHF or dedicated safety fine-tuning has been applied. Standard content moderation and use-case-appropriate guardrails should be layered on top if this model is used in any user-facing context.
Usage
Loading with the native training code
import torch
from tokenizers import Tokenizer
from model import Model
tok = Tokenizer.from_file("sovythos_tokenizer.json")
model = Model.from_pretrained("sovythos_sft_ckpt/best", device="cuda")
model.eval()
eos_id = tok.token_to_id("<|endoftext|>")
prompt = "<|user|>\nุฅุฒูู ูุง ุณูููุซูุณุ ุนุฑููู ุจููุณู\n<|assistant|>\n"
input_ids = torch.tensor([tok.encode(prompt).ids], device="cuda")
output = model.generate(
input_ids,
max_new_tokens=200,
temperature=0.7,
top_k=40,
top_p=0.9,
repetition_penalty=1.15,
eos_token_id=eos_id,
)
print(tok.decode(output[0].tolist(), skip_special_tokens=True))
Chat template
<|system|>
{system prompt, optional}
<|user|>
{user message}
<|assistant|>
{model response}<|endoftext|>
The system prompt is optional at inference time โ the model was trained with one present roughly half the time, so it should behave reasonably either way, though including one generally produces more consistent tone.
Roadmap
| Milestone | Status |
|---|---|
| Sovythos-V2-Base | Released |
| Sovythos-V2-Instruct (this checkpoint) | In progress โ intermediate SFT checkpoint |
| Sovythos-V2-Instruct (converged) | Planned, pending further training compute |
| Larger instruction dataset (Egyptian-weighted) | Planned |
| GGUF export of instruct weights | Planned |
| Dedicated instance with stable training infrastructure | Under consideration |
The Sovythos Project
Sovythos is an independent, Egyptian-built language model project. The name combines sovereign and mythos โ the intent being a model with its own independent lineage (built without initializing from an existing foundation model's weights) and its own identity, rather than a repackaging of someone else's work.
This is a project built and trained by one person, Mahmoud Yasser, under real hardware constraints. It is shared publicly, including at intermediate and imperfect stages, in the belief that transparent, documented small-model development has value on its own terms โ separate from whether it can compete with frontier labs on raw benchmark numbers. It can't, and pretending otherwise would undermine the point.
If there is a reason for cautious pride here, it isn't that Sovythos is the best model available. It's that it's a real, working instruction-tuning pipeline โ pretraining, tokenizer, architecture, and now supervised fine-tuning โ built independently and end-to-end, running on constrained infrastructure, with results that are documented honestly rather than dressed up.
Citation
@misc{sovythosv2instruct,
title={Sovythos-V2-Instruct: An Egyptian-Built Instruction-Tuned Language Model},
author={Mahmoud Yasser},
year={2026},
publisher={Hugging Face},
howpublished={\url{https://huggingface.co/sovythos/Sovythos-V2-Instruct}}
}
Creator
Mahmoud Yasser โ Founder, Sovythos AI Project.
License
Apache License 2.0.
This is a checkpoint from an active, ongoing training process, shared as-is with its current strengths and limitations documented above. If you use it, evaluate it on your own tasks before relying on it โ that applies to any model at this scale, and doubly to one still mid-training.