🦋 SOVYTHOS-66M-Base

The first open-source pretrained language model developed completely from scratch under the Sovereign AI Project.

---

🦋 Overview

SOVYTHOS-66M-Base is a decoder-only Transformer language model trained entirely from scratch as part of the Sovereign AI Project.

Unlike models that continue training from existing checkpoints, SOVYTHOS begins from randomly initialized weights and learns language directly from its training corpus.

This repository contains the Base (Pretrained) release intended for:

• Continued Pretraining
• Fine-tuning
• Instruction Tuning
• Research
• Educational Purposes
• Building downstream AI applications

This model is NOT instruction tuned and is not intended to behave like a production chatbot.

---

🚀 Experience the Sovereign Platform

SOVYTHOS is the foundation of the Sovereign AI Platform.

If you'd like to experience the latest AI capabilities through a complete interface, you can try the live platform below.

🌐 Live Demo

https://awaken-creation-guidance.ngrok-free.dev/

Available Features

• 💬 Modern AI Chat Interface
• ⚡ Flash AI Models
• 🧠 Advanced Reasoning Models
• 💻 Coding Assistant
• 🌍 Arabic & English Support
• 🔥 Continuous Platform Updates

Note: The Sovereign platform may use newer models, APIs, and experimental capabilities beyond this Base release.

---

📊 Model Specifications

Property	Value
Model	SOVYTHOS-66M-Base
Parameters	~66.7 Million
Architecture	Decoder-only Transformer
Hidden Size	512
Layers	12
Attention Heads	16
Framework	PyTorch
Tokenizer	Custom
Context Length	Configured during training
Training	From Scratch
Release	Base

---

🌍 Languages

SOVYTHOS has been trained on multilingual data including:

• 🇪🇬 Arabic
• 🇪🇬 Egyptian Arabic
• 🇺🇸 English
• Programming Languages
• Technical Documentation
• Mathematical Content
• General Web Text

---

🎯 Intended Uses

This release is designed for:

• Research
• Fine-tuning
• Continued Pretraining
• Instruction Tuning
• Educational Experiments
• Language Model Development
• Custom AI Applications

---

⚠️ Current Limitations

This repository contains the Base Language Model only.

It has NOT been instruction tuned.

Current limitations include:

• Hallucination of factual information
• Incorrect reasoning in some cases
• Repetition
• Incomplete generations
• Limited instruction following

These limitations are expected for a pretrained base model and will be addressed in future releases.

---

💻 Example

prompt = "The capital of Egypt is"

output = model.generate(prompt)

---

🏗️ Training

SOVYTHOS was trained completely from scratch using:

• Custom Tokenizer
• PyTorch
• CUDA
• Mixed Precision Training
• Automatic Best Checkpoint Saving

No pretrained language model weights were used.

---

🛣️ Roadmap

✅ SOVYTHOS-66M-Base

🟡 SOVYTHOS-66M-Instruct (In Development)

⬜ SOVYTHOS Chat

⬜ Larger Parameter Models

⬜ Advanced Reasoning

⬜ Improved Coding

⬜ Better Arabic Understanding

⬜ Long Context Support

---

🌌 Sovereign Project

SOVYTHOS is the first public language model released under the Sovereign AI Project.

The long-term vision of Sovereign is to build an open ecosystem of multilingual language models with strong Arabic support, modern reasoning capabilities, and accessible open-source AI technologies.

---

📖 Citation

@misc{sovythos66m,
  title={SOVYTHOS-66M-Base},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
}

---

👨‍💻 Creator

Mahmoud Yasser

Founder of the Sovereign AI Project

---

📄 License

Apache License 2.0

---

🦋 Welcome to the beginning of the Sovereign ecosystem.

The current release represents the first public pretrained foundation model.

Future releases will introduce instruction tuning, stronger reasoning, improved coding capabilities, and more advanced multilingual intelligence.

⭐ If you find this project interesting, consider following the repository and trying the Sovereign AI Platform.