| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="description" content="Soofi S 30B-A3B: A sovereign, open-source foundation model for German and English."> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Soofi S: A Sovereign, Open-Source Foundation Model for German and English</title> |
| <link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet"> |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css"> |
| <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css"> |
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css"> |
| <style> |
| body { font-family: 'Noto Sans', sans-serif; } |
| |
| .hero.is-primary { background-color: #e7cef6; } |
| .hero.is-primary .title, .hero.is-primary .subtitle { color: #363636; } |
| .publication-authors { line-height: 1.6; } |
| .author-group { margin-top: 0.6em; } |
| .stat-box { border: 1px solid #e0e0e0; border-radius: 8px; padding: 1em; height: 100%; } |
| .stat-box .heading { color: #7a7a7a; } |
| .link-block { margin: 0.15em; display: inline-block; } |
| figure img { max-width: 100%; height: auto; } |
| </style> |
| </head> |
| <body> |
|
|
| <section class="hero is-primary"> |
| <div class="hero-body"> |
| <div class="container has-text-centered"> |
| |
| <img src="logo.png" alt="Soofi S" style="max-width: 340px; height: auto;"> |
| <p class="subtitle is-5">A sovereign, open-source Mixture-of-Experts hybrid Mamba–Transformer foundation model for German and English</p> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="hero"> |
| <div class="hero-body"> |
| <div class="container is-max-desktop"> |
| <div class="columns is-centered"> |
| <div class="column has-text-centered"> |
| <h1 class="title is-2 publication-title">A Sovereign, Open-Source Foundation Model for German and English</h1> |
| <p class="subtitle is-5">Soofi S Pretraining Report v1.0</p> |
|
|
| <div class="is-size-4 publication-authors"> |
| <p class="author-group"><strong>The Soofi-Team</strong></p> |
| </div> |
|
|
| <div class="is-size-6 publication-authors" style="margin-top: 1em;"> |
| <span class="author-block">KI Bundesverband,</span> |
| <span class="author-block">DFKI,</span> |
| <span class="author-block">Fraunhofer IAIS,</span> |
| <span class="author-block">Fraunhofer IIS,</span> |
| <span class="author-block">Technische Universität Darmstadt,</span> |
| <span class="author-block">Universität Würzburg,</span> |
| <span class="author-block">Berliner Hochschule für Technik,</span> |
| <span class="author-block">L3S Research Center,</span> |
| <span class="author-block">Lamarr,</span> |
| <span class="author-block">ellamind,</span> |
| <span class="author-block">hessian.AI,</span> |
| <span class="author-block">Merantix Momentum</span> |
| </div> |
|
|
| <p class="is-size-6" style="margin-top: 1em;"> |
| Consortium coordinated by the KI Bundesverband. Funded by the German Federal Ministry for |
| Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra. |
| </p> |
|
|
| <div class="column has-text-centered" style="margin-top: 0.5em;"> |
| <span class="link-block"> |
| <a href="https://arxiv.org/abs/2607.09424" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="ai ai-arxiv"></i></span> |
| <span>arXiv</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://huggingface.co/Soofi-Project" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><img src="https://huggingface.co/front/assets/huggingface_logo.svg" alt="Hugging Face" style="height: 1.0em; vertical-align: middle;"></span> |
| <span>Models & Checkpoints</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fab fa-github"></i></span> |
| <span>Training & Data Code</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://github.com/ellamind/base-eval" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fab fa-github"></i></span> |
| <span>Eval: base-eval</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://github.com/ellamind/eval-hive" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fab fa-github"></i></span> |
| <span>Eval: eval-hive</span> |
| </a> |
| </span> |
| <span class="link-block"> |
| <a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank" class="external-link button is-normal is-rounded is-dark"> |
| <span class="icon"><i class="fas fa-chart-line"></i></span> |
| <span>Training Logs (W&B)</span> |
| </a> |
| </span> |
| </div> |
|
|
| <div class="notification is-warning is-light" style="margin-top: 1em;"> |
| <strong>Note:</strong> During the current beta phase, the model repositories are gated — |
| you need to accept the access conditions on Hugging Face before downloading. |
| Once the beta phase ends, the models will be freely available without access request. |
| </div> |
| </div> |
| </div> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content is-max-desktop"> |
| <p> |
| <strong>Soofi S 30B-A3B</strong> is a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba–Transformer |
| foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and |
| keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense |
| models for long-context, high-concurrency deployment. |
| </p> |
| <p> |
| Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14–27B |
| models on aggregate English and German benchmarks, achieves the best code aggregates in both languages among |
| 17 open base models, and outperforms every European sovereign baseline in our comparison — including ones far |
| larger in active parameters. Among fully open models, it obtains the highest English and German evaluation |
| scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, |
| a sovereign HPC-scale AI infrastructure operated by Deutsche Telekom in Munich. |
| </p> |
| </div> |
| </section> |
|
|
| <section class="section pt-0"> |
| <div class="container is-max-desktop"> |
| <h2 class="title is-3">📌 At a Glance</h2> |
| <div class="columns is-multiline"> |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Parameters</p><p class="title is-4">31.6B total<br>3.2B active</p></div></div> |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Architecture</p><p class="title is-4">52 layers<br>23 Mamba-2 · 23 MoE · 6 GQA</p></div></div> |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Pretraining</p><p class="title is-4">~26.68T tokens<br>DE up to 15.3%</p></div></div> |
| <div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Context</p><p class="title is-4">up to 1M<br>tokens</p></div></div> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content is-max-desktop"> |
| <h2 class="title is-3">🧩 Highlights</h2> |
| <ol> |
| <li><strong>🏆 German–English champion:</strong> best English and German code aggregates among all 17 measured |
| open base models; strongest fully open model on the English and German aggregates; matches or outperforms |
| every European sovereign baseline on every German benchmark in our suite — at a fraction of the |
| active-parameter cost of dense 14–27B models.</li> |
| <li><strong>📋 Full data transparency:</strong> complete per-source, per-language token accounting for all |
| three training phases (including sources we evaluated and <em>excluded</em>), with reproducible corpus |
| construction scripts. ~99% of the mixture can be independently reconstructed.</li> |
| <li><strong>🔁 Reproducible recipe:</strong> full Warmup–Stable–Decay learning-rate schedule, optimizer, all |
| hyperparameters, per-phase token budgets, and phase boundaries — a third party can rebuild the run.</li> |
| <li><strong>⚡ Long-context serving efficiency:</strong> only 6 of 52 layers keep a KV cache, so the |
| per-sequence cache stays near-constant with context length. Measured aggregate decode TPS/GPU is 8–9× that |
| of dense 14–24B models at 40K context (batch 32) and stays essentially flat from 4K to 256K.</li> |
| <li><strong>🇩🇪 Sovereign end to end:</strong> trained from 24 March to 13 May 2026 on up to 512 NVIDIA B200 |
| GPUs of the German Industrial AI Cloud (Deutsche Telekom, Munich), under European operational and |
| data-protection requirements.</li> |
| </ol> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content is-max-desktop"> |
| <h2 class="title is-3">📊 Results</h2> |
| <ul> |
| <li><strong>Aggregates:</strong> English 70.1 / German 79.1 — highest among fully open models (Olmo 3 32B: |
| 67.3 / 69.2; Apertus 70B: 62.4 / 72.8), on par with dense open-weight 14–27B models.</li> |
| <li><strong>Code:</strong> HumanEval 73.8, MBPP 70.2, MBPP-DE 84.2 — best in both comparison sets.</li> |
| <li><strong>Mathematics:</strong> GSM8K 86.1, GSM8K-Platinum-DE 87.1, Minerva-500 79.4.</li> |
| <li><strong>German:</strong> first on every German benchmark against European open-source baselines |
| (GLP-DE 88.8, INCLUDE-DE 61.2, ARC-Challenge-DE 92.3).</li> |
| <li><strong>Serving:</strong> 4.82k aggregate decode TPS/GPU at 40K context (batch 32, single B200, vLLM) — |
| 9.2× Ministral 3 14B — with near-flat decode throughput from 4K to 256K.</li> |
| </ul> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content is-max-desktop"> |
| <h2 class="title is-3">📁 Available Artifacts</h2> |
| <ul> |
| <li><a href="https://huggingface.co/Soofi-Project" target="_blank">🤗 Soofi S 30B-A3B Base weights and selected intermediate checkpoints</a> |
| <em>(currently gated during the beta phase; freely available once the beta ends)</em></li> |
| <li><a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank">🛠️ Training code and reproducible data-construction scripts</a></li> |
| <li><a href="https://github.com/ellamind/base-eval" target="_blank">🧪 Evaluation code (base-eval)</a> and <a href="https://github.com/ellamind/eval-hive" target="_blank">eval-hive</a>, incl. parallel English/German suites</li> |
| <li><a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank">📈 Weights & Biases dashboard of the full ~27T-token run</a></li> |
| <li>📋 Exact per-source token accounting for all three phases; commercially licensed sources (Genios) documented via aggregate statistics</li> |
| </ul> |
| <p>All artifacts are released under permissive licenses for transparent audit and extension.</p> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <div class="container content is-max-desktop"> |
| <h2 class="title is-3">📜 Citation</h2> |
| <p>If you use Soofi S or its artifacts, please cite the report |
| (<a href="https://arxiv.org/abs/2607.09424" target="_blank">arXiv:2607.09424</a>):</p> |
| <pre><code>@misc{soofi2026soofis, |
| title = {A Sovereign, Open-Source Foundation Model for German and English: Soofi S Pretraining Report v1.0}, |
| author = {{The Soofi-Team}}, |
| year = {2026}, |
| eprint = {2607.09424}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CL}, |
| url = {https://arxiv.org/abs/2607.09424} |
| }</code></pre> |
| </div> |
| </section> |
|
|
| <footer class="footer"> |
| <div class="container content has-text-centered is-size-7"> |
| <p> |
| Funded by the German Federal Ministry for Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra |
| through “Soofi: Souveräne KI für Europa” (grant no. 13IPC040A-J). |
| Website template adapted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a>/JQL project pages (Bulma). |
| </p> |
| </div> |
| </footer> |
|
|
| </body> |
| </html> |
|
|
