File size: 13,308 Bytes
7b117b8 778f7ff 7b117b8 55a438a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 | <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="description" content="Soofi S 30B-A3B: A sovereign, open-source foundation model for German and English.">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Soofi S: A Sovereign, Open-Source Foundation Model for German and English</title>
<link href="https://fonts.googleapis.com/css?family=Google+Sans|Noto+Sans|Castoro" rel="stylesheet">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.4/css/bulma.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.4/css/all.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/jpswalsh/academicons@1/css/academicons.min.css">
<style>
body { font-family: 'Noto Sans', sans-serif; }
/* soofipurple from the report: RGB(231,206,246) */
.hero.is-primary { background-color: #e7cef6; }
.hero.is-primary .title, .hero.is-primary .subtitle { color: #363636; }
.publication-authors { line-height: 1.6; }
.author-group { margin-top: 0.6em; }
.stat-box { border: 1px solid #e0e0e0; border-radius: 8px; padding: 1em; height: 100%; }
.stat-box .heading { color: #7a7a7a; }
.link-block { margin: 0.15em; display: inline-block; }
figure img { max-width: 100%; height: auto; }
</style>
</head>
<body>
<section class="hero is-primary">
<div class="hero-body">
<div class="container has-text-centered">
<!-- Optional: replace the emoji with the project wordmark, e.g. <img src="logo.png" style="max-width:340px"> -->
<img src="logo.png" alt="Soofi S" style="max-width: 340px; height: auto;">
<p class="subtitle is-5">A sovereign, open-source Mixture-of-Experts hybrid Mamba–Transformer foundation model for German and English</p>
</div>
</div>
</section>
<section class="hero">
<div class="hero-body">
<div class="container is-max-desktop">
<div class="columns is-centered">
<div class="column has-text-centered">
<h1 class="title is-2 publication-title">A Sovereign, Open-Source Foundation Model for German and English</h1>
<p class="subtitle is-5">Soofi S Pretraining Report v1.0</p>
<div class="is-size-4 publication-authors">
<p class="author-group"><strong>The Soofi-Team</strong></p>
</div>
<div class="is-size-6 publication-authors" style="margin-top: 1em;">
<span class="author-block">KI Bundesverband,</span>
<span class="author-block">DFKI,</span>
<span class="author-block">Fraunhofer IAIS,</span>
<span class="author-block">Fraunhofer IIS,</span>
<span class="author-block">Technische Universität Darmstadt,</span>
<span class="author-block">Universität Würzburg,</span>
<span class="author-block">Berliner Hochschule für Technik,</span>
<span class="author-block">L3S Research Center,</span>
<span class="author-block">Lamarr,</span>
<span class="author-block">ellamind,</span>
<span class="author-block">hessian.AI,</span>
<span class="author-block">Merantix Momentum</span>
</div>
<p class="is-size-6" style="margin-top: 1em;">
Consortium coordinated by the KI Bundesverband. Funded by the German Federal Ministry for
Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra.
</p>
<div class="column has-text-centered" style="margin-top: 0.5em;">
<span class="link-block">
<a href="https://arxiv.org/abs/2607.09424" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="ai ai-arxiv"></i></span>
<span>arXiv</span>
</a>
</span>
<span class="link-block">
<a href="https://huggingface.co/Soofi-Project" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><img src="https://huggingface.co/front/assets/huggingface_logo.svg" alt="Hugging Face" style="height: 1.0em; vertical-align: middle;"></span>
<span>Models & Checkpoints</span>
</a>
</span>
<span class="link-block">
<a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fab fa-github"></i></span>
<span>Training & Data Code</span>
</a>
</span>
<span class="link-block">
<a href="https://github.com/ellamind/base-eval" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fab fa-github"></i></span>
<span>Eval: base-eval</span>
</a>
</span>
<span class="link-block">
<a href="https://github.com/ellamind/eval-hive" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fab fa-github"></i></span>
<span>Eval: eval-hive</span>
</a>
</span>
<span class="link-block">
<a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank" class="external-link button is-normal is-rounded is-dark">
<span class="icon"><i class="fas fa-chart-line"></i></span>
<span>Training Logs (W&B)</span>
</a>
</span>
</div>
<div class="notification is-warning is-light" style="margin-top: 1em;">
<strong>Note:</strong> During the current beta phase, the model repositories are gated —
you need to accept the access conditions on Hugging Face before downloading.
Once the beta phase ends, the models will be freely available without access request.
</div>
</div>
</div>
</div>
</div>
</section>
<section class="section">
<div class="container content is-max-desktop">
<p>
<strong>Soofi S 30B-A3B</strong> is a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba–Transformer
foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and
keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense
models for long-context, high-concurrency deployment.
</p>
<p>
Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14–27B
models on aggregate English and German benchmarks, achieves the best code aggregates in both languages among
17 open base models, and outperforms every European sovereign baseline in our comparison — including ones far
larger in active parameters. Among fully open models, it obtains the highest English and German evaluation
scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud,
a sovereign HPC-scale AI infrastructure operated by Deutsche Telekom in Munich.
</p>
</div>
</section>
<section class="section pt-0">
<div class="container is-max-desktop">
<h2 class="title is-3">📌 At a Glance</h2>
<div class="columns is-multiline">
<div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Parameters</p><p class="title is-4">31.6B total<br>3.2B active</p></div></div>
<div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Architecture</p><p class="title is-4">52 layers<br>23 Mamba-2 · 23 MoE · 6 GQA</p></div></div>
<div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Pretraining</p><p class="title is-4">~26.68T tokens<br>DE up to 15.3%</p></div></div>
<div class="column is-3"><div class="stat-box has-text-centered"><p class="heading">Context</p><p class="title is-4">up to 1M<br>tokens</p></div></div>
</div>
</div>
</section>
<section class="section">
<div class="container content is-max-desktop">
<h2 class="title is-3">🧩 Highlights</h2>
<ol>
<li><strong>🏆 German–English champion:</strong> best English and German code aggregates among all 17 measured
open base models; strongest fully open model on the English and German aggregates; matches or outperforms
every European sovereign baseline on every German benchmark in our suite — at a fraction of the
active-parameter cost of dense 14–27B models.</li>
<li><strong>📋 Full data transparency:</strong> complete per-source, per-language token accounting for all
three training phases (including sources we evaluated and <em>excluded</em>), with reproducible corpus
construction scripts. ~99% of the mixture can be independently reconstructed.</li>
<li><strong>🔁 Reproducible recipe:</strong> full Warmup–Stable–Decay learning-rate schedule, optimizer, all
hyperparameters, per-phase token budgets, and phase boundaries — a third party can rebuild the run.</li>
<li><strong>⚡ Long-context serving efficiency:</strong> only 6 of 52 layers keep a KV cache, so the
per-sequence cache stays near-constant with context length. Measured aggregate decode TPS/GPU is 8–9× that
of dense 14–24B models at 40K context (batch 32) and stays essentially flat from 4K to 256K.</li>
<li><strong>🇩🇪 Sovereign end to end:</strong> trained from 24 March to 13 May 2026 on up to 512 NVIDIA B200
GPUs of the German Industrial AI Cloud (Deutsche Telekom, Munich), under European operational and
data-protection requirements.</li>
</ol>
</div>
</section>
<section class="section">
<div class="container content is-max-desktop">
<h2 class="title is-3">📊 Results</h2>
<ul>
<li><strong>Aggregates:</strong> English 70.1 / German 79.1 — highest among fully open models (Olmo 3 32B:
67.3 / 69.2; Apertus 70B: 62.4 / 72.8), on par with dense open-weight 14–27B models.</li>
<li><strong>Code:</strong> HumanEval 73.8, MBPP 70.2, MBPP-DE 84.2 — best in both comparison sets.</li>
<li><strong>Mathematics:</strong> GSM8K 86.1, GSM8K-Platinum-DE 87.1, Minerva-500 79.4.</li>
<li><strong>German:</strong> first on every German benchmark against European open-source baselines
(GLP-DE 88.8, INCLUDE-DE 61.2, ARC-Challenge-DE 92.3).</li>
<li><strong>Serving:</strong> 4.82k aggregate decode TPS/GPU at 40K context (batch 32, single B200, vLLM) —
9.2× Ministral 3 14B — with near-flat decode throughput from 4K to 256K.</li>
</ul>
</div>
</section>
<section class="section">
<div class="container content is-max-desktop">
<h2 class="title is-3">📁 Available Artifacts</h2>
<ul>
<li><a href="https://huggingface.co/Soofi-Project" target="_blank">🤗 Soofi S 30B-A3B Base weights and selected intermediate checkpoints</a>
<em>(currently gated during the beta phase; freely available once the beta ends)</em></li>
<li><a href="https://github.com/soofi-project/Soofi-Pretraining" target="_blank">🛠️ Training code and reproducible data-construction scripts</a></li>
<li><a href="https://github.com/ellamind/base-eval" target="_blank">🧪 Evaluation code (base-eval)</a> and <a href="https://github.com/ellamind/eval-hive" target="_blank">eval-hive</a>, incl. parallel English/German suites</li>
<li><a href="https://api.wandb.ai/links/soofi-exchange/j11vi7rg" target="_blank">📈 Weights & Biases dashboard of the full ~27T-token run</a></li>
<li>📋 Exact per-source token accounting for all three phases; commercially licensed sources (Genios) documented via aggregate statistics</li>
</ul>
<p>All artifacts are released under permissive licenses for transparent audit and extension.</p>
</div>
</section>
<section class="section">
<div class="container content is-max-desktop">
<h2 class="title is-3">📜 Citation</h2>
<p>If you use Soofi S or its artifacts, please cite the report
(<a href="https://arxiv.org/abs/2607.09424" target="_blank">arXiv:2607.09424</a>):</p>
<pre><code>@misc{soofi2026soofis,
title = {A Sovereign, Open-Source Foundation Model for German and English: Soofi S Pretraining Report v1.0},
author = {{The Soofi-Team}},
year = {2026},
eprint = {2607.09424},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2607.09424}
}</code></pre>
</div>
</section>
<footer class="footer">
<div class="container content has-text-centered is-size-7">
<p>
Funded by the German Federal Ministry for Economic Affairs and Energy (BMWE) in the context of IPCEI-CIS and 8ra
through “Soofi: Souveräne KI für Europa” (grant no. 13IPC040A-J).
Website template adapted from the <a href="https://nerfies.github.io" target="_blank">Nerfies</a>/JQL project pages (Bulma).
</p>
</div>
</footer>
</body>
</html>
|