| --- |
| license: other |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - soofi |
| - mamba-2 |
| - moe |
| - text-generation |
| - sovereign-ai |
| - base |
| - preview |
| language: |
| - en |
| - de |
| --- |
| |
| # Soofi-S-30B-A3B Overview |
|
|
| > ⚠️ **Preview / internal checkpoint.** Weights and metadata may still change. |
|
|
| ## Description |
|
|
| **Soofi-S-30B-A3B** is a pre-trained base model that generates free-form text continuations for a given prompt. |
| It was developed by a German research consortium. SOOFI (Sovereign Open |
| Source Foundation Models) is designed to provide a secure, European open-source |
| alternative to US and Chinese AI models for industrial use, featuring strong |
| reasoning and AI agent capabilities. |
|
|
| As a base model, Soofi-S-30B-A3B has **not** been instruction-tuned, aligned, |
| or safety-tuned. It is intended as a foundation for further post-training |
| (e.g. SFT, preference tuning, domain adaptation) and for research — not for |
| direct deployment as an end-user assistant. |
|
|
| For general assistant and instruction-following tasks, see the post-trained |
| variant |
| [Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview). |
| For explicit chain-of-thought reasoning, see the thinking variants |
| [Soofi-S-Isar-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Isar-Preview) |
| and [Soofi-S-Rhine-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Rhine-Preview). |
|
|
| This model is for research and development only (Preview). |
|
|
| ## License/Terms of Use |
|
|
| Released under a custom license ("Other"). TODO: add the full license text / |
| link — the official card references a License section that is not yet filled in. |
|
|
| ### Deployment Geography |
|
|
| Global (open release on the Hugging Face Hub). Development and training |
| infrastructure are located in Europe (see Computational Load). |
|
|
| ### Use Case |
|
|
| Enterprise developers and researchers seeking a sovereign, European open-source |
| base LLM as a starting point for fine-tuning, continued pre-training, domain |
| adaptation, and LLM research (e.g. building custom assistant or agent models). |
| English and German are the primary languages. |
|
|
| ### Release Date |
|
|
| Hugging Face Hub — Preview at |
| <https://huggingface.co/Soofi-Project/Soofi-S-30B-A3B>. TODO: final |
| release date (MM/DD/YYYY). |
|
|
| ## Reference(s) |
|
|
| - Project: <https://soofi.info> |
| - Related models: see the *Related models* section below. |
| - TODO: link the technical report / paper once published. |
|
|
| ## Model Architecture |
|
|
| **Architecture Type:** Transformer-based hybrid Mixture-of-Experts (MoE) with |
| Mamba-2 state-space (SSM) layers and attention layers. <br> |
| **Network Architecture:** Custom Hybrid Mamba-2/MoE (Nemotron-style), designed |
| from scratch — 23 Mamba-2/MoE layers + 6 attention layers; 128 routing experts |
| + 1 shared expert per MoE layer; 6 experts activated per token. <br> |
|
|
| **This model was developed from scratch** (no base model). <br> |
| **Number of model parameters:** 3.0×10^10 total (30B), with ~3.5B active |
| parameters during inference. |
|
|
| ## Computational Load |
|
|
| **Cumulative Compute:** TODO. <br> |
| **Estimated Energy and Emissions for Model Training:** TODO. Training |
| infrastructure is hosted entirely in Europe on T-Systems' Industrial AI Cloud |
| (Deutsche Telekom) to ensure data sovereignty. |
|
|
| ## Input |
|
|
| **Input Type(s):** Text <br> |
| **Input Format(s):** String <br> |
| **Input Parameters:** One-Dimensional (1D) <br> |
| **Other Properties Related to Input:** Plain-text prompts (completion-style). |
| As a base model, it uses no chat template and expects no system prompt. Context |
| length: see `config.json` (TODO: confirm maximum context). |
|
|
| ## Output |
|
|
| **Output Type(s):** Text <br> |
| **Output Format(s):** String <br> |
| **Output Parameters:** One-Dimensional (1D) <br> |
| **Other Properties Related to Output:** Raw text continuation of the input |
| prompt (next-token prediction). No explicit reasoning trace, no chat or |
| tool-calling format, and no alignment for helpfulness or safety. |
|
|
| ## Software Integration |
|
|
| **Runtime Engine(s):** <br> |
| * Hugging Face `transformers` (`trust_remote_code=True`) <br> |
| * vLLM, llama.cpp/Ollama via quantized variants (TODO: link once available) <br> |
|
|
| **Supported Hardware Microarchitecture Compatibility:** <br> |
| * NVIDIA GPUs (Ampere and newer recommended) <br> |
|
|
| **Preferred/Supported Operating System(s):** <br> |
| * Linux <br> |
|
|
| The integration of foundation and fine-tuned models into AI systems requires |
| additional testing using use-case-specific data to ensure safe and effective |
| deployment. |
|
|
| ## Model Version(s) |
|
|
| * **Soofi-S-30B-A3B** — bf16 safetensors, unquantized (this repo). <br> |
| * Post-trained variants: Instruct and thinking models (see *Related models*). <br> |
| * Quantized derivatives (`…-GGUF`, `…-FP8`): TODO — link once available. |
|
|
| ## Installation & Usage |
|
|
| SOOFI-S ships with custom modeling code. You must load it using `trust_remote_code=True` with `transformers`. |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "Soofi-Project/Soofi-S-30B-A3B" |
| tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto" |
| ) |
| |
| # Base model: plain text completion — no chat template, no system prompt. |
| prompt = "AI sovereignty is the idea that" |
| inputs = tok(prompt, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=128) # sampling defaults come from generation_config.json |
| |
| print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| # Training, Testing, and Evaluation Datasets |
|
|
| ### Dataset Overview |
|
|
| * **Total Size:** ~2.5×10^13 tokens (25 trillion). <br> |
| * **Languages:** English, German (primary); French, Italian, Spanish (limited). |
| English acts as the pivot language. <br> |
| * **Knowledge Cutoff:** End of 2025. <br> |
| * **Training Start:** April 2026. |
|
|
| ## Training Dataset |
|
|
| **Link:** TODO. <br> |
| **Data Modality:** Text. <br> |
| **Text Training Data Size:** More than 10 Trillion Tokens (~25T). <br> |
| **Data Collection Method by dataset:** Hybrid (freely available, high-quality |
| sources). TODO: refine. <br> |
| **Labeling Method by dataset:** TODO. <br> |
| **Properties:** Trained entirely from scratch on freely available, high-quality |
| tokens. |
|
|
| ## Testing Dataset |
|
|
| **Link:** TODO. <br> |
| **Properties:** TODO. |
|
|
| ## Evaluation Dataset |
|
|
| **Link:** TODO. <br> |
| **Benchmark Score:** TODO — add key base-model benchmarks (e.g. few-shot |
| reasoning, multilingual) once available. <br> |
| **Properties:** TODO. |
|
|
| ## Inference |
|
|
| **Acceleration Engine:** `transformers`; vLLM / llama.cpp via quantized |
| variants (TODO). <br> |
| **Specific Test Hardware:** TODO. |
|
|
| ## Ethical Considerations |
|
|
| The SOOFI consortium believes Trustworthy AI is a shared responsibility and has |
| established policies and practices to enable development for a wide array of AI |
| applications. When downloaded or used, developers should work with their |
| internal model team to ensure this model meets requirements for the relevant |
| industry and use case and addresses unforeseen product misuse. Note that |
| Soofi-S-30B-A3B is an unaligned base model: it has undergone no instruction |
| tuning or safety alignment, and downstream developers are responsible for |
| appropriate post-training, evaluation, and guardrails before any deployment. |
|
|
| For more detailed information, see the Model Card++ subcards below. Please report |
| model quality, risk, security vulnerabilities, or concerns to |
| <a href="mailto:contact@soofi.info">contact@soofi.info</a>. |
|
|
| ### Bias Subcard |
|
|
| | Field | Response | |
| |:---|:---| |
| | Participation considerations from adversely impacted groups in model design and testing | TODO | |
| | Measures taken to mitigate against unwanted bias | TODO | |
| | Bias Metric (if measured) | TODO | |
|
|
| ### Explainability Subcard |
|
|
| | Field | Response | |
| |:---|:---| |
| | Intended Task/Domain | Text completion; foundation (base) model for downstream fine-tuning and research | |
| | Model Type | Hybrid Mixture-of-Experts (MoE) autoregressive language model | |
| | Intended Users | Enterprise developers and researchers (fine-tuning and research, not end users) | |
| | Output | Text (String) | |
| | Describe how the model works | Generates text autoregressively; a router activates 6 of 128 experts per token across hybrid Mamba-2/MoE and attention layers | |
| | Technical Limitations | Preview checkpoint; unaligned base model — raw completions may be unhelpful, repetitive, or unsafe without post-training; non-primary languages (FR/IT/ES) are limited; may produce inaccurate or outdated content (knowledge cutoff end of 2025) | |
| | Verified to have met prescribed quality standards | TODO | |
| | Performance Metrics | TODO (see Evaluation Dataset) | |
| | Potential Known Risks and Mitigation | May generate incorrect, biased, or unsafe content; as an unaligned base model, apply post-training, use-case-specific testing, and guardrails before deployment | |
| | Terms of Use/Licensing | Other (see License/Terms of Use) | |
|
|
| ### Privacy Subcard |
|
|
| | Field | Response | |
| |:---|:---| |
| | Generatable or reverse engineerable personal data? | TODO | |
| | Personal data used to create this model? | TODO | |
| | Was consent obtained for any personal data used? | TODO | |
| | How often is dataset reviewed? | TODO | |
| | Was data from user interactions with the AI model used to train the model? | No | |
| | Is there provenance for all datasets used in training? | TODO | |
| | Applicable Privacy Policy | TODO | |
|
|
| ### Safety & Security Subcard |
|
|
| | Field | Response | |
| |:---|:---| |
| | Model Application Field(s) | Foundation model for industrial use; basis for fine-tuned assistant and agent applications | |
| | Describe the life critical impact (if present) | None intended. Not for use in life-critical or safety-critical decision-making without independent validation | |
| | Use Case Restrictions | Abide by the applicable license agreement (see License/Terms of Use). Not intended for direct end-user deployment without further post-training and safety alignment | |
| | Model and dataset restrictions | TODO | |
|
|
| ## Related models |
|
|
| - Instruction-tuned variant: [Soofi-Project/Soofi-S-Instruct-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview) |
| - Reasoning variants: [Soofi-Project/Soofi-S-Isar-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Isar-Preview) and [Soofi-Project/Soofi-S-Rhine-Preview](https://huggingface.co/Soofi-Project/Soofi-S-Rhine-Preview) |
| - Quantized derivatives of the Instruct variant: [Soofi-Project/Soofi-S-Instruct-Preview-GGUF](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview-GGUF), [Soofi-Project/Soofi-S-Instruct-Preview-FP8](https://huggingface.co/Soofi-Project/Soofi-S-Instruct-Preview-FP8) |
| - TODO: quantized derivatives of this base model, if released. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{soofi_s_30b_a3b, |
| title = {Soofi-S-30B-A3B}, |
| author = {SOOFI Consortium}, |
| year = {2026}, |
| url = {https://huggingface.co/Soofi-Project/Soofi-S-30B-A3B} |
| } |
| ``` |
|
|
