README.md

---
license: apache-2.0
language:
- en
- fr
tags:
- text-generation
- multilingual
- foundation-model
- sparse-transformer
- amforge
- sparsemind
library_name: pytorch
pipeline_tag: text-generation
inference: false
---

# Cofos General 600M — Bilingual Foundation Model

**Cofos General 600M** is a 640M-parameter foundation language model trained from scratch on curated French and English web-scale text. It is the base model in the **Cofos General** family by **AMEFORGE**, built on the proprietary **SparseMind** architecture and designed to serve as a substrate for downstream specialization through fine-tuning.

This model is **not** intended as a standalone assistant. Its purpose is to provide a clean, bilingual, controllable foundation that downstream models (code assistants, personalized assistants, domain-specific tools) can build upon.

---

## Model Summary

| Field | Value |
|---|---|
| **Developer** | AMEFORGE |
| **Architecture** | SparseMind v15 (proprietary) |
| **Parameters** | 640M |
| **Context length** | 2048 tokens |
| **Vocabulary** | 32,000 (custom NexusBPE, multilingual) |
| **Languages** | French (50%), English (50%) |
| **Training data** | Public web-scale text (educational subsets) |
| **Model type** | Causal language model (base, no instruction tuning) |
| **License** | Apache 2.0 |
| **Status** | Active training |

---

## Intended Use

### Primary use cases

- **Foundation for fine-tuning** into specialized downstream models (code assistants, personalized assistants, domain experts)
- **Bilingual text-completion** in French and English where a small, controllable base is required
- **Research** on small bilingual foundation models, sparse architectures, and balanced cross-lingual representations

### Out-of-scope

This model is **not designed for**:
- Direct deployment as a user-facing assistant (it has no instruction tuning and no RLHF)
- Languages other than French and English
- Tasks requiring extensive factual knowledge or current information (training data has a cutoff and limited coverage)
- Safety-critical applications without additional alignment and filtering layers
- Long-context reasoning beyond 2048 tokens

If you want an instruction-following code assistant, look at downstream models in the Cofos family (forthcoming `cofos_general_code_600m`, `cofos_logo_600m`).

---

## Why a small bilingual foundation model?

The model landscape is dominated by either very large general-purpose models or specialized models built on English-only foundations. Cofos General 600M occupies a deliberate niche:

1. **Balanced bilingual representation**: Trained 50/50 on French and English educational web text, providing native-quality coverage in both languages rather than the typical 95%+ English bias of comparable open models.
2. **Small enough for on-device fine-tuning**: At 640M parameters, fine-tuning is tractable on a single high-end consumer GPU, making downstream specialization accessible.
3. **Curated training data**: Trained on educational subsets of public web crawls rather than raw uncurated web text, reducing noise and improving the foundation's quality-per-token ratio.
4. **Controllable substrate**: As the training data is documented and reproducible, downstream users know what their fine-tuned models inherited from the base.

---

## Performance

This is a **base model under active training**. Performance characteristics are reported as training progresses. Refer to the latest model card revision on the HuggingFace repository for current metrics.

The model is evaluated primarily on:
- Cross-entropy loss on held-out French and English validation sets
- Downstream task performance after fine-tuning (which is the intended use)

Direct zero-shot benchmark performance is not the design target. **A base model that is uninteresting standalone but excellent under fine-tuning is, by design, doing its job.**

---

## Usage

### Loading

```python
from huggingface_hub import hf_hub_download
import torch

checkpoint_path = hf_hub_download(repo_id="AMFORGE/cofos_general_600m", filename="cofos_model.pt")
tokenizer_path  = hf_hub_download(repo_id="AMFORGE/cofos_general_600m", filename="cofos_tokenizer.model")
```

Loading and inference require the AMEFORGE SparseMind runtime. The model architecture is proprietary; contact AMEFORGE for access to the runtime, or wait for the public inference utilities released with downstream models.

### Recommended workflow

The recommended usage is **not** direct generation but **fine-tuning** for a specific task. Typical pipeline:

1. Download this base model
2. Prepare a task-specific dataset
3. Fine-tune with standard transfer-learning hyperparameters (low learning rate, fresh optimizer, small number of epochs)
4. Deploy the fine-tuned variant

The forthcoming `cofos_general_code_600m` and `cofos_logo_600m` repos illustrate this workflow concretely.

---

## Training

Cofos General 600M is trained from scratch on a curated mix of public, openly-licensed web text:
- **English educational web text** (filtered for educational quality)
- **French web text** (multilingual web corpus, French subset)

Training is conducted on the AMEFORGE SparseMind training pipeline with periodic safety checkpointing to HuggingFace to ensure recoverability. Mixed sampling preserves a strict 50/50 ratio between French and English throughout training.

**Tokenizer**: [AMFORGE/cofos_general_tok](https://huggingface.co/AMFORGE/cofos_general_tok) — a 32,000-token custom SentencePiece model with multilingual byte fallback for full Unicode coverage and structural tokens reserved for downstream task formatting.

---

## Lineage

```
cofos_general_tok (tokenizer)
       ↓
cofos_general_600m (this model) — bilingual foundation
       ↓
cofos_general_code_600m (forthcoming) — instruction-tuned for code
       ↓
cofos_logo_600m (forthcoming) — personalized variant
```

Cofos General 600M is a **from-scratch base model**. It is not derived from any other published model.

---

## Limitations & Biases

- **No instruction tuning**: This is a raw base model. It will not naturally follow instructions, refuse harmful requests, or behave like an assistant. It is a text-completion engine.
- **Limited training data**: At ~3-6 billion tokens of training (compared to trillion-scale corpora for SOTA models), Cofos General 600M's knowledge breadth is much smaller than models like SmolLM2 or Qwen-0.5B. It is **not competitive on broad knowledge benchmarks** — this is by design, as breadth is sacrificed for tractable specialization.
- **Bias inheritance**: The model will reflect biases present in the FineWeb-Edu (English) and FineWeb-2 (French) training corpora. These are large public web corpora with all the typical biases of such sources.
- **No safety alignment**: Cofos General 600M has no RLHF, no refusal training, and no harm-prevention filtering. It should never be deployed in user-facing products without a downstream safety layer.
- **Capacity limits**: 640M parameters is small by modern standards. Complex multi-step reasoning and long-context coherence will be inferior to larger models. The intended remediation is task-specific fine-tuning, not direct use.

---

## Environmental Considerations

Cofos General 600M is intentionally small to minimize the compute footprint of training and to make downstream fine-tuning accessible to individual researchers and small teams. The model can be fine-tuned and deployed on a single consumer GPU.

---

## License

This model is released under the **Apache 2.0** license. You are free to use, modify, and redistribute it, including for commercial purposes, subject to the terms of the license.

Note: training data was sourced from publicly available datasets (FineWeb-Edu, FineWeb-2). Users redistributing this model or derivatives should ensure compliance with the original source licenses.

---

## Citation

If you use Cofos General 600M in your work, please cite:

```bibtex
@misc{cofos_general_600m_2026,
  title  = {Cofos General 600M: A Bilingual Foundation Model for Downstream Specialization},
  author = {{AMEFORGE}},
  year   = {2026},
  url    = {https://huggingface.co/AMFORGE/cofos_general_600m}
}
```

---

## Contact

For questions, collaborations, or access to the AMEFORGE SparseMind runtime:

- **Organization**: AMEFORGE
- **HuggingFace**: [@AMFORGE](https://huggingface.co/AMFORGE)

---

*Cofos General 600M is the foundation layer of the Cofos model family by AMEFORGE. See the [AMFORGE organization page](https://huggingface.co/AMFORGE) for downstream specialized models built on this base.*