---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
pipeline_tag: text-generation
library_name: transformers
---


# 🩵 Rain-100M — Model Card

**Rain-100M** is an experimental language model trained from scratch based on the **Qwen3 architecture**.

## 🧠 Training & Data

* **Training corpus**: `HuggingFaceFW/fineweb-edu`
* **Total tokens**: ~**3B**
* **Language**: English only
* **Tokenizer**: Newly trained **16k BPE** (optimized for small/compact models)
* **Max sequence length**: 4096

**Sample training metrics**:

```text
train/grad_norm: 0.6640625
train/learning_rate: 0.00000000002171853813
train/loss: 3.4459
```

## 🏗️ Architecture (Qwen3-style)

* **Parameters**: ~100M
* **Layers**: 12 Transformer layers
* **Hidden size**: 768
* **Attention heads**: 12
* **MLP dimension**: 2048
* **Activation**: SiLU
* **Weight dtype**: bfloat16
* **RMSNorm eps**: 1e-6
* **RoPE θ**: 10000
* **Inference framework**: `transformers`

## ⚠️ Limitations

* Trained only on English data; weak or no capabilities in other languages, not suitable as a general-purpose chat model or for safety-critical use cases.
* No system-level alignment or safety fine-tuning has been applied.

## 📄 License

When using this model locally, please also comply with the licenses of the `fineweb-edu` dataset and the `transformers` / Qwen3-related components.

---

# 中文版本

# 🩵 Rain-100M — Model Card

**Rain-100M** 是一个基于 **Qwen3 架构** 从零训练的实验语言模型。

## 🧠 训练与数据

* **训练语料**：HuggingFaceFW/fineweb-edu
* **Tokens 数量**：约 **3B**
* **语言**：仅英语
* **Tokenizer**：全新训练的 **16k BPE**（面向轻量模型优化）
* **最大序列长度**：4096

**训练参数**：

```

train/grad_norm:0.6640625
train/learning_rate:0.00000000002171853813
train/loss:3.4459

```

## 🏗️ 模型结构（Qwen3 规格）

* **参数量**：约 100M
* **层数**：12 层 Transformer
* **隐藏维度**：768
* **注意力头数**：12
* **中间层维度**：2048
* **激活函数**：SiLU
* **权重类型**：bfloat16
* **RMSNorm eps**：1e-6
* **RoPE θ**：10000
* **推理框架**：transformers

## ⚠️ 限制

* 仅使用英文语料，小语言能力有限，不适合作为通用聊天或安全敏感任务；
* 无系统对齐与安全强化。

## 📄 License

请在本地使用时遵循 fineweb-edu 数据集与 transformers/Qwen3 相关许可证。