Text Generation
Transformers
Safetensors
PyTorch
English
gpt2
lm
language-model
causal-lm
causal-language-model
decoder-only
base-model
pretraining
small-language-model
applemind
applemind10
applemind10-mini
fineweb-edu
fineweb-hq
smollm-corpus
cosmopedia-v2
custom-code
custom-architecture
trust-remote-code
conversational
text-generation-inference
File size: 8,402 Bytes
2a21980 7f8f669 2a21980 edd1e6e 2a21980 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 | ---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- epfml/FineWeb-HQ
- HuggingFaceTB/smollm-corpus
tags:
- lm
- language-model
- causal-lm
- causal-language-model
- decoder-only
- base-model
- pretraining
- small-language-model
- applemind
- applemind10
- applemind10-mini
- fineweb-edu
- fineweb-hq
- smollm-corpus
- cosmopedia-v2
- pytorch
- safetensors
- custom-code
- custom-architecture
- trust-remote-code
---
# AppleMind-1.0-Mini
AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.
The model has **1,020,480 parameters**, a **256-token context window**, and a **50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens)**.
## Model Details
| Field | Value |
| -------------------- | ------------------------------------------------------: |
| Parameters | **1,020,480** |
| Architecture | **AppleMind 1.0 Mini decoder-only Transformer** |
| Layers | **4** |
| Hidden size | **128** |
| Intermediate size | **512** |
| Attention heads | **4** |
| KV heads | **4** |
| Head dim | **32** |
| Attention style | **Multi-head causal self-attention** |
| MLP | **GELU** |
| Position embeddings | **Learned positional embeddings** |
| Normalization | **LayerNorm** |
| Vocabulary size | **50,263** |
| Context length | **256** |
| Embeddings | **Tied input/output embeddings** |
| Tokenizer | **Digit-aware byte-level BPE (GPT-2 + special tokens)** |
| Weight format | **safetensors** |
| HF architecture | **`GPT2LMHeadModel`** |
| HF model type | **`gpt2`** |
| Final training steps | **2,288** |
| Tokens seen | **299,892,736** |
| Tokens/parameter | **293.87:1** |
| Training data | **FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus** |
| Training mixture | **100M + 100M + 100M tokens** |
| Precision | **BF16** |
| Final model | **AppleMind 1.0 Mini** |
### Credits to BananaMind for inspiring me to make AppleMind.
## Tokenizer
AppleMind 1.0 Mini uses a **50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer**, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.
| Special token | ID |
| ------------- | --: |
| `<|pad|>` | **50,260** |
| `<|bos|>` | **50,261** |
| `<|eos|>` | **50,262** |
## Training Data
| Dataset | Target Tokens | Share |
| ------------- | ------------: | ----: |
| FineWeb-Edu | 100M | 33.33% |
| FineWeb-HQ | 100M | 33.33% |
| SmolLM-Corpus | 100M | 33.33% |
| **Total** | **300M** | **100%** |
The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.
## Training Setup
| Field | Value |
| --------------------------- | ------------: |
| Sequence length | 256 |
| Micro batch | 512 sequences |
| Gradient accumulation | 1 |
| Effective batch | 512 sequences |
| Tokens per optimizer step | 131,072 |
| Final optimizer step | 2,288 |
| Configured optimizer steps | 2,289 |
| Optimizer | AdamW |
| Peak learning rate | 0.0001 |
| Learning rate at final step | 3.114e-06 |
| LR schedule | Cosine decay |
| Gradient clipping | 1 |
| Weight format | safetensors |
| Training tokens | 299,892,736 |
| Target tokens | 300,000,000 |
| Tokens/parameter | 293.87:1 |
## Evaluation
AppleMind 1.0 Mini has **not been formally evaluated with `lm_eval` yet**. No benchmark scores are currently reported.
| Benchmark | Score | Metric |
| ------------- | ------: | --------------- |
| **Average** | **N/A** | `mean` |
| ARC Easy | N/A | `acc_norm,none` |
| PIQA | N/A | `acc_norm,none` |
| ARC Challenge | N/A | `acc_norm,none` |
| HellaSwag | N/A | `acc_norm,none` |
The model's current generation quality has been checked with basic text-generation prompts, including:
* `Once upon a time`
* `The little boy`
* `In the forest`
Prompt: Once upon a time
------------------------------------------------------------
Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked
its G during work used after several There and at there
b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many
------------------------------------------------------------
Prompt: The little boy
------------------------------------------------------------
The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many
------------------------------------------------------------
Prompt: In the forest
------------------------------------------------------------
In the forest often
form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used
to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but
------------------------------------------------------------
## Usage
AppleMind 1.0 Mini uses custom architecture code, so load it with `trust_remote_code=True`.
```bash
pip install -U transformers safetensors torch
```
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "AppleMind-AI/AppleMind-1.0-Mini"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
torch.bfloat16
if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
else torch.float32
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=dtype,
).to(device).eval()
prompt = "The color of the sky is"
inputs = tokenizer(
prompt,
return_tensors="pt",
).to(device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=96,
do_sample=True,
temperature=0.7,
top_p=0.9,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(
tokenizer.decode(
output[0],
skip_special_tokens=True,
)
)
```
**Note:** AppleMind 1.0 Mini has a **256-token context window**, so the prompt plus generated tokens should stay within that limit.
## License
Apache 2.0
|