README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-edu
- epfml/FineWeb-HQ
- HuggingFaceTB/smollm-corpus
tags:
- lm
- language-model
- causal-lm
- causal-language-model
- decoder-only
- base-model
- pretraining
- small-language-model
- applemind
- applemind10
- applemind10-mini
- fineweb-edu
- fineweb-hq
- smollm-corpus
- cosmopedia-v2
- pytorch
- safetensors
- custom-code
- custom-architecture
- trust-remote-code
---

# AppleMind-1.0-Mini
![Banner](banner.webp)

AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.

The model has **1,020,480 parameters**, a **256-token context window**, and a **50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens)**.

## Model Details

| Field                |                                                   Value |
| -------------------- | ------------------------------------------------------: |
| Parameters           |                                           **1,020,480** |
| Architecture         |         **AppleMind 1.0 Mini decoder-only Transformer** |
| Layers               |                                                   **4** |
| Hidden size          |                                                 **128** |
| Intermediate size    |                                                 **512** |
| Attention heads      |                                                   **4** |
| KV heads             |                                                   **4** |
| Head dim             |                                                  **32** |
| Attention style      |                    **Multi-head causal self-attention** |
| MLP                  |                                                **GELU** |
| Position embeddings  |                       **Learned positional embeddings** |
| Normalization        |                                           **LayerNorm** |
| Vocabulary size      |                                              **50,263** |
| Context length       |                                                 **256** |
| Embeddings           |                        **Tied input/output embeddings** |
| Tokenizer            | **Digit-aware byte-level BPE (GPT-2 + special tokens)** |
| Weight format        |                                         **safetensors** |
| HF architecture      |                                   **`GPT2LMHeadModel`** |
| HF model type        |                                              **`gpt2`** |
| Final training steps |                                               **2,288** |
| Tokens seen          |                                         **299,892,736** |
| Tokens/parameter     |                                            **293.87:1** |
| Training data        |            **FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus** |
| Training mixture     |                           **100M + 100M + 100M tokens** |
| Precision            |                                                **BF16** |
| Final model          |                                  **AppleMind 1.0 Mini** |

### Credits to BananaMind for inspiring me to make AppleMind.

## Tokenizer

AppleMind 1.0 Mini uses a **50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer**, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.

| Special token | ID |
| ------------- | --: |
| `<&#124;pad&#124;>` | **50,260** |
| `<&#124;bos&#124;>` | **50,261** |
| `<&#124;eos&#124;>` | **50,262** |

## Training Data

| Dataset       | Target Tokens | Share |
| ------------- | ------------: | ----: |
| FineWeb-Edu   |          100M | 33.33% |
| FineWeb-HQ    |          100M | 33.33% |
| SmolLM-Corpus |          100M | 33.33% |
| **Total**     |      **300M** | **100%** |

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

## Training Setup

| Field                       |         Value |
| --------------------------- | ------------: |
| Sequence length             |           256 |
| Micro batch                 | 512 sequences |
| Gradient accumulation       |             1 |
| Effective batch             | 512 sequences |
| Tokens per optimizer step   |       131,072 |
| Final optimizer step        |         2,288 |
| Configured optimizer steps  |         2,289 |
| Optimizer                   |         AdamW |
| Peak learning rate          |        0.0001 |
| Learning rate at final step |     3.114e-06 |
| LR schedule                 |  Cosine decay |
| Gradient clipping           |             1 |
| Weight format               |   safetensors |
| Training tokens             |   299,892,736 |
| Target tokens               |   300,000,000 |
| Tokens/parameter            |      293.87:1 |

## Evaluation

AppleMind 1.0 Mini has **not been formally evaluated with `lm_eval` yet**. No benchmark scores are currently reported.

| Benchmark     |   Score | Metric          |
| ------------- | ------: | --------------- |
| **Average**   | **N/A** | `mean`          |
| ARC Easy      |     N/A | `acc_norm,none` |
| PIQA          |     N/A | `acc_norm,none` |
| ARC Challenge |     N/A | `acc_norm,none` |
| HellaSwag     |     N/A | `acc_norm,none` |

The model's current generation quality has been checked with basic text-generation prompts, including:

* `Once upon a time`
* `The little boy`
* `In the forest`

Prompt: Once upon a time
------------------------------------------------------------
Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked
 its G during work used after several There and at there
b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many
------------------------------------------------------------

Prompt: The little boy
------------------------------------------------------------
The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many
------------------------------------------------------------

Prompt: In the forest
------------------------------------------------------------
In the forest often
 form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used
 to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but
------------------------------------------------------------

## Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with `trust_remote_code=True`.

```bash
pip install -U transformers safetensors torch
```

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)
```

**Note:** AppleMind 1.0 Mini has a **256-token context window**, so the prompt plus generated tokens should stay within that limit.

## License

Apache 2.0