AppleMind-1.0-Mini

AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum.

The model has 1,020,480 parameters, a 256-token context window, and a 50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens).

Model Details

Field	Value
Parameters	1,020,480
Architecture	AppleMind 1.0 Mini decoder-only Transformer
Layers	4
Hidden size	128
Intermediate size	512
Attention heads	4
KV heads	4
Head dim	32
Attention style	Multi-head causal self-attention
MLP	GELU
Position embeddings	Learned positional embeddings
Normalization	LayerNorm
Vocabulary size	50,263
Context length	256
Embeddings	Tied input/output embeddings
Tokenizer	Digit-aware byte-level BPE (GPT-2 + special tokens)
Weight format	safetensors
HF architecture	GPT2LMHeadModel
HF model type	gpt2
Final training steps	2,288
Tokens seen	299,892,736
Tokens/parameter	293.87:1
Training data	FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus
Training mixture	100M + 100M + 100M tokens
Precision	BF16
Final model	AppleMind 1.0 Mini

Credits to BananaMind for inspiring me to make AppleMind.

Tokenizer

AppleMind 1.0 Mini uses a 50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens.

Special token	ID
<&#124;pad&#124;>	50,260
<&#124;bos&#124;>	50,261
<&#124;eos&#124;>	50,262

Training Data

Dataset	Target Tokens	Share
FineWeb-Edu	100M	33.33%
FineWeb-HQ	100M	33.33%
SmolLM-Corpus	100M	33.33%
Total	300M	100%

The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset.

Training Setup

Field	Value
Sequence length	256
Micro batch	512 sequences
Gradient accumulation	1
Effective batch	512 sequences
Tokens per optimizer step	131,072
Final optimizer step	2,288
Configured optimizer steps	2,289
Optimizer	AdamW
Peak learning rate	0.0001
Learning rate at final step	3.114e-06
LR schedule	Cosine decay
Gradient clipping	1
Weight format	safetensors
Training tokens	299,892,736
Target tokens	300,000,000
Tokens/parameter	293.87:1

Evaluation

AppleMind 1.0 Mini has not been formally evaluated with lm_eval yet. No benchmark scores are currently reported.

Benchmark	Score	Metric
Average	N/A	mean
ARC Easy	N/A	acc_norm,none
PIQA	N/A	acc_norm,none
ARC Challenge	N/A	acc_norm,none
HellaSwag	N/A	acc_norm,none

The model's current generation quality has been checked with basic text-generation prompts, including:

• Once upon a time
• The little boy
• In the forest

Prompt: Once upon a time

Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked its G during work used after several There and at there b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many

Prompt: The little boy

The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many

Prompt: In the forest

In the forest often form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but

Usage

AppleMind 1.0 Mini uses custom architecture code, so load it with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AppleMind-AI/AppleMind-1.0-Mini"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"

dtype = (
    torch.bfloat16
    if torch.cuda.is_available() and torch.cuda.is_bf16_supported()
    else torch.float32
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device).eval()

prompt = "The color of the sky is"

inputs = tokenizer(
    prompt,
    return_tensors="pt",
).to(device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        repetition_penalty=1.1,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(
    tokenizer.decode(
        output[0],
        skip_special_tokens=True,
    )
)

Note: AppleMind 1.0 Mini has a 256-token context window, so the prompt plus generated tokens should stay within that limit.

License

Apache 2.0