--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation datasets: - HuggingFaceFW/fineweb-edu - epfml/FineWeb-HQ - HuggingFaceTB/smollm-corpus tags: - lm - language-model - causal-lm - causal-language-model - decoder-only - base-model - pretraining - small-language-model - applemind - applemind10 - applemind10-mini - fineweb-edu - fineweb-hq - smollm-corpus - cosmopedia-v2 - pytorch - safetensors - custom-code - custom-architecture - trust-remote-code --- # AppleMind-1.0-Mini ![Banner](banner.webp) AppleMind-1.0-Mini is a compact decoder-only causal language model trained from scratch by AppleMind on a 300M-token curriculum. The model has **1,020,480 parameters**, a **256-token context window**, and a **50,263-token digit-aware byte-level BPE tokenizer (GPT-2 + special tokens)**. ## Model Details | Field | Value | | -------------------- | ------------------------------------------------------: | | Parameters | **1,020,480** | | Architecture | **AppleMind 1.0 Mini decoder-only Transformer** | | Layers | **4** | | Hidden size | **128** | | Intermediate size | **512** | | Attention heads | **4** | | KV heads | **4** | | Head dim | **32** | | Attention style | **Multi-head causal self-attention** | | MLP | **GELU** | | Position embeddings | **Learned positional embeddings** | | Normalization | **LayerNorm** | | Vocabulary size | **50,263** | | Context length | **256** | | Embeddings | **Tied input/output embeddings** | | Tokenizer | **Digit-aware byte-level BPE (GPT-2 + special tokens)** | | Weight format | **safetensors** | | HF architecture | **`GPT2LMHeadModel`** | | HF model type | **`gpt2`** | | Final training steps | **2,288** | | Tokens seen | **299,892,736** | | Tokens/parameter | **293.87:1** | | Training data | **FineWeb-Edu + FineWeb-HQ + SmolLM-Corpus** | | Training mixture | **100M + 100M + 100M tokens** | | Precision | **BF16** | | Final model | **AppleMind 1.0 Mini** | ### Credits to BananaMind for inspiring me to make AppleMind. ## Tokenizer AppleMind 1.0 Mini uses a **50,263-token digit-aware byte-level BPE tokenizer based on the GPT-2 tokenizer**, with 3 additional special tokens. Digits are handled individually rather than being collapsed into large number tokens. | Special token | ID | | ------------- | --: | | `<|pad|>` | **50,260** | | `<|bos|>` | **50,261** | | `<|eos|>` | **50,262** | ## Training Data | Dataset | Target Tokens | Share | | ------------- | ------------: | ----: | | FineWeb-Edu | 100M | 33.33% | | FineWeb-HQ | 100M | 33.33% | | SmolLM-Corpus | 100M | 33.33% | | **Total** | **300M** | **100%** | The training run used an equal mixture of FineWeb-Edu, FineWeb-HQ, and SmolLM-Corpus, with 100M tokens sampled from each dataset. ## Training Setup | Field | Value | | --------------------------- | ------------: | | Sequence length | 256 | | Micro batch | 512 sequences | | Gradient accumulation | 1 | | Effective batch | 512 sequences | | Tokens per optimizer step | 131,072 | | Final optimizer step | 2,288 | | Configured optimizer steps | 2,289 | | Optimizer | AdamW | | Peak learning rate | 0.0001 | | Learning rate at final step | 3.114e-06 | | LR schedule | Cosine decay | | Gradient clipping | 1 | | Weight format | safetensors | | Training tokens | 299,892,736 | | Target tokens | 300,000,000 | | Tokens/parameter | 293.87:1 | ## Evaluation AppleMind 1.0 Mini has **not been formally evaluated with `lm_eval` yet**. No benchmark scores are currently reported. | Benchmark | Score | Metric | | ------------- | ------: | --------------- | | **Average** | **N/A** | `mean` | | ARC Easy | N/A | `acc_norm,none` | | PIQA | N/A | `acc_norm,none` | | ARC Challenge | N/A | `acc_norm,none` | | HellaSwag | N/A | `acc_norm,none` | The model's current generation quality has been checked with basic text-generation prompts, including: * `Once upon a time` * `The little boy` * `In the forest` Prompt: Once upon a time ------------------------------------------------------------ Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked its G during work used after several There and at there b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many ------------------------------------------------------------ Prompt: The little boy ------------------------------------------------------------ The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many ------------------------------------------------------------ Prompt: In the forest ------------------------------------------------------------ In the forest often form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but ------------------------------------------------------------ ## Usage AppleMind 1.0 Mini uses custom architecture code, so load it with `trust_remote_code=True`. ```bash pip install -U transformers safetensors torch ``` ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "AppleMind-AI/AppleMind-1.0-Mini" tokenizer = AutoTokenizer.from_pretrained( model_id, trust_remote_code=True, ) device = "cuda" if torch.cuda.is_available() else "cpu" dtype = ( torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float32 ) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, torch_dtype=dtype, ).to(device).eval() prompt = "The color of the sky is" inputs = tokenizer( prompt, return_tensors="pt", ).to(device) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=96, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1, pad_token_id=tokenizer.eos_token_id, eos_token_id=tokenizer.eos_token_id, ) print( tokenizer.decode( output[0], skip_special_tokens=True, ) ) ``` **Note:** AppleMind 1.0 Mini has a **256-token context window**, so the prompt plus generated tokens should stay within that limit. ## License Apache 2.0