LOGS.md

 Requirement already satisfied: transformers in /usr/local/lib/python3.12/dist-packages (5.13.1)
Collecting transformers
  Downloading transformers-5.15.0-py3-none-any.whl.metadata (32 kB)
Requirement already satisfied: datasets in /usr/local/lib/python3.12/dist-packages (4.0.0)
Collecting datasets
  Downloading datasets-5.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: accelerate in /usr/local/lib/python3.12/dist-packages (1.14.0)
Requirement already satisfied: huggingface_hub in /usr/local/lib/python3.12/dist-packages (1.23.0)
Collecting huggingface_hub
  Downloading huggingface_hub-1.27.0-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: hf_xet in /usr/local/lib/python3.12/dist-packages (1.5.1)
Collecting hf_xet
  Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.12/dist-packages (from transformers) (2.0.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (26.2)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.12/dist-packages (from transformers) (6.0.3)
Requirement already satisfied: regex>=2025.10.22 in /usr/local/lib/python3.12/dist-packages (from transformers) (2025.11.3)
Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.22.2)
Requirement already satisfied: typer in /usr/local/lib/python3.12/dist-packages (from transformers) (0.26.8)
Requirement already satisfied: safetensors>=0.8.0 in /usr/local/lib/python3.12/dist-packages (from transformers) (0.8.0)
Requirement already satisfied: tqdm>=4.60 in /usr/local/lib/python3.12/dist-packages (from transformers) (4.67.3)
Requirement already satisfied: filelock in /usr/local/lib/python3.12/dist-packages (from datasets) (3.29.7)
Collecting pyarrow>=21.0.0 (from datasets)
  Downloading pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Requirement already satisfied: dill<0.4.2,>=0.3.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.3.8)
Requirement already satisfied: pandas in /usr/local/lib/python3.12/dist-packages (from datasets) (2.2.2)
Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.12/dist-packages (from datasets) (2.32.4)
Requirement already satisfied: httpx<1.0.0 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.28.1)
Requirement already satisfied: xxhash in /usr/local/lib/python3.12/dist-packages (from datasets) (3.8.1)
Requirement already satisfied: multiprocess<0.70.20 in /usr/local/lib/python3.12/dist-packages (from datasets) (0.70.16)
Requirement already satisfied: fsspec<=2026.6.0,>=2023.1.0 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (2025.3.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.12/dist-packages (from accelerate) (5.9.5)
Requirement already satisfied: torch>=2.0.0 in /usr/local/lib/python3.12/dist-packages (from accelerate) (2.11.0+cu128)
Requirement already satisfied: click<9.0.0,>=8.4.2 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (8.4.2)
Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.12/dist-packages (from huggingface_hub) (4.16.0)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.12/dist-packages (from fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (3.14.1)
Requirement already satisfied: anyio in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (4.14.2)
Requirement already satisfied: certifi in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (2026.6.17)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (1.0.9)
Requirement already satisfied: idna in /usr/local/lib/python3.12/dist-packages (from httpx<1.0.0->datasets) (3.18)
Requirement already satisfied: h11>=0.16 in /usr/local/lib/python3.12/dist-packages (from httpcore==1.*->httpx<1.0.0->datasets) (0.16.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (3.4.9)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.12/dist-packages (from requests>=2.32.2->datasets) (2.5.0)
Requirement already satisfied: setuptools<82 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (75.2.0)
Requirement already satisfied: sympy>=1.13.3 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.1.6)
Requirement already satisfied: cuda-toolkit==12.8.1 in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.1)
Requirement already satisfied: cuda-bindings<13,>=12.9.4 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (12.9.7)
Requirement already satisfied: nvidia-cudnn-cu12==9.19.0.56 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (9.19.0.56)
Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (0.7.1)
Requirement already satisfied: nvidia-nccl-cu12==2.28.9 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (2.28.9)
Requirement already satisfied: nvidia-nvshmem-cu12==3.4.5 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.4.5)
Requirement already satisfied: triton==3.6.0 in /usr/local/lib/python3.12/dist-packages (from torch>=2.0.0->accelerate) (3.6.0)
Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.4.1)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (11.3.3.83)
Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (1.13.1.3)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
Requirement already satisfied: nvidia-curand-cu12==10.3.9.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (10.3.9.90)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (11.7.3.90)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.5.8.93)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.93)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.93)
Requirement already satisfied: nvidia-nvtx-cu12==12.8.90.* in /usr/local/lib/python3.12/dist-packages (from cuda-toolkit[cublas,cudart,cufft,cufile,cupti,curand,cusolver,cusparse,nvjitlink,nvrtc,nvtx]==12.8.1; platform_system == "Linux"->torch>=2.0.0->accelerate) (12.8.90)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.12/dist-packages (from pandas->datasets) (2026.3)
Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (1.5.4)
Requirement already satisfied: rich>=13.8.0 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (13.9.4)
Requirement already satisfied: annotated-doc>=0.0.2 in /usr/local/lib/python3.12/dist-packages (from typer->transformers) (0.0.4)
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (2.7.1)
Requirement already satisfied: aiosignal>=1.4.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.4.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (26.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.8.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (6.7.1)
Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (0.5.2)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.12/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.6.0,>=2023.1.0->datasets) (1.24.2)
Requirement already satisfied: cuda-pathfinder~=1.1 in /usr/local/lib/python3.12/dist-packages (from cuda-bindings<13,>=12.9.4->torch>=2.0.0->accelerate) (1.5.6)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.12/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers) (4.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.12/dist-packages (from rich>=13.8.0->typer->transformers) (2.20.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.12/dist-packages (from sympy>=1.13.3->torch>=2.0.0->accelerate) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.12/dist-packages (from jinja2->torch>=2.0.0->accelerate) (3.0.3)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.12/dist-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers) (0.1.2)
Downloading transformers-5.15.0-py3-none-any.whl (11.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.7/11.7 MB 133.0 MB/s eta 0:00:00
Downloading datasets-5.0.1-py3-none-any.whl (559 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 559.1/559.1 kB 48.0 MB/s eta 0:00:00
Downloading huggingface_hub-1.27.0-py3-none-any.whl (784 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 784.9/784.9 kB 61.8 MB/s eta 0:00:00
Downloading hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 132.9 MB/s eta 0:00:00
Downloading pyarrow-25.0.1-cp312-cp312-manylinux_2_28_x86_64.whl (50.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.1/50.1 MB 49.6 MB/s eta 0:00:00
Installing collected packages: pyarrow, hf_xet, huggingface_hub, datasets, transformers
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 18.1.0
    Uninstalling pyarrow-18.1.0:
      Successfully uninstalled pyarrow-18.1.0
  Attempting uninstall: hf_xet
    Found existing installation: hf-xet 1.5.1
    Uninstalling hf-xet-1.5.1:
      Successfully uninstalled hf-xet-1.5.1
  Attempting uninstall: huggingface_hub
    Found existing installation: huggingface_hub 1.23.0
    Uninstalling huggingface_hub-1.23.0:
      Successfully uninstalled huggingface_hub-1.23.0
  Attempting uninstall: datasets
    Found existing installation: datasets 4.0.0
    Uninstalling datasets-4.0.0:
      Successfully uninstalled datasets-4.0.0
  Attempting uninstall: transformers
    Found existing installation: transformers 5.13.1
    Uninstalling transformers-5.13.1:
      Successfully uninstalled transformers-5.13.1
Successfully installed datasets-5.0.1 hf_xet-1.6.0 huggingface_hub-1.27.0 pyarrow-25.0.1 transformers-5.15.0
To log in, open this URL and enter the code:https://hf.co/oauth/device1JHU-54EFWaiting for authorization...Login successful. Logged in as GGUFGuy (token: oauth-GGUFGuy).This token will be refreshed automatically when it expires.
======================================================================
AppleMind 1.0 Mini
Colab L4 High-RAM
300M TOKEN PRETRAINING
======================================================================

Device: cuda
GPU: NVIDIA L4
VRAM: 22.03 GB
CUDA: 12.8

Total training tokens: 300,000,000
Target ratio: ~300:1

======================================================================
DATASET MIXTURE
======================================================================

FineWeb-Edu sample-10BT : 100,000,000
FineWeb-HQ              : 100,000,000
SmolLM-Corpus           : 100,000,000

TOTAL                   : 300,000,000

TinyStories: REMOVED
WikiText: REMOVED
Local datasets: REMOVED
Dataset cycling: DISABLED

======================================================================
LOADING GPT-2 TOKENIZER
======================================================================
tokenizer_config.json: 100% 26.0/26.0 [00:00<00:00, 3.04kB/s]vocab.json: 100% 1.04M/1.04M [00:00<00:00, 15.7MB/s]merges.txt: 100% 456k/456k [00:00<00:00, 5.83MB/s]tokenizer.json: 100% 1.36M/1.36M [00:00<00:00, 28.9MB/s]
Vocabulary: 50,260
Special tokens added: 3


======================================================================
CREATING APPLEMIND
======================================================================

Parameters: 1,020,480
Trainable: 1,020,480

Target tokens: 300,000,000
Target token/parameter ratio: 293.98:1


======================================================================
SETTING UP HUGGING FACE
======================================================================

Logged in as:
GGUFGuy

Checkpoint repository ready:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints

Final model repository ready:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini
Precision: BF16

======================================================================
TRAINING CONFIGURATION
======================================================================

Parameters: 1,020,480
Training tokens: 300,000,000
Tokens/parameter: 293.98:1

Micro batch size: 64
Gradient accumulation: 8
Effective batch: 512
Context length: 256
Tokens/micro batch: 16,384
Tokens/optimizer step: 131,072
Optimizer steps: 2,289
Warmup steps: 114

FineWeb-Edu: 100M
FineWeb-HQ:  100M
SmolLM:      100M

Checkpoint repository:
AppleMind-AI/AppleMind-1.0-Mini-Checkpoints

Final repository:
AppleMind-AI/AppleMind-1.0-Mini

======================================================================
STARTING 300M-TOKEN PRETRAINING
======================================================================

Opening FineWeb-Edu sample-10BT...
README.md: 100% 26.4k/26.4k [00:00<00:00, 2.88MB/s]Resolving data files: 100% 2410/2410 [00:00<00:00, 23346.46it/s]
======================================================================
STREAMING FineWeb-Edu
======================================================================

Target tokens: 100,000,000
Progress state: 0

[transformers] `loss_type=None` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`.
FineWeb-Edu: 1,000,131 / 100,000,000 tokens
FineWeb-Edu: 2,001,441 / 100,000,000 tokens
FineWeb-Edu: 3,001,677 / 100,000,000 tokens
FineWeb-Edu: 4,003,530 / 100,000,000 tokens
FineWeb-Edu: 5,003,864 / 100,000,000 tokens
FineWeb-Edu: 6,014,497 / 100,000,000 tokens
FineWeb-Edu: 7,014,690 / 100,000,000 tokens
FineWeb-Edu: 8,014,892 / 100,000,000 tokens
FineWeb-Edu: 9,015,759 / 100,000,000 tokens
FineWeb-Edu: 10,016,701 / 100,000,000 tokens
FineWeb-Edu: 11,042,613 / 100,000,000 tokens
FineWeb-Edu: 12,043,456 / 100,000,000 tokens
FineWeb-Edu: 13,044,297 / 100,000,000 tokens

Step 100/2,289 | Loss 10.8201 | LR 2.632e-05 | 100,991 tok/s | 13,107,200 tokens | 4.37% | 0.04h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-100...
Writing model shards: 100% 1/1 [00:00<00:00, 19.95it/s]
======================================================================
UPLOADING CHECKPOINT 100
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-100
FineWeb-Edu: 14,051,468 / 100,000,000 tokens
FineWeb-Edu: 15,057,858 / 100,000,000 tokens
FineWeb-Edu: 16,058,350 / 100,000,000 tokens
FineWeb-Edu: 17,058,910 / 100,000,000 tokens
FineWeb-Edu: 18,059,120 / 100,000,000 tokens
FineWeb-Edu: 19,059,214 / 100,000,000 tokens
FineWeb-Edu: 20,060,034 / 100,000,000 tokens
FineWeb-Edu: 21,061,034 / 100,000,000 tokens
FineWeb-Edu: 22,061,243 / 100,000,000 tokens
FineWeb-Edu: 23,061,857 / 100,000,000 tokens
FineWeb-Edu: 24,068,376 / 100,000,000 tokens
FineWeb-Edu: 25,068,451 / 100,000,000 tokens
FineWeb-Edu: 26,069,133 / 100,000,000 tokens

Step 200/2,289 | Loss 10.7533 | LR 2.990e-05 | 103,793 tok/s | 26,214,400 tokens | 8.74% | 0.07h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-200...
Writing model shards: 100% 1/1 [00:00<00:00, 11.00it/s]
======================================================================
UPLOADING CHECKPOINT 200
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-200
FineWeb-Edu: 27,075,071 / 100,000,000 tokens
FineWeb-Edu: 28,080,714 / 100,000,000 tokens
FineWeb-Edu: 29,080,972 / 100,000,000 tokens
FineWeb-Edu: 30,085,136 / 100,000,000 tokens
FineWeb-Edu: 31,085,689 / 100,000,000 tokens
FineWeb-Edu: 32,086,415 / 100,000,000 tokens
FineWeb-Edu: 33,095,884 / 100,000,000 tokens
FineWeb-Edu: 34,096,837 / 100,000,000 tokens
FineWeb-Edu: 35,097,143 / 100,000,000 tokens
FineWeb-Edu: 36,097,343 / 100,000,000 tokens
FineWeb-Edu: 37,104,064 / 100,000,000 tokens
FineWeb-Edu: 38,104,207 / 100,000,000 tokens
FineWeb-Edu: 39,105,073 / 100,000,000 tokens

Step 300/2,289 | Loss 10.6609 | LR 2.952e-05 | 104,146 tok/s | 39,321,600 tokens | 13.11% | 0.11h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-300...
Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
======================================================================
UPLOADING CHECKPOINT 300
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-300
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-100
FineWeb-Edu: 40,107,907 / 100,000,000 tokens
FineWeb-Edu: 41,108,319 / 100,000,000 tokens
FineWeb-Edu: 42,108,687 / 100,000,000 tokens
FineWeb-Edu: 43,108,744 / 100,000,000 tokens
FineWeb-Edu: 44,109,584 / 100,000,000 tokens
FineWeb-Edu: 45,109,641 / 100,000,000 tokens
FineWeb-Edu: 46,110,333 / 100,000,000 tokens
FineWeb-Edu: 47,110,722 / 100,000,000 tokens
FineWeb-Edu: 48,111,125 / 100,000,000 tokens
FineWeb-Edu: 49,111,530 / 100,000,000 tokens
FineWeb-Edu: 50,113,218 / 100,000,000 tokens
FineWeb-Edu: 51,113,865 / 100,000,000 tokens
FineWeb-Edu: 52,114,128 / 100,000,000 tokens

Step 400/2,289 | Loss 10.5818 | LR 2.887e-05 | 104,108 tok/s | 52,428,800 tokens | 17.48% | 0.14h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-400...
Writing model shards: 100% 1/1 [00:00<00:00, 10.91it/s]
======================================================================
UPLOADING CHECKPOINT 400
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-400
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-200
FineWeb-Edu: 53,114,351 / 100,000,000 tokens
FineWeb-Edu: 54,115,268 / 100,000,000 tokens
FineWeb-Edu: 55,115,867 / 100,000,000 tokens
FineWeb-Edu: 56,118,747 / 100,000,000 tokens
FineWeb-Edu: 57,118,792 / 100,000,000 tokens
FineWeb-Edu: 58,129,696 / 100,000,000 tokens
FineWeb-Edu: 59,129,772 / 100,000,000 tokens
FineWeb-Edu: 60,130,463 / 100,000,000 tokens
FineWeb-Edu: 61,130,846 / 100,000,000 tokens
FineWeb-Edu: 62,131,974 / 100,000,000 tokens
FineWeb-Edu: 63,145,958 / 100,000,000 tokens
FineWeb-Edu: 64,146,051 / 100,000,000 tokens
FineWeb-Edu: 65,146,257 / 100,000,000 tokens

Step 500/2,289 | Loss 10.5081 | LR 2.797e-05 | 103,733 tok/s | 65,536,000 tokens | 21.85% | 0.18h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-500...
Writing model shards: 100% 1/1 [00:00<00:00, 10.95it/s]
======================================================================
UPLOADING CHECKPOINT 500
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-500
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-300
FineWeb-Edu: 66,147,176 / 100,000,000 tokens
FineWeb-Edu: 67,153,903 / 100,000,000 tokens
FineWeb-Edu: 68,163,652 / 100,000,000 tokens
FineWeb-Edu: 69,164,783 / 100,000,000 tokens
FineWeb-Edu: 70,166,355 / 100,000,000 tokens
FineWeb-Edu: 71,166,545 / 100,000,000 tokens
FineWeb-Edu: 72,167,733 / 100,000,000 tokens
FineWeb-Edu: 73,168,881 / 100,000,000 tokens
FineWeb-Edu: 74,175,780 / 100,000,000 tokens
FineWeb-Edu: 75,176,435 / 100,000,000 tokens
FineWeb-Edu: 76,177,413 / 100,000,000 tokens
FineWeb-Edu: 77,177,431 / 100,000,000 tokens
FineWeb-Edu: 78,179,729 / 100,000,000 tokens

Step 600/2,289 | Loss 10.4357 | LR 2.682e-05 | 104,154 tok/s | 78,643,200 tokens | 26.21% | 0.21h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-600...
Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
======================================================================
UPLOADING CHECKPOINT 600
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-600
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-400
FineWeb-Edu: 79,187,841 / 100,000,000 tokens
FineWeb-Edu: 80,187,924 / 100,000,000 tokens
FineWeb-Edu: 81,188,888 / 100,000,000 tokens
FineWeb-Edu: 82,189,255 / 100,000,000 tokens
FineWeb-Edu: 83,189,443 / 100,000,000 tokens
FineWeb-Edu: 84,192,547 / 100,000,000 tokens
FineWeb-Edu: 85,192,626 / 100,000,000 tokens
FineWeb-Edu: 86,193,916 / 100,000,000 tokens
FineWeb-Edu: 87,194,129 / 100,000,000 tokens
FineWeb-Edu: 88,196,386 / 100,000,000 tokens
FineWeb-Edu: 89,203,193 / 100,000,000 tokens
FineWeb-Edu: 90,203,499 / 100,000,000 tokens
FineWeb-Edu: 91,204,016 / 100,000,000 tokens

Step 700/2,289 | Loss 10.3676 | LR 2.546e-05 | 104,272 tok/s | 91,750,400 tokens | 30.58% | 0.25h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-700...
Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
======================================================================
UPLOADING CHECKPOINT 700
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-700
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-500
FineWeb-Edu: 92,205,445 / 100,000,000 tokens
FineWeb-Edu: 93,206,406 / 100,000,000 tokens
FineWeb-Edu: 94,215,721 / 100,000,000 tokens
FineWeb-Edu: 95,216,609 / 100,000,000 tokens
FineWeb-Edu: 96,218,844 / 100,000,000 tokens
FineWeb-Edu: 97,219,420 / 100,000,000 tokens
FineWeb-Edu: 98,219,961 / 100,000,000 tokens
FineWeb-Edu: 99,220,378 / 100,000,000 tokens

FineWeb-Edu complete: 100,000,000 tokens
Opening FineWeb-HQ...
README.md: 100% 4.65k/4.65k [00:00<00:00, 385kB/s]Resolving data files: 100% 9246/9246 [00:00<00:00, 25238.04it/s]
======================================================================
STREAMING FineWeb-HQ
======================================================================

Target tokens: 100,000,000
Progress state: 0

FineWeb-HQ: 1,000,237 / 100,000,000 tokens
FineWeb-HQ: 2,000,353 / 100,000,000 tokens
FineWeb-HQ: 3,000,648 / 100,000,000 tokens
FineWeb-HQ: 4,001,578 / 100,000,000 tokens

Step 800/2,289 | Loss 10.3038 | LR 2.391e-05 | 98,189 tok/s | 104,857,600 tokens | 34.95% | 0.28h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-800...
Writing model shards: 100% 1/1 [00:00<00:00, 10.89it/s]
======================================================================
UPLOADING CHECKPOINT 800
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-800
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-600
FineWeb-HQ: 5,003,165 / 100,000,000 tokens
FineWeb-HQ: 6,003,192 / 100,000,000 tokens
FineWeb-HQ: 7,003,245 / 100,000,000 tokens
FineWeb-HQ: 8,005,095 / 100,000,000 tokens
FineWeb-HQ: 9,005,500 / 100,000,000 tokens
FineWeb-HQ: 10,005,995 / 100,000,000 tokens
FineWeb-HQ: 11,006,275 / 100,000,000 tokens
FineWeb-HQ: 12,006,703 / 100,000,000 tokens
FineWeb-HQ: 13,008,348 / 100,000,000 tokens
FineWeb-HQ: 14,039,910 / 100,000,000 tokens
FineWeb-HQ: 15,040,042 / 100,000,000 tokens
FineWeb-HQ: 16,040,179 / 100,000,000 tokens
FineWeb-HQ: 17,040,530 / 100,000,000 tokens

Step 900/2,289 | Loss 10.2507 | LR 2.221e-05 | 103,602 tok/s | 117,964,800 tokens | 39.32% | 0.32h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-900...
Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
======================================================================
UPLOADING CHECKPOINT 900
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-900
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-700
FineWeb-HQ: 18,045,133 / 100,000,000 tokens
FineWeb-HQ: 19,053,155 / 100,000,000 tokens
FineWeb-HQ: 20,053,155 / 100,000,000 tokens
FineWeb-HQ: 21,053,631 / 100,000,000 tokens
FineWeb-HQ: 22,061,664 / 100,000,000 tokens
FineWeb-HQ: 23,061,940 / 100,000,000 tokens
FineWeb-HQ: 24,062,978 / 100,000,000 tokens
FineWeb-HQ: 25,063,229 / 100,000,000 tokens
FineWeb-HQ: 26,073,081 / 100,000,000 tokens
FineWeb-HQ: 27,073,220 / 100,000,000 tokens
FineWeb-HQ: 28,073,681 / 100,000,000 tokens
FineWeb-HQ: 29,073,966 / 100,000,000 tokens
FineWeb-HQ: 30,074,272 / 100,000,000 tokens

Step 1,000/2,289 | Loss 10.1952 | LR 2.039e-05 | 103,234 tok/s | 131,072,000 tokens | 43.69% | 0.35h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1000...
Writing model shards: 100% 1/1 [00:00<00:00, 10.99it/s]
======================================================================
UPLOADING CHECKPOINT 1000
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1000
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-800
FineWeb-HQ: 31,075,196 / 100,000,000 tokens
FineWeb-HQ: 32,075,679 / 100,000,000 tokens
FineWeb-HQ: 33,077,041 / 100,000,000 tokens
FineWeb-HQ: 34,077,283 / 100,000,000 tokens
FineWeb-HQ: 35,077,442 / 100,000,000 tokens
FineWeb-HQ: 36,078,144 / 100,000,000 tokens
FineWeb-HQ: 37,078,751 / 100,000,000 tokens
FineWeb-HQ: 38,080,143 / 100,000,000 tokens
FineWeb-HQ: 39,082,841 / 100,000,000 tokens
FineWeb-HQ: 40,083,517 / 100,000,000 tokens
FineWeb-HQ: 41,084,347 / 100,000,000 tokens
FineWeb-HQ: 42,087,132 / 100,000,000 tokens
FineWeb-HQ: 43,087,302 / 100,000,000 tokens
FineWeb-HQ: 44,087,920 / 100,000,000 tokens

Step 1,100/2,289 | Loss 10.1467 | LR 1.849e-05 | 103,958 tok/s | 144,179,200 tokens | 48.06% | 0.39h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1100...
Writing model shards: 100% 1/1 [00:00<00:00, 10.97it/s]
======================================================================
UPLOADING CHECKPOINT 1100
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1100
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-900
FineWeb-HQ: 45,091,588 / 100,000,000 tokens
FineWeb-HQ: 46,091,666 / 100,000,000 tokens
FineWeb-HQ: 47,091,765 / 100,000,000 tokens
FineWeb-HQ: 48,093,667 / 100,000,000 tokens
FineWeb-HQ: 49,095,400 / 100,000,000 tokens
FineWeb-HQ: 50,100,158 / 100,000,000 tokens
FineWeb-HQ: 51,103,033 / 100,000,000 tokens
FineWeb-HQ: 52,103,833 / 100,000,000 tokens
FineWeb-HQ: 53,103,842 / 100,000,000 tokens
FineWeb-HQ: 54,105,589 / 100,000,000 tokens
FineWeb-HQ: 55,106,976 / 100,000,000 tokens
FineWeb-HQ: 56,108,077 / 100,000,000 tokens
FineWeb-HQ: 57,108,658 / 100,000,000 tokens

Step 1,200/2,289 | Loss 10.0994 | LR 1.655e-05 | 103,466 tok/s | 157,286,400 tokens | 52.43% | 0.42h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1200...
Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
======================================================================
UPLOADING CHECKPOINT 1200
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1200
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1000
FineWeb-HQ: 58,108,705 / 100,000,000 tokens
FineWeb-HQ: 59,109,817 / 100,000,000 tokens
FineWeb-HQ: 60,115,751 / 100,000,000 tokens
FineWeb-HQ: 61,116,056 / 100,000,000 tokens
FineWeb-HQ: 62,116,477 / 100,000,000 tokens
FineWeb-HQ: 63,116,951 / 100,000,000 tokens
FineWeb-HQ: 64,118,012 / 100,000,000 tokens
FineWeb-HQ: 65,122,172 / 100,000,000 tokens
FineWeb-HQ: 66,122,389 / 100,000,000 tokens
FineWeb-HQ: 67,124,930 / 100,000,000 tokens
FineWeb-HQ: 68,124,933 / 100,000,000 tokens
FineWeb-HQ: 69,126,235 / 100,000,000 tokens
FineWeb-HQ: 70,127,674 / 100,000,000 tokens

Step 1,300/2,289 | Loss 10.0603 | LR 1.461e-05 | 103,354 tok/s | 170,393,600 tokens | 56.80% | 0.46h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1300...
Writing model shards: 100% 1/1 [00:00<00:00, 10.95it/s]
======================================================================
UPLOADING CHECKPOINT 1300
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1300
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1100
FineWeb-HQ: 71,128,320 / 100,000,000 tokens
FineWeb-HQ: 72,130,508 / 100,000,000 tokens
FineWeb-HQ: 73,131,425 / 100,000,000 tokens
FineWeb-HQ: 74,145,259 / 100,000,000 tokens
FineWeb-HQ: 75,168,512 / 100,000,000 tokens
FineWeb-HQ: 76,169,236 / 100,000,000 tokens
FineWeb-HQ: 77,170,751 / 100,000,000 tokens
FineWeb-HQ: 78,171,712 / 100,000,000 tokens
FineWeb-HQ: 79,171,903 / 100,000,000 tokens
FineWeb-HQ: 80,172,653 / 100,000,000 tokens
FineWeb-HQ: 81,180,767 / 100,000,000 tokens
FineWeb-HQ: 82,209,091 / 100,000,000 tokens
FineWeb-HQ: 83,210,028 / 100,000,000 tokens

Step 1,400/2,289 | Loss 10.0248 | LR 1.270e-05 | 103,746 tok/s | 183,500,800 tokens | 61.17% | 0.49h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1400...
Writing model shards: 100% 1/1 [00:00<00:00, 10.83it/s]
======================================================================
UPLOADING CHECKPOINT 1400
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1400
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1200
FineWeb-HQ: 84,221,481 / 100,000,000 tokens
FineWeb-HQ: 85,221,902 / 100,000,000 tokens
FineWeb-HQ: 86,221,927 / 100,000,000 tokens
FineWeb-HQ: 87,224,781 / 100,000,000 tokens
FineWeb-HQ: 88,226,957 / 100,000,000 tokens
FineWeb-HQ: 89,227,905 / 100,000,000 tokens
FineWeb-HQ: 90,230,257 / 100,000,000 tokens
FineWeb-HQ: 91,234,535 / 100,000,000 tokens
FineWeb-HQ: 92,235,135 / 100,000,000 tokens
FineWeb-HQ: 93,235,228 / 100,000,000 tokens
FineWeb-HQ: 94,235,284 / 100,000,000 tokens
FineWeb-HQ: 95,242,150 / 100,000,000 tokens
FineWeb-HQ: 96,254,650 / 100,000,000 tokens

Step 1,500/2,289 | Loss 9.9967 | LR 1.088e-05 | 103,650 tok/s | 196,608,000 tokens | 65.54% | 0.53h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1500...
Writing model shards: 100% 1/1 [00:00<00:00, 10.86it/s]
======================================================================
UPLOADING CHECKPOINT 1500
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1500
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1300
FineWeb-HQ: 97,258,984 / 100,000,000 tokens
FineWeb-HQ: 98,259,041 / 100,000,000 tokens
FineWeb-HQ: 99,263,648 / 100,000,000 tokens

FineWeb-HQ complete: 100,000,000 tokens
Opening SmolLM-Corpus cosmopedia-v2...
README.md: 100% 7.05k/7.05k [00:00<00:00, 847kB/s]Resolving data files: 100% 104/104 [00:00<00:00, 6458.13it/s]Resolving data files: 100% 104/104 [00:00<00:00, 7784.83it/s]
======================================================================
STREAMING SmolLM-Corpus
======================================================================

Target tokens: 100,000,000
Progress state: 0

SmolLM-Corpus: 1,000,389 / 100,000,000 tokens
SmolLM-Corpus: 2,000,612 / 100,000,000 tokens
SmolLM-Corpus: 3,000,937 / 100,000,000 tokens
SmolLM-Corpus: 4,001,355 / 100,000,000 tokens
SmolLM-Corpus: 5,001,629 / 100,000,000 tokens
SmolLM-Corpus: 6,002,147 / 100,000,000 tokens
SmolLM-Corpus: 7,002,160 / 100,000,000 tokens
SmolLM-Corpus: 8,002,233 / 100,000,000 tokens
SmolLM-Corpus: 9,002,382 / 100,000,000 tokens

Step 1,600/2,289 | Loss 10.0072 | LR 9.168e-06 | 101,065 tok/s | 209,715,200 tokens | 69.91% | 0.57h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1600...
Writing model shards: 100% 1/1 [00:00<00:00, 10.94it/s]
======================================================================
UPLOADING CHECKPOINT 1600
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1600
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1400
SmolLM-Corpus: 10,002,523 / 100,000,000 tokens
SmolLM-Corpus: 11,002,661 / 100,000,000 tokens
SmolLM-Corpus: 12,003,098 / 100,000,000 tokens
SmolLM-Corpus: 13,003,147 / 100,000,000 tokens
SmolLM-Corpus: 14,004,159 / 100,000,000 tokens
SmolLM-Corpus: 15,004,183 / 100,000,000 tokens
SmolLM-Corpus: 16,004,330 / 100,000,000 tokens
SmolLM-Corpus: 17,005,124 / 100,000,000 tokens
SmolLM-Corpus: 18,005,586 / 100,000,000 tokens
SmolLM-Corpus: 19,006,013 / 100,000,000 tokens
SmolLM-Corpus: 20,006,056 / 100,000,000 tokens
SmolLM-Corpus: 21,006,417 / 100,000,000 tokens
SmolLM-Corpus: 22,006,642 / 100,000,000 tokens

Step 1,700/2,289 | Loss 9.9950 | LR 7.613e-06 | 104,388 tok/s | 222,822,400 tokens | 74.27% | 0.60h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1700...
Writing model shards: 100% 1/1 [00:00<00:00, 11.01it/s]
======================================================================
UPLOADING CHECKPOINT 1700
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1700
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1500
SmolLM-Corpus: 23,007,410 / 100,000,000 tokens
SmolLM-Corpus: 24,007,513 / 100,000,000 tokens
SmolLM-Corpus: 25,008,024 / 100,000,000 tokens
SmolLM-Corpus: 26,008,152 / 100,000,000 tokens
SmolLM-Corpus: 27,008,820 / 100,000,000 tokens
SmolLM-Corpus: 28,009,655 / 100,000,000 tokens
SmolLM-Corpus: 29,009,777 / 100,000,000 tokens
SmolLM-Corpus: 30,010,160 / 100,000,000 tokens
SmolLM-Corpus: 31,010,762 / 100,000,000 tokens
SmolLM-Corpus: 32,010,887 / 100,000,000 tokens
SmolLM-Corpus: 33,011,339 / 100,000,000 tokens
SmolLM-Corpus: 34,011,816 / 100,000,000 tokens
SmolLM-Corpus: 35,012,785 / 100,000,000 tokens

Step 1,800/2,289 | Loss 9.9749 | LR 6.242e-06 | 104,425 tok/s | 235,929,600 tokens | 78.64% | 0.63h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1800...
Writing model shards: 100% 1/1 [00:00<00:00, 11.00it/s]
======================================================================
UPLOADING CHECKPOINT 1800
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1800
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1600
SmolLM-Corpus: 36,013,511 / 100,000,000 tokens
SmolLM-Corpus: 37,013,686 / 100,000,000 tokens
SmolLM-Corpus: 38,015,046 / 100,000,000 tokens
SmolLM-Corpus: 39,015,520 / 100,000,000 tokens
SmolLM-Corpus: 40,016,051 / 100,000,000 tokens
SmolLM-Corpus: 41,016,215 / 100,000,000 tokens
SmolLM-Corpus: 42,016,528 / 100,000,000 tokens
SmolLM-Corpus: 43,016,537 / 100,000,000 tokens
SmolLM-Corpus: 44,016,544 / 100,000,000 tokens
SmolLM-Corpus: 45,017,221 / 100,000,000 tokens
SmolLM-Corpus: 46,017,703 / 100,000,000 tokens
SmolLM-Corpus: 47,017,830 / 100,000,000 tokens
SmolLM-Corpus: 48,018,530 / 100,000,000 tokens
SmolLM-Corpus: 49,019,055 / 100,000,000 tokens

Step 1,900/2,289 | Loss 9.9581 | LR 5.086e-06 | 104,428 tok/s | 249,036,800 tokens | 83.01% | 0.67h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-1900...
Writing model shards: 100% 1/1 [00:00<00:00, 11.07it/s]
======================================================================
UPLOADING CHECKPOINT 1900
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-1900
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1700
SmolLM-Corpus: 50,019,088 / 100,000,000 tokens
SmolLM-Corpus: 51,019,166 / 100,000,000 tokens
SmolLM-Corpus: 52,019,182 / 100,000,000 tokens
SmolLM-Corpus: 53,019,341 / 100,000,000 tokens
SmolLM-Corpus: 54,019,557 / 100,000,000 tokens
SmolLM-Corpus: 55,019,622 / 100,000,000 tokens
SmolLM-Corpus: 56,020,576 / 100,000,000 tokens
SmolLM-Corpus: 57,021,061 / 100,000,000 tokens
SmolLM-Corpus: 58,021,198 / 100,000,000 tokens
SmolLM-Corpus: 59,022,144 / 100,000,000 tokens
SmolLM-Corpus: 60,022,635 / 100,000,000 tokens
SmolLM-Corpus: 61,022,949 / 100,000,000 tokens
SmolLM-Corpus: 62,023,318 / 100,000,000 tokens

Step 2,000/2,289 | Loss 9.9444 | LR 4.167e-06 | 104,400 tok/s | 262,144,000 tokens | 87.38% | 0.70h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-2000...
Writing model shards: 100% 1/1 [00:00<00:00, 10.99it/s]
======================================================================
UPLOADING CHECKPOINT 2000
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2000
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1800
SmolLM-Corpus: 63,024,246 / 100,000,000 tokens
SmolLM-Corpus: 64,024,602 / 100,000,000 tokens
SmolLM-Corpus: 65,025,944 / 100,000,000 tokens
SmolLM-Corpus: 66,027,471 / 100,000,000 tokens
SmolLM-Corpus: 67,027,613 / 100,000,000 tokens
SmolLM-Corpus: 68,028,290 / 100,000,000 tokens
SmolLM-Corpus: 69,028,776 / 100,000,000 tokens
SmolLM-Corpus: 70,028,797 / 100,000,000 tokens
SmolLM-Corpus: 71,030,241 / 100,000,000 tokens
SmolLM-Corpus: 72,030,477 / 100,000,000 tokens
SmolLM-Corpus: 73,030,534 / 100,000,000 tokens
SmolLM-Corpus: 74,030,705 / 100,000,000 tokens
SmolLM-Corpus: 75,030,996 / 100,000,000 tokens

Step 2,100/2,289 | Loss 9.9332 | LR 3.505e-06 | 104,456 tok/s | 275,251,200 tokens | 91.75% | 0.74h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-2100...
Writing model shards: 100% 1/1 [00:00<00:00, 10.96it/s]
======================================================================
UPLOADING CHECKPOINT 2100
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2100
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-1900
SmolLM-Corpus: 76,031,414 / 100,000,000 tokens
SmolLM-Corpus: 77,031,466 / 100,000,000 tokens
SmolLM-Corpus: 78,032,243 / 100,000,000 tokens
SmolLM-Corpus: 79,032,688 / 100,000,000 tokens
SmolLM-Corpus: 80,032,933 / 100,000,000 tokens
SmolLM-Corpus: 81,033,099 / 100,000,000 tokens
SmolLM-Corpus: 82,034,041 / 100,000,000 tokens
SmolLM-Corpus: 83,034,272 / 100,000,000 tokens
SmolLM-Corpus: 84,035,283 / 100,000,000 tokens
SmolLM-Corpus: 85,035,460 / 100,000,000 tokens
SmolLM-Corpus: 86,035,589 / 100,000,000 tokens
SmolLM-Corpus: 87,036,259 / 100,000,000 tokens
SmolLM-Corpus: 88,036,685 / 100,000,000 tokens

Step 2,200/2,289 | Loss 9.9239 | LR 3.114e-06 | 104,304 tok/s | 288,358,400 tokens | 96.12% | 0.77h
VRAM: 1.57 GB allocated / 13.91 GB reserved


Saving checkpoint-2200...
Writing model shards: 100% 1/1 [00:00<00:00, 10.98it/s]
======================================================================
UPLOADING CHECKPOINT 2200
======================================================================
No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

Checkpoint uploaded to:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints/tree/main/checkpoint-2200
Removing old local checkpoint: /content/applemind-1.0-mini-checkpoints/checkpoint-2000
SmolLM-Corpus: 89,036,736 / 100,000,000 tokens
SmolLM-Corpus: 90,037,952 / 100,000,000 tokens
SmolLM-Corpus: 91,038,162 / 100,000,000 tokens
SmolLM-Corpus: 92,038,464 / 100,000,000 tokens
SmolLM-Corpus: 93,038,712 / 100,000,000 tokens
SmolLM-Corpus: 94,038,967 / 100,000,000 tokens
SmolLM-Corpus: 95,039,446 / 100,000,000 tokens
SmolLM-Corpus: 96,039,542 / 100,000,000 tokens
SmolLM-Corpus: 97,040,355 / 100,000,000 tokens
SmolLM-Corpus: 98,041,440 / 100,000,000 tokens
SmolLM-Corpus: 99,041,658 / 100,000,000 tokens

SmolLM-Corpus complete: 100,000,000 tokens

======================================================================
DATASET STREAM EXHAUSTED
======================================================================


Final dataset batch was incomplete.
Stopping without an incomplete optimizer step.

======================================================================
TRAINING LOOP FINISHED
======================================================================

Completed optimizer steps: 2,288
Configured optimizer steps: 2,289

Dataset progress:
  FineWeb-Edu: 100,000,000 / 100,000,000
  FineWeb-HQ:  100,000,000 / 100,000,000
  SmolLM:      100,000,000 / 100,000,000

======================================================================
PRETRAINING FINISHED
======================================================================

Final steps: 2,288
Final tokens: 299,892,736
Target tokens: 300,000,000

Actual tokens/parameter: 293.87:1

======================================================================
SAVING FINAL MODEL
======================================================================
Writing model shards: 100% 1/1 [00:00<00:00, 21.09it/s]
======================================================================
UPLOADING COMPLETE MODEL TO HUGGING FACE
======================================================================

No files have been modified since last commit. Skipping to prevent empty commit.
WARNING:huggingface_hub._upload_pipeline:No files have been modified since last commit. Skipping to prevent empty commit.

FINAL MODEL UPLOAD SUCCESSFUL!

Final model:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini

======================================================================
CHAT TEMPLATE TEST
======================================================================

<|system|>
You are AppleMind, a helpful AI assistant.
<|user|>
Hello!
<|assistant|>


======================================================================
GENERATION TEST
======================================================================

Prompt: Once upon a time
------------------------------------------------------------
Once upon a time It the several times- nowThis, does to form provide from find another times work not atl The couldWhen on all be way H It lives among times always, worked
 its G during work used after several There and at there
b known came be very that It thought It betweenIn course does. case other It 5?: or often I at's the: enough could in many
------------------------------------------------------------

Prompt: The little boy
------------------------------------------------------------
The little boy's form among. I and H be from� used. still all- G, lives take same always often its find amonged provide- number because: another now? then there among use course not thought case work usel result It between 5 It well atWhen new.: thatThis work on think 3 interestB then It does could do the among Ire use among now does to and many
------------------------------------------------------------

Prompt: In the forest
------------------------------------------------------------
In the forest often
 form and times on during severall find, several The then number between: well work take there provide, times among anotherWhen same Ged but lives could the to times its- course same enough and same I used
 to same other not thought There H think� same 2 I body nowe cameb 5- do among's and several? same after- still,This- interest but
------------------------------------------------------------

======================================================================
APPLE MIND 1.0 MINI COMPLETE
======================================================================

Parameters: 1,020,480
Tokens: 299,892,736
Tokens/parameter: 293.87:1

Dataset mixture:
  FineWeb-Edu: 100M
  FineWeb-HQ:  100M
  SmolLM:      100M

Checkpoint repository:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini-Checkpoints

Final model repository:
https://huggingface.co/AppleMind-AI/AppleMind-1.0-Mini

AppleMind 1.0 Mini complete!