| ---
|
| license: apache-2.0
|
| language:
|
| - en
|
| pipeline_tag: text-generation
|
| library_name: transformers
|
| datasets:
|
| - HuggingFaceFW/fineweb-edu
|
| tags:
|
| - causal-lm
|
| - custom-code
|
| - educational
|
| - pytorch
|
| ---
|
|
|
| # FineWeb 100M Base
|
|
|
| A 97.5M-parameter decoder-only base language model trained from scratch on four billion FineWeb-Edu tokens.
|
|
|
| > This is a small educational and experimental model. It is not a production assistant and should not be relied on for factual, medical, legal, financial, safety-critical, or other consequential advice.
|
|
|
| ## Model details
|
|
|
| | Property | Value |
|
| |---|---:|
|
| | Parameters | 97,536,768 |
|
| | Layers | 12 |
|
| | Hidden width | 768 |
|
| | Attention heads | 12 |
|
| | SwiGLU hidden width | 2048 |
|
| | Vocabulary | 16,384 BPE tokens |
|
| | Context window | 1,024 tokens |
|
| | Architecture | Decoder-only Transformer |
|
| | Position encoding | RoPE |
|
| | Normalization | RMSNorm |
|
| | Embeddings | Tied input/output embeddings |
|
|
|
| The architecture was implemented from scratch in PyTorch. This repository includes custom Transformers-compatible configuration and modeling files.
|
|
|
| ## Training
|
|
|
| The base model was trained from scratch on 4,000,055,296 packed tokens selected from the FineWeb-Edu `sample-10BT` corpus. Documents were selected deterministically by hashing document IDs rather than taking the first source shards. Text was tokenized with a locally trained byte-level BPE tokenizer, an `<eos>` token was appended after every document, and tokens were packed into fixed 1,024-token blocks.
|
|
|
| Base training used AdamW, cosine learning-rate decay, gradient clipping, bfloat16 autocast, and a global batch size of 65,536 tokens on an NVIDIA RTX 3090.
|
|
|
| ## Evaluation
|
|
|
| Best validation loss: 2.8886 (next-token validation).
|
|
|
| These losses are internal held-out validation measurements. Base and SFT losses use different objectives, and losses from the 8K and 16K tokenizers are not directly comparable. No broad academic benchmark suite or human preference evaluation was run.
|
|
|
| ## Usage
|
|
|
| Install the dependencies:
|
|
|
| ```bash
|
| pip install -r requirements.txt
|
| ```
|
|
|
| Run the included example after cloning this repository:
|
|
|
| ```bash
|
| python generate.py --model . --prompt "The purpose of education is"
|
| ```
|
|
|
| Or load it through Transformers:
|
|
|
| ```python
|
| from transformers import AutoModelForCausalLM
|
|
|
| model = AutoModelForCausalLM.from_pretrained(
|
| ".",
|
| trust_remote_code=True,
|
| )
|
| ```
|
|
|
| `trust_remote_code=True` is required because this is a custom architecture. Review `configuration_fineweb.py` and `modeling_fineweb.py` before loading code from the Hub.
|
|
|
| ## Limitations
|
|
|
| - The model is very small by modern language-model standards and often produces incorrect, repetitive, incoherent, or fabricated text.
|
| - Its knowledge is limited to patterns learned from the training data; it has no live information retrieval, tools, memory, or reliable sense of current time.
|
| - The 1,024-token context window includes both the prompt and generated output.
|
| - FineWeb-Edu is English-focused web data and can contain errors, biases, sensitive material, and uneven topic coverage.
|
| - The SFT variants received a small amount of instruction tuning and are not robust chat assistants.
|
| - The model has not been evaluated for safety, fairness, memorization, or production deployment.
|
|
|
| ## Training data and attribution
|
|
|
| - [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), licensed ODC-By 1.0, was used for base pretraining.
|
|
|
|
|
| The source datasets are not redistributed in this model repository and remain subject to their respective licences and attribution requirements.
|
|
|
| ## Licence
|
|
|
| The model weights and original code in this repository are released under the Apache License 2.0. This does not replace or override the licences of the training datasets.
|
|
|
