README.md

---
license: cc-by-nc-4.0
datasets:
  - HuggingFaceFW/fineweb-edu
language:
  - en
tags:
  - pretrained-from-scratch
  - causal-lm
  - gpt
  - rope
  - swiglu
  - grouped-query-attention
pipeline_tag: text-generation
library_name: transformers
---
# LLMPretrain — a 101M decoder-only LM trained from scratch

Every weight in this model was produced by this project. There is no
`from_pretrained`, no base checkpoint, and no adapter: the tokenizer, the
architecture code, the data pipeline and the training loop are all bespoke, and
training started from a random initialisation.

Trained on a **single RTX 5070 Ti (16 GB)**.

## Architecture

Written by hand — no `transformers` model classes were used to define it.

| | |
|---|---|
| Type | decoder-only, pre-norm transformer |
| Parameters | **100,684,032** |
| Layers | 12 |
| d_model | 768 |
| Attention heads | 12 query / 4 KV (**GQA**) |
| Head dim | 64 |
| FFN | **SwiGLU**, hidden 2048 (≈ 8/3 × d_model) |
| Normalisation | **RMSNorm** (pre-norm) + **QK-norm** |
| Positions | **RoPE** (θ=10000) — no learned position embeddings |
| Context | 1024 |
| Vocab | 32,768 (own byte-level BPE, padded to a multiple of 128) |
| Biases | none, in any linear layer |
| Embeddings | tied between input and LM head |

Parameter split: **25,165,824** embedding
(25.0%) /
**75,517,440** transformer blocks.

## Training

| | |
|---|---|
| Tokens seen | 9,999,745,024 |
| Steps | 19,073 |
| Schedule | warmup-stable-decay (WSD) |
| Optimizer | fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only |
| Precision | bf16 autocast, fp32 master weights |
| Hardware | 1 × RTX 5070 Ti 16 GB |

## Evaluation

**Bits per byte is the headline metric.** This model's vocabulary
(32,768) differs from GPT-2's (50,257), and cross-entropy per
*token* is not comparable across tokenizers. BPB divides that out and is
directly comparable.

| Metric | This model | GPT-2 124M |
|---|---|---|
| Parameters | 100,684,032 | 124,439,808 |
| Validation loss | 2.9725 | — |
| **Bits per byte** | **0.9312** | — |
| HellaSwag (0-shot, len-norm) | _pending_ | 29.55% |


Validation loss is measured on a held-out slice of the training corpus and is
*not* comparable to GPT-2's OpenWebText number. Bits per byte is, because it is
tokenizer-independent.

## Data provenance and licensing

| Source | HF id | License |
|---|---|---|
| fineweb-edu | `HuggingFaceFW/fineweb-edu` | ODC-By 1.0 |

Attribution: FineWeb-Edu (HuggingFaceFW/fineweb-edu), ODC-By 1.0

ODC-By requires attribution, which is given above. Whether pretraining-data
licenses flow through to model weights is legally unsettled; permissively
licensed sources were chosen deliberately as the most defensible posture.
Model weights are released under **cc-by-nc-4.0**.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Dikshan1234/LLMPretrain")
model = AutoModelForCausalLM.from_pretrained("Dikshan1234/LLMPretrain",
                                             trust_remote_code=True)

inputs = tok("Once upon a time", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100, do_sample=True,
                     temperature=0.8, top_p=0.95)
print(tok.decode(out[0]))
```

`trust_remote_code=True` is required because the architecture (RoPE + SwiGLU +
GQA + QK-norm) is not a stock `transformers` model. The two files it loads,
`configuration_llmpretrain.py` and `modeling_llmpretrain.py`, are in this repo
and are short enough to read.

## Sample generations

```
===== step 19073 =====

--- [greedy] 'Once upon a time'
Once upon a time, the world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power.
The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power.
The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great

--- [t=0.8] 'Once upon a time'
Once upon a time, a man had a big dream. What would Jesus have done to make that dream come true?
Jesus answered that: “It is not possible to make any other man come to my aid than you.” (John 8:45)
I believe it is the ultimate goal of every successful leader to be his or her own helper. What we must do is to not only make sure that we will be the ideal leader, but that we will be his or her own helper.
Jesus went on to say: “I have chosen the time to do all it takes to bring the

--- [greedy] 'The capital of France is'
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is

--- [t=0.8] 'The capital of France is'
The capital of France is Paris. The capital of France is Paris.
French is the official language. The French language is known as Creole. French spoken in France is French, but not Creole. France is a French-speaking country.
French is a Germanic language. The French language is a Germanic language.
French is a Germanic language. The French language is a Germanic language. France is a Germanic-speaking country.
French is a Germanic language. The French language is a Germanic language.
French is a Germanic language. The French language is a Germanic-speaking country.
French is a Germanic

--- [greedy] 'In 1969, humans first'
In 1969, humans first began to use the word “machine” to describe the machines that they used to make the world. The word “machine” was first used in the context of the computer industry, and it was used to describe the machines that were used to make the world.
The word “machine” is also used to describe the machines that are used to make the world. The word “machine” is used to describe the machines that are used to make the world. The word “machine” is also used to describe the machines that are used to make the world. The word “

--- [t=0.8] 'In 1969, humans first'
In 1969, humans first began to show signs of humanlike behavior, and as many as 10,000 people have been diagnosed with autism.
In 2009, a study published in the journal Nature Neuroscience shed light on the link between autism and the gut microbiota. By studying the gut microbiota in autistic children, researchers were able to trace the genes that are responsible for the onset of the condition.
There was some evidence that the human gut bacteria play an important role in the development of autism. Specifically, researchers found that the gut bacteria in autistic children have genetic links to autism.
Researchers also found that the gut bacteria in autistic

--- [greedy] 'The three states of matter are'
The three states of matter are:
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance th
```

## Limitations

This is a base model trained on a modest token budget. It has had no
instruction tuning, no RLHF and no safety alignment. It will produce
confidently wrong text, and it reflects whatever biases exist in its training
corpus. It is a research and educational artifact, not a product.