Release Pollock 1.0 (r002)

CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+Revision numbers identify published model states independently of release names.
+
+## r002 — Pollock 1.0
+
+- Expanded the architecture from 12/12/768 to 12/14/896 while remaining below 128M total parameters.
+- Reduced context from 2,048 to 1,024 tokens and increased the effective batch from 262,144 to 491,520 tokens.
+- Changed the learning-rate range from 6e-4→6e-5 to 3e-4→3e-5 and warmup from 500 to 440 iterations.
+- Trained with BF16 on 2× RTX 4090 instead of 1× RTX 5090.
+- Introduced sampled training validation and a seven-task English zero-shot benchmark suite.
+
+Full record: [`training-history/r002.md`](./training-history/r002.md)
+
+## r001 — experimental predecessor
+
+- Initial experimental release with a 12/12/768 architecture, 2,048-token context, and 95.96M nanoGPT-reported parameters.
+- Trained for four dataset passes and selected by deterministic full-validation reevaluation of retained checkpoints.
+- No downstream benchmark suite was run.
+
+Full record and generation samples: [`training-history/r001.md`](./training-history/r001.md)

README.md

@@ -1,99 +1,107 @@
 ---
-language:
-  - en
+language: [en]
 library_name: transformers
 pipeline_tag: text-generation
 license: other
 license_name: mixed-upstream-dataset-terms
 license_link: https://huggingface.co/SlayerLab/pollock-mini-lm-125m/blob/main/LICENSE.md
-datasets:
-  - SlayerLab/minimal-en-corpus-2.5b
-tags:
-  - causal-lm
-  - gpt2
-  - nanogpt
-  - bpe
-  - educational
+datasets: [SlayerLab/minimal-en-corpus-2.5b]
+tags: [causal-lm, gpt2, nanogpt, bpe, educational, base-model]
 model-index:
-  - name: pollock-mini-lm
+  - name: Pollock 1.0
     results:
-      - task:
-          type: text-generation
-          name: Text Generation
-        dataset:
-          type: SlayerLab/minimal-en-corpus-2.5b
-          name: Minimal EN validation
-          split: validation
-        metrics:
-          - type: loss
-            value: 3.40875361
-          - type: perplexity
-            value: 30.227545
+      - task: {type: text-generation, name: Language modeling}
+        dataset: {type: SlayerLab/minimal-en-corpus-2.5b, name: Minimal EN validation (sampled), split: validation}
+        metrics: [{type: loss, value: 2.577547, name: Final sampled validation loss}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: blimp, name: BLiMP, split: train}
+        metrics: [{type: acc, value: 0.7669701493}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: EleutherAI/lambada_openai, name: LAMBADA OpenAI, split: test}
+        metrics: [{type: acc, value: 0.2780904328}, {type: perplexity, value: 53.66752251}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: hellaswag, name: HellaSwag, split: validation}
+        metrics: [{type: acc_norm, value: 0.2987452699}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: piqa, name: PIQA, split: validation}
+        metrics: [{type: acc_norm, value: 0.6033732318}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: sciq, name: SciQ, split: test}
+        metrics: [{type: acc_norm, value: 0.658}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: allenai/ai2_arc, config: ARC-Easy, name: ARC-Easy, split: test}
+        metrics: [{type: acc_norm, value: 0.422979798}]
+      - task: {type: text-generation, name: Zero-shot evaluation}
+        dataset: {type: allenai/ai2_arc, config: ARC-Challenge, name: ARC-Challenge, split: test}
+        metrics: [{type: acc_norm, value: 0.2414675768}]
 ---
 
-# pollock-mini-lm
+# Pollock 1.0 — r002
 
-![Avatar pollock-mini-lm](./assets/pollock-mini-lm-avatar-320.png)
+![Pollock avatar](./assets/pollock-mini-lm-avatar-320.png)
 
-> **Wersja eksperymentalna / Experimental release.** Model nie jest przeznaczony do zastosowań produkcyjnych ani wysokiego ryzyka. / The model is not intended for production or high-risk use.
+> **Wydanie eksperymentalne / Experimental release.** Model nie jest przeznaczony do zastosowań produkcyjnych ani wysokiego ryzyka. Rozmiar może różnić się między rewizjami, ale cały model zawsze pozostaje poniżej 128M parametrów. / The model is not intended for production or high-risk use. Its size may vary between revisions, but the complete model always remains below 128M parameters.
 
 ## Polski
 
 ### Opis
 
-`pollock-mini-lm` to niewielki, anglojęzyczny model przyczynowy stworzony jako czytelny eksperyment edukacyjny. Nazwa podkreśla miniaturowy charakter projektu. Implementacja bazuje na [nanoGPT](https://github.com/karpathy/nanoGPT), uproszczonym przepisaniu minGPT, oraz na własnym tokenizerze byte-level BPE.
+Pollock 1.0 (`r002`) to niewielki, anglojęzyczny model bazowy typu decoder-only, wytrenowany od zera jako czytelny eksperyment edukacyjny. Implementacja bazuje na [nanoGPT](https://github.com/karpathy/nanoGPT) i własnym tokenizerze byte-level BPE. Jest to model do uzupełniania tekstu, nie asystent konwersacyjny.
 
-Nazwa modelu luźno nawiązuje do gestu malarskiego Jacksona Pollocka i idei kontrolowanego eksperymentu z przypadkiem. nanoGPT pełni tu rolę płótna, na którym różne dane, konfiguracje i decyzje treningowe tworzą odmienne wzorce zachowania modelu. Istotny jest nie tylko końcowy rezultat, ale również obserwacja procesu i wpływu poszczególnych składników eksperymentu.
-
-Prostota jest celowa: projekt ma pomagać osobom rozpoczynającym tworzenie modeli — włączając w to autora — zrozumieć, które decyzje dotyczące architektury, danych i treningu wpływają na wynik, a następnie przenieść te metody do własnych eksperymentów.
+Nazwa luźno nawiązuje do gestu malarskiego Jacksona Pollocka: nanoGPT jest płótnem, na którym dane, konfiguracja i decyzje treningowe tworzą różne wzorce zachowania. Pełne dane techniczne tej wersji znajdują się w [`training-history/r002.md`](./training-history/r002.md), a różnice między wydaniami w [`CHANGELOG.md`](./CHANGELOG.md).
 
 ### Architektura i tokenizer
 
 | Właściwość | Wartość |
 |---|---:|
-| Typ | decoder-only Transformer, klasa GPT-2 small |
-| Parametry nanoGPT | 95 963 904 (około 96M) |
-| Warstwy / głowy / embedding | 12 / 12 / 768 |
-| Maksymalny kontekst | 2048 tokenów |
+| Rewizja / wydanie | r002 / Pollock 1.0 |
+| Typ | decoder-only Transformer w stylu GPT-2 |
+| Warstwy / głowy / embedding | 12 / 14 / 896 |
+| Maksymalny kontekst | 1024 tokeny |
 | Słownik | 12 288 tokenów |
+| Parametry nanoGPT | 126 637 952 |
+| Łączne unikalne parametry trenowalne | 127 555 456 |
 | Tokenizer | byte-level BPE, pretokenizacja w stylu GPT-2 |
 | Tokeny specjalne | <code>&lt;&#124;endoftext&#124;&gt;</code>, <code>&lt;&#124;im_start&#124;&gt;</code>, <code>&lt;&#124;im_end&#124;&gt;</code> |
 
-### Dane
-
-Model wytrenowano na [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b): mieszance 15 anglojęzycznych źródeł dobranej subiektywnie na zasadzie „common sense”, a nie poprzez naukową optymalizację proporcji.
-
-- około 2,5 mld tokenów oszacowanych przed finalną tokenizacją;
-- 2 703 768 815 tokenów BPE w zbiorze treningowym;
-- 5 267 742 tokeny BPE w zbiorze walidacyjnym;
-- deduplikacja dokładna i przybliżona, filtrowanie języka oraz decontaminacja benchmarków zostały wykonane na etapie przygotowania datasetu.
+Artefakt Transformers ma 127 674 624 parametrów, w tym 119 168 zerowych parametrów bias dla zgodności z `GPT2LMHeadModel`. Natywny model był trenowany z `bias=False`.
 
-Karta datasetu zawiera pełną listę źródeł i informacje o ich licencjach. Mieszanka nie nadaje dokumentom jednej wspólnej licencji.
+### Dane i trening
 
-### Trening
+Model wytrenowano na [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), subiektywnie dobranej mieszance 15 anglojęzycznych źródeł. Przygotowanie obejmowało deduplikację dokładną i przybliżoną, filtrowanie języka oraz decontaminację benchmarków.
 
 | Parametr | Wartość |
 |---|---:|
-| Kroki optymalizatora | 41 257 |
-| Przetworzone tokeny | 10 815 275 008 (4 epoki) |
-| Effective batch | 262 144 tokeny |
-| Sekwencja / micro-batch / akumulacja | 2048 / 32 / 4 |
+| Tokeny treningowe / walidacyjne | 2 703 768 815 / 5 267 742 |
+| Finalny checkpoint | iteracja 22 004 |
+| Przetworzone tokeny | 10 815 406 080 (około 4 epoki) |
+| Sekwencja / micro-batch na GPU | 1024 / 12 |
+| Akumulacja globalna / na GPU | 40 / 20 micro-stepów |
+| Effective batch | 491 520 tokenów |
 | Optymalizator | fused AdamW, betas 0.9/0.95 |
-| Learning rate | 6e-4 → 6e-5, cosine decay |
-| Warmup / weight decay / grad clip | 500 / 0.1 / 1.0 |
+| Learning rate | 3e-4 → 3e-5, cosine decay |
+| Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
 | Precyzja | BF16 |
-| Sprzęt | 1× NVIDIA GeForce RTX 5090 32 GB |
+| Sprzęt | 2× NVIDIA GeForce RTX 4090 24 GB |
 | Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Ewaluacja
 
-Wydanie wykorzystuje końcowy checkpoint, który uzyskał najlepszy wynik spośród wszystkich zachowanych checkpointów w deterministycznej ewaluacji pełnego `val.bin`.
+Loss treningowy szacowano na 100 losowo próbkowanych batchach na split. Finalny checkpoint uzyskał validation loss **2.577547**; najlepszy próbkowany wynik to **2.5600** w kroku 20 000. Nie należy porównywać tych wartości bezpośrednio z pełną, deterministyczną walidacją r001.
 
-| Zbiór | Tokeny ocenione | Loss | Perplexity |
-|---|---:|---:|---:|
-| Minimal EN validation | 5 267 456 | 3.40875361 | 30.227545 |
+Benchmarki wykonano zero-shot na pełnych splitach przy użyciu `lm-evaluation-harness` 0.4.12, batch size 8 i BF16.
 
-Ewaluacja użyła 2572 kolejnych, niepokrywających się okien po 2048 tokenów w BF16. Nie przeprowadzono benchmarków downstream. Podczas treningu wykryto błąd historycznej logiki `best_val_loss`; dlatego wszystkie zachowane checkpointy porównano ponownie tą samą deterministyczną metodą, zamiast ufać nazwie `checkpoint_best_val.pt`.
+| Benchmark | Główna metryka | Wynik | Próbki |
+|---|---|---:|---:|
+| BLiMP | accuracy | 0.766970 | 67 000 |
+| LAMBADA OpenAI | accuracy | 0.278090 | 5 153 |
+| HellaSwag | normalized accuracy | 0.298745 | 10 042 |
+| PIQA | normalized accuracy | 0.603373 | 1 838 |
+| SciQ | normalized accuracy | 0.658000 | 1 000 |
+| ARC-Easy | normalized accuracy | 0.422980 | 2 376 |
+| ARC-Challenge | normalized accuracy | 0.241468 | 1 172 |
+
+LAMBADA osiągnęła perplexity 53.667523. Pełne metryki i protokół zapisano w historii r002.
 
 ### Użycie z Transformers
 
@@ -106,141 +114,102 @@ model = AutoModelForCausalLM.from_pretrained(model_id)
 
 inputs = tokenizer("Once upon a time", return_tensors="pt")
 output = model.generate(
-    **inputs,
-    max_new_tokens=100,
-    do_sample=True,
-    temperature=0.7,
-    top_k=50,
+    **inputs, max_new_tokens=100, do_sample=True, temperature=0.7, top_k=50
 )
 print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
 
-Model korzysta ze standardowego `GPT2LMHeadModel` i nie wymaga `trust_remote_code`.
-
-### Przykładowe generacje
-
-Poniższe próbki wygenerowano bez ręcznej selekcji wyników. Ustawienia: seed 1337 resetowany dla każdego promptu, temperature 0.7, top-k 50, maksymalnie 100 nowych tokenów. Prompt jest częścią pokazanego wyniku.
-
-#### Wiedza ogólna / General knowledge
-
-Prompt:
-
-```text
-The capital of France is
-```
-
-Wynik / Output:
-
-```text
-The capital of France is a first of the most important element in the construction of the Socket.
-
-Most of the socket is not a type of sandstone but it is generally a rocket.
-
-As a result, there is also an exception of the rocket site. The socket is called a rocket.
-
-The rocket is the only rocket.
-The rocket is the highest rocket.
-
-The rocket is the highest rocket.
-
-The rocket
-```
-
-#### Proste wyjaśnienie / Explanation
-
-Prompt:
-
-```text
-Photosynthesis is the process by which
-```
-
-Wynik / Output:
-
-```text
-Photosynthesis is the process by which the first two events are found. A key option is to do a function in some cases, so it is called the first two terms. This is what I want to think. The first one is to multiply the number by the number of digits. (The second one is to multiply the number by the first two terms. The first two terms are divided by the second one, so that is the first four terms. The third one is to multiply the number by the number of digits. The third
-```
-
-#### Narracja / Story
-
-Prompt:
-
-```text
-In a distant future, humanity discovered
-```
-
-Wynik / Output:
-
-```text
-In a distant future, humanity discovered to change, as well as the "coneurship and a continuance of the body.
-This is not a matter of the general sense, but a lot of things we call us to the body. If we feel the body is the body’s internal structure, we’re going to change the way. The body is the body’s potential structure, meaning that it is used for a body to balance and develop a sensory to control each other.
-What is the
-```
-
-#### Kod / Code
-
-Prompt:
-
-```text
-def fibonacci(n):
-```
-
-Wynik / Output:
-
-```text
-def fibonacci(n):)) = (n - 1) + (n - 1) + ( n - 1) + ( n - 2) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n
-```
+Model używa standardowego `GPT2LMHeadModel`; `trust_remote_code=True` nie jest potrzebne. Dla r002 nie opublikowano jeszcze próbek generacji. Historyczne próbki są przypisane do poprzednich wag w historii r001.
 
 ### Ograniczenia i odpowiedzialne użycie
 
-- Jest to surowy model bazowy bez instruction tuningu, RLHF ani alignmentu. Nie jest asystentem konwersacyjnym.
-- Może halucynować, zaprzeczać samemu sobie oraz generować treści toksyczne, stronnicze, niebezpieczne lub przypominające dane treningowe.
-- Rozmiar około 96M ogranicza wiedzę, rozumowanie, spójność długiego tekstu i jakość kodu.
-- Dane są głównie anglojęzyczne; działanie w innych językach nie było oceniane.
-- Nie należy używać modelu do porad medycznych, prawnych, finansowych, decyzji o ludziach ani innych zastosowań wysokiego ryzyka.
-- Użytkownik odpowiada za ocenę licencji i warunków źródeł danych dla swojego zastosowania.
+- Jest to surowy model bazowy bez instruction tuningu, RLHF ani alignmentu.
+- Może halucynować oraz generować treści toksyczne, stronnicze, niebezpieczne lub podobne do danych treningowych.
+- Mały rozmiar ogranicza wiedzę, rozumowanie, spójność długiego tekstu i jakość kodu.
+- Model trenowano i oceniano głównie po angielsku.
+- Nie należy używać go do porad medycznych, prawnych i finansowych, decyzji o ludziach ani zastosowań wysokiego ryzyka.
+- Użytkownik odpowiada za ocenę warunków źródeł danych dla swojego zastosowania.
 
 ## English
 
 ### Overview
 
-`pollock-mini-lm` is a small English causal language model built as a readable educational experiment. Its name emphasizes the project's miniature scope. The implementation is based on [nanoGPT](https://github.com/karpathy/nanoGPT), itself a simplified rewrite of minGPT, and uses a custom byte-level BPE tokenizer.
-
-The model's name loosely refers to Jackson Pollock's painterly gesture and to the idea of a controlled experiment with chance. nanoGPT serves as a canvas on which different data, configurations, and training decisions create distinct patterns of model behavior. What matters is not only the final result, but also observing the process and the influence of each component of the experiment.
+Pollock 1.0 (`r002`) is a small English decoder-only base language model trained from scratch as a readable educational experiment. It is based on [nanoGPT](https://github.com/karpathy/nanoGPT) and a custom byte-level BPE tokenizer. It is a completion model, not a conversational assistant.
 
-The simplicity is intentional. The project is meant to help newcomers — including the author — understand how architecture, data, and training choices affect model quality and reuse the same methods in their own experiments.
+The name loosely refers to Jackson Pollock's painterly gesture: nanoGPT is the canvas on which data, configuration, and training decisions create different behavioral patterns. See [`training-history/r002.md`](./training-history/r002.md) for the complete technical record and [`CHANGELOG.md`](./CHANGELOG.md) for release-to-release changes.
 
 ### Architecture and tokenizer
 
-The model is a roughly 96M-parameter GPT-2-small-class decoder with 12 layers, 12 attention heads, a hidden size of 768, a 2048-token context, and a 12,288-token vocabulary. The special tokens are `<|endoftext|>`, `<|im_start|>`, and `<|im_end|>`.
+| Property | Value |
+|---|---:|
+| Revision / release | r002 / Pollock 1.0 |
+| Type | GPT-2-style decoder-only Transformer |
+| Layers / heads / width | 12 / 14 / 896 |
+| Maximum context | 1,024 tokens |
+| Vocabulary | 12,288 tokens |
+| nanoGPT parameters | 126,637,952 |
+| Total unique trainable parameters | 127,555,456 |
+| Tokenizer | byte-level BPE, GPT-2-style pretokenization |
+| Special tokens | `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` |
+
+The Transformers artifact has 127,674,624 parameters, including 119,168 zero-valued compatibility bias parameters required by `GPT2LMHeadModel`. The native model was trained with `bias=False`.
 
 ### Data and training
 
-The model was trained for four epochs (10,815,275,008 tokens seen) on [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b). The 15-source mixture was chosen subjectively using a “common sense” heuristic rather than scientifically optimized proportions. It contains about 2.5B estimated pre-tokenization tokens and 2,703,768,815 final BPE training tokens.
+The model was trained on [`SlayerLab/minimal-en-corpus-2.5b`](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), a subjectively selected mixture of 15 English-language sources. Preparation included exact and approximate deduplication, language filtering, and benchmark decontamination.
 
-Training used one RTX 5090, BF16, fused AdamW, a 262,144-token effective batch, and cosine learning-rate decay from 6e-4 to 6e-5. See the Polish section above for the full configuration.
+| Setting | Value |
+|---|---:|
+| Training / validation tokens | 2,703,768,815 / 5,267,742 |
+| Final checkpoint | iteration 22,004 |
+| Token presentations | 10,815,406,080 (approximately 4 epochs) |
+| Sequence / micro-batch per GPU | 1,024 / 12 |
+| Global / per-GPU accumulation | 40 / 20 micro-steps |
+| Effective batch | 491,520 tokens |
+| Optimizer | fused AdamW, betas 0.9/0.95 |
+| Learning rate | 3e-4 → 3e-5, cosine decay |
+| Warmup / weight decay / grad clip | 440 / 0.1 / 1.0 |
+| Precision | BF16 |
+| Hardware | 2× NVIDIA GeForce RTX 4090 24 GB |
+| Framework | PyTorch 2.8.0+cu128, nanoGPT commit `3adf61e` |
 
 ### Evaluation
 
-The release checkpoint was selected by deterministic evaluation of every retained checkpoint over 5,267,456 validation tokens. It achieved loss **3.40875361** and perplexity **30.227545**. No downstream benchmarks were run. A historical checkpoint-selection bug was discovered, so retained checkpoints were reevaluated instead of trusting the `checkpoint_best_val.pt` filename.
+Training-time loss was estimated from 100 randomly sampled batches per split. The final checkpoint achieved validation loss **2.577547**; the best sampled result was **2.5600** at step 20,000. These values are not directly comparable with r001's full deterministic validation.
 
-### Usage
+Benchmarks used complete splits with `lm-evaluation-harness` 0.4.12, zero-shot evaluation, batch size 8, and BF16.
 
-Use the standard `AutoTokenizer` and `AutoModelForCausalLM` example from the Polish section. No `trust_remote_code` flag is required.
+| Benchmark | Primary metric | Score | Samples |
+|---|---|---:|---:|
+| BLiMP | accuracy | 0.766970 | 67,000 |
+| LAMBADA OpenAI | accuracy | 0.278090 | 5,153 |
+| HellaSwag | normalized accuracy | 0.298745 | 10,042 |
+| PIQA | normalized accuracy | 0.603373 | 1,838 |
+| SciQ | normalized accuracy | 0.658000 | 1,000 |
+| ARC-Easy | normalized accuracy | 0.422980 | 2,376 |
+| ARC-Challenge | normalized accuracy | 0.241468 | 1,172 |
 
-### Example generations
+LAMBADA perplexity was 53.667523. Full metrics and protocol details are recorded in the r002 history.
+
+### Usage
 
-The four samples shown in the Polish section were generated without cherry-picking. The seed was reset to 1337 for each prompt; temperature was 0.7, top-k was 50, and generation was limited to 100 new tokens. The prompt is included in each displayed output.
+Use the Transformers example in the Polish section. The artifact uses standard `GPT2LMHeadModel`; `trust_remote_code=True` is unnecessary. No r002 generation samples have been published yet. Historical samples are explicitly attached to the predecessor weights in the r001 history.
 
 ### Limitations and responsible use
 
-This is an unaligned base model, not a conversational assistant. It can hallucinate, contradict itself, produce biased, toxic, unsafe, or training-like content, and perform poorly outside English. Its small size limits knowledge, reasoning, long-form coherence, and code quality. Do not use it for medical, legal, financial, high-impact, or production decisions. Users must assess upstream dataset terms for their intended use.
+- This is a raw base model without instruction tuning, RLHF, or safety alignment.
+- It may hallucinate and generate toxic, biased, unsafe, or training-like text.
+- Its small size limits knowledge, reasoning, long-form coherence, and code quality.
+- It was trained and evaluated primarily in English.
+- Do not use it for medical, legal, financial, high-impact, or production decisions without independent safeguards.
+- Users must evaluate upstream dataset terms for their intended use.
 
 ## Licencja / License
 
-Kod nanoGPT jest udostępniany na licencji MIT. Korpus treningowy stanowi agregat źródeł o różnych warunkach i nie nadaje dokumentom jednej wspólnej licencji. Z tego powodu model oznaczono na Hugging Face jako `license: other`, a zastosowanie wag powinno uwzględniać warunki właściwych źródeł opisanych w karcie datasetu. Szczegóły znajdują się w [`LICENSE.md`](./LICENSE.md).
+Kod nanoGPT jest na licencji MIT. Korpus łączy źródła o różnych warunkach i nie ma jednej wspólnej licencji, dlatego model oznaczono jako `license: other`. Szczegóły: [`LICENSE.md`](./LICENSE.md) i karta datasetu.
 
-nanoGPT code is distributed under the MIT License. The training corpus aggregates sources governed by different terms and does not assign a single common license to its documents. The model is therefore marked as `license: other` on Hugging Face, and use of the weights should account for the applicable upstream terms documented in the dataset card. See [`LICENSE.md`](./LICENSE.md) for details.
+nanoGPT code is MIT-licensed. The corpus combines sources governed by different terms and has no single common license, so the model is marked `license: other`. See [`LICENSE.md`](./LICENSE.md) and the dataset card.
 
 ## Pochodzenie / Attribution
 
-Model i dataset: Dawid Majewski / SlayerLab. Implementacja treningowa bazuje na [`karpathy/nanoGPT`](https://github.com/karpathy/nanoGPT).
+Model i dataset / Model and dataset: Dawid Majewski / SlayerLab. Trening bazuje na / Training is based on [karpathy/nanoGPT](https://github.com/karpathy/nanoGPT).

config.json

@@ -12,12 +12,13 @@
   "initializer_range": 0.02,
   "layer_norm_epsilon": 1e-05,
   "model_type": "gpt2",
-  "n_ctx": 2048,
-  "n_embd": 768,
-  "n_head": 12,
+  "model_version": "1.0",
+  "n_ctx": 1024,
+  "n_embd": 896,
+  "n_head": 14,
   "n_inner": null,
   "n_layer": 12,
-  "n_positions": 2048,
+  "n_positions": 1024,
   "pad_token_id": 12285,
   "reorder_and_upcast_attn": false,
   "resid_pdrop": 0.0,

generation_config.json

@@ -1,9 +1,10 @@
 {
+  "_from_model_config": true,
   "bos_token_id": 12285,
-  "do_sample": true,
   "eos_token_id": 12285,
+  "output_attentions": false,
+  "output_hidden_states": false,
   "pad_token_id": 12285,
-  "temperature": 0.7,
-  "top_k": 50,
-  "transformers_version": "5.15.1"
+  "transformers_version": "5.15.1",
+  "use_cache": true
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:acaa1531322cce78ddb919700f173d4c6ea371185bea5052a3ae4c30f57b368e
-size 384279168
+oid sha256:2aada5b26abe9ce3b70af393d77be4ed1a799a7205b9c0dd5cfaff8bcaabce68
+size 510713512

release_manifest.json

@@ -1,49 +1,91 @@
 {
-  "schema_version": 1,
+  "schema_version": 2,
+  "revision": 2,
+  "revision_id": "r002",
+  "release": "Pollock 1.0",
   "model_id": "SlayerLab/pollock-mini-lm-125m",
   "source_checkpoint": {
-    "filename": "checkpoint_final.pt",
-    "sha256": "b957195badd87e1980fbe2e91c2774177be7f4d8a569f7156d8a24775404a976",
-    "iteration": 41257,
-    "epoch": 4.00007387761812,
-    "tokens_seen": 10815275008
+    "path_in_training_workspace": "runs/minimal-en-125m-4ep/ckpt.pt",
+    "sha256": "026f54a390036b35792aa8fb131c8b0d394efb6d933753fb0e1257e55a67374b",
+    "iteration": 22004,
+    "tokens_seen": 10815406080,
+    "native_nanogpt_parameters": 126637952,
+    "native_unique_trainable_parameters": 127555456
   },
-  "artifacts": {
-    "README.md": {
-      "bytes": 12057,
-      "sha256": "bc4300a7944d3e9ed2725478458266979a42e496e17a289fbd4737d95149d366"
-    },
-    "LICENSE.md": {
-      "bytes": 2257,
-      "sha256": "46cbe928ed0aa24875f02f774313b27ec9d8abdf41f0c9ef67e0adb4e0614de4"
-    },
-    "assets/pollock-mini-lm-avatar-320.png": {
-      "bytes": 132202,
-      "sha256": "eba4d381d7c750a0dc7e8ae9356cac1b54b07d8ec0c00f0da045fe8e747687aa"
-    },
-    "config.json": {
-      "bytes": 846,
-      "sha256": "683a3c7486296c007a7a36a45430d8e8c5ef7c36e6af887cfa24a0158d171b4d"
-    },
-    "generation_config.json": {
-      "bytes": 172,
-      "sha256": "d8ad5dfa82f75a9fc944e92b39036ed0794caf3088c610159ee5436b56aa883f"
-    },
-    "model.safetensors": {
-      "bytes": 384279168,
-      "sha256": "acaa1531322cce78ddb919700f173d4c6ea371185bea5052a3ae4c30f57b368e"
-    },
-    "special_tokens_map.json": {
-      "bytes": 99,
-      "sha256": "8b2257a17ea997bb038f43b133aefec82344ad2b8abc2b8a02a6c0a994ed624e"
-    },
-    "tokenizer.json": {
-      "bytes": 830547,
-      "sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51"
-    },
-    "tokenizer_config.json": {
-      "bytes": 266,
-      "sha256": "39a075b24f05384c6d759521563d879cab4bb80b0fe94381e6e792966ab4c2ba"
+  "architecture": {
+    "n_layer": 12,
+    "n_head": 14,
+    "n_embd": 896,
+    "block_size": 1024,
+    "vocab_size": 12288,
+    "dropout": 0.0,
+    "bias": false,
+    "tied_word_embeddings": true
+  },
+  "training": {
+    "dataset": "SlayerLab/minimal-en-corpus-2.5b",
+    "init_from": "scratch",
+    "micro_batch_per_gpu": 12,
+    "gradient_accumulation_global": 40,
+    "ddp_world_size": 2,
+    "effective_batch_tokens": 491520,
+    "optimizer": "fused AdamW",
+    "learning_rate": 0.0003,
+    "min_learning_rate": 0.00003,
+    "schedule": "cosine",
+    "warmup_iters": 440,
+    "lr_decay_iters": 22003,
+    "beta1": 0.9,
+    "beta2": 0.95,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "precision": "bfloat16",
+    "compile": true,
+    "backend": "nccl",
+    "seed": 1337,
+    "hardware": "2x NVIDIA GeForce RTX 4090 24 GB",
+    "nanogpt_commit": "3adf61e"
+  },
+  "evaluation": {
+    "final_sampled_validation_loss": 2.577547,
+    "best_sampled_validation_loss": 2.56,
+    "best_sampled_validation_step": 20000,
+    "training_eval_batches": 100,
+    "benchmark_harness": "lm-evaluation-harness 0.4.12",
+    "benchmark_num_fewshot": 0,
+    "benchmark_batch_size": 8,
+    "benchmarks": {
+      "blimp": {"acc": 0.7669701492537313, "samples": 67000},
+      "lambada_openai": {"acc": 0.2780904327576169, "perplexity": 53.66752251060021, "samples": 5153},
+      "hellaswag": {"acc_norm": 0.29874526986656047, "acc": 0.2818163712407887, "samples": 10042},
+      "piqa": {"acc_norm": 0.6033732317736671, "acc": 0.6137105549510338, "samples": 1838},
+      "sciq": {"acc_norm": 0.658, "acc": 0.737, "samples": 1000},
+      "arc_easy": {"acc_norm": 0.42297979797979796, "acc": 0.4659090909090909, "samples": 2376},
+      "arc_challenge": {"acc_norm": 0.24146757679180889, "acc": 0.20819112627986347, "samples": 1172}
     }
+  },
+  "conversion": {
+    "target_class": "GPT2LMHeadModel",
+    "transformers_version": "5.15.1",
+    "unique_serialized_parameters": 127674624,
+    "compatibility_zero_bias_parameters": 119168,
+    "validation_probe_shape": [2, 64],
+    "max_absolute_logit_error": 0.0
+  },
+  "artifacts": {
+    "README.md": {"sha256": "789645020a3864ad81e8a5271118aeed322c9a56039209e715689866d781342f"},
+    "CHANGELOG.md": {"sha256": "379bf0c49f8a8c4a2d40e3ae1775876c37de2e6dac8125385be77742c96529d0"},
+    "training-history/r001.md": {"sha256": "5e3cb7664fa09902369bb650d1f94cc63b1f0bd1c8a7ffea5c1e6c16c8c4c5df"},
+    "training-history/r002.md": {"sha256": "bf4364b27d966fe32676d75c269dfd1e53a4939246a0888c0cce7cd63641952d"},
+    "benchmarks/english.json": {"sha256": "eff82f7d45c82d715c536ac728787ea1eee42515501d4a0544934badab69ce2d"},
+    "LICENSE.md": {"sha256": "46cbe928ed0aa24875f02f774313b27ec9d8abdf41f0c9ef67e0adb4e0614de4"},
+    "config.json": {"sha256": "0ad9e47efb8d2ddf4666f016a286604365faa4ff6ece9654718c3e4daa43c41e"},
+    "generation_config.json": {"sha256": "435beb27be51f0ed054f4a011e5109d125cdadc118b8799b18b155cc798d94d2"},
+    "model.safetensors": {"sha256": "2aada5b26abe9ce3b70af393d77be4ed1a799a7205b9c0dd5cfaff8bcaabce68"},
+    "special_tokens_map.json": {"sha256": "8b2257a17ea997bb038f43b133aefec82344ad2b8abc2b8a02a6c0a994ed624e"},
+    "tokenizer.json": {"sha256": "6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51"},
+    "tokenizer_config.json": {"sha256": "4cdabe37dbdc1adfcc017ee9a1f86ab89bdf184827d2d9f05181cae0f8af19bf"},
+    "logs/training.log": {"sha256": "ff856088bdf504759600553a4e9ad5a23f965d54646838adcf0599bb1621d6e1"},
+    "logs/benchmark-english.log": {"sha256": "f81ad6c915ae49a0d49f61c15ea86a265c31364e5418150c35aa416d4809e22a"}
   }
 }

tokenizer_config.json

@@ -6,7 +6,7 @@
     "<|im_start|>",
     "<|im_end|>"
   ],
-  "model_max_length": 2048,
+  "model_max_length": 1024,
   "pad_token": "<|endoftext|>",
   "tokenizer_class": "TokenizersBackend"
 }

training-history/r001.md

@@ -0,0 +1,99 @@
+# r001 — experimental predecessor training record
+
+## Identity
+
+`r001` is the first published experimental predecessor of Pollock 1.0. All values and samples in this file describe the r001 weights, not the current r002 weights.
+
+## Architecture and tokenizer
+
+| Setting | Value |
+|---|---:|
+| Model type | GPT-2-style decoder-only Transformer |
+| Layers / heads / width | 12 / 12 / 768 |
+| nanoGPT reported parameters | approximately 95.96M |
+| Context length | 2,048 tokens |
+| Vocabulary | 12,288 tokens |
+| Tokenizer | custom byte-level BPE with GPT-2-style pretokenization |
+| Special tokens | `<|endoftext|>`, `<|im_start|>`, `<|im_end|>` |
+
+## Data and training
+
+The model used [Minimal EN Corpus 2.5B](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), the same 15-source English mixture documented by the dataset card. The train split contained 2,703,768,815 final BPE tokens and the validation split 5,267,742 tokens.
+
+| Setting | Value |
+|---|---:|
+| Optimizer steps | 41,257 |
+| Token presentations | 10,815,275,008 (4 passes) |
+| Sequence / micro-batch / accumulation | 2,048 / 32 / 4 |
+| Effective batch | 262,144 tokens |
+| Optimizer | fused AdamW |
+| Adam betas | 0.9 / 0.95 |
+| Maximum / minimum LR | 6e-4 / 6e-5 |
+| Schedule / warmup | cosine / 500 iterations |
+| Weight decay / grad clip | 0.1 / 1.0 |
+| Precision | BF16 |
+| Hardware | 1× NVIDIA GeForce RTX 5090 32 GB |
+| PyTorch / nanoGPT commit | 2.8.0+cu128 / `3adf61e` |
+
+## Evaluation
+
+The release checkpoint was selected after deterministically reevaluating every retained checkpoint over the full validation binary. A historical bug affected the training-time `best_val_loss` bookkeeping, so selection did not rely on the `checkpoint_best_val.pt` filename.
+
+| Dataset | Evaluated tokens | Windows | Loss | Perplexity |
+|---|---:|---:|---:|---:|
+| Minimal EN validation | 5,267,456 | 2,572 non-overlapping × 2,048 | 3.40875361 | 30.227545 |
+
+Evaluation used BF16. No downstream benchmark suite was run for r001. Its validation result is not directly comparable with r002's randomly sampled 100-batch training validation.
+
+## Historical generation samples
+
+These four samples were produced by the **r001 predecessor weights** without cherry-picking. Seed 1337 was reset for each prompt; temperature was 0.7, top-k was 50, and generation was limited to 100 new tokens. Each prompt is included in its output.
+
+### General knowledge
+
+Prompt: `The capital of France is`
+
+```text
+The capital of France is a first of the most important element in the construction of the Socket.
+
+Most of the socket is not a type of sandstone but it is generally a rocket.
+
+As a result, there is also an exception of the rocket site. The socket is called a rocket.
+
+The rocket is the only rocket.
+The rocket is the highest rocket.
+
+The rocket is the highest rocket.
+
+The rocket
+```
+
+### Explanation
+
+Prompt: `Photosynthesis is the process by which`
+
+```text
+Photosynthesis is the process by which the first two events are found. A key option is to do a function in some cases, so it is called the first two terms. This is what I want to think. The first one is to multiply the number by the number of digits. (The second one is to multiply the number by the first two terms. The first two terms are divided by the second one, so that is the first four terms. The third one is to multiply the number by the number of digits. The third
+```
+
+### Story
+
+Prompt: `In a distant future, humanity discovered`
+
+```text
+In a distant future, humanity discovered to change, as well as the "coneurship and a continuance of the body.
+This is not a matter of the general sense, but a lot of things we call us to the body. If we feel the body is the body’s internal structure, we’re going to change the way. The body is the body’s potential structure, meaning that it is used for a body to balance and develop a sensory to control each other.
+What is the
+```
+
+### Code
+
+Prompt: `def fibonacci(n):`
+
+```text
+def fibonacci(n):)) = (n - 1) + (n - 1) + ( n - 1) + ( n - 2) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n - 1) + ( n
+```
+
+## Limitations
+
+r001 is an unaligned English base model with limited knowledge, reasoning, long-form coherence, and code quality. It may hallucinate or produce unsafe, biased, toxic, or training-like content. It is not intended for high-risk or production use. Dataset sources have different upstream terms; see the dataset card and [`../LICENSE.md`](../LICENSE.md).

training-history/r002.md

@@ -0,0 +1,106 @@
+# r002 — Pollock 1.0 training record
+
+## Identity and provenance
+
+| Field | Value |
+|---|---|
+| Revision / release | `r002` / Pollock 1.0 |
+| Model ID | `SlayerLab/pollock-mini-lm-125m` |
+| Source checkpoint | `runs/minimal-en-125m-4ep/ckpt.pt` |
+| Checkpoint iteration / SHA-256 | 22,004 / `026f54a390036b35792aa8fb131c8b0d394efb6d933753fb0e1257e55a67374b` |
+| nanoGPT commit | `3adf61e` |
+| W&B run | [`hlbpqpoe`](https://wandb.ai/dawidmajewski-ezotic/minimal-en/runs/hlbpqpoe) |
+
+Iteration 22,004 is one greater than configured `max_iters=22003` because of nanoGPT's inclusive loop/checkpoint behavior.
+
+## Architecture and parameter accounting
+
+| Setting | Value |
+|---|---:|
+| Type | GPT-2-style decoder-only Transformer |
+| Layers / heads / width | 12 / 14 / 896 |
+| Head dimension / MLP width | 64 / 3,584 |
+| Context / vocabulary | 1,024 / 12,288 |
+| Activation / dropout / bias | GELU / 0.0 / false |
+| Position representation | learned absolute embeddings |
+| Input/output token weights | tied |
+| Transformer excluding embeddings | 115,627,904 |
+| Tied token embedding/output matrix | 11,010,048 |
+| nanoGPT reported count | 126,637,952 |
+| Learned position embedding | 917,504 |
+| Native unique trainable total | 127,555,456 |
+| Transformers serialized total | 127,674,624 |
+| Zero-valued compatibility bias parameters | 119,168 |
+
+Conversion to `GPT2LMHeadModel` produced a maximum absolute logit difference of 0 on a deterministic 2×64-token probe.
+
+## Tokenizer and dataset
+
+The custom tokenizer is byte-level BPE with GPT-2-style pretokenization, 12,288 vocabulary entries, and 12,029 merges. `<|endoftext|>` (12285) is EOD/BOS/EOS/padding; `<|im_start|>` (12286) and `<|im_end|>` (12287) are additional special tokens. No chat template is defined. Tokenizer SHA-256: `6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51`.
+
+Training used [Minimal EN Corpus 2.5B](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), a subjectively selected mixture of 15 English sources with exact and approximate deduplication, language filtering, and benchmark decontamination.
+
+| Split | Documents | BPE tokens | Binary SHA-256 |
+|---|---:|---:|---|
+| Train | 2,044,652 | 2,703,768,815 | `5396834c25664080c14793a4880f86b86ae94074a6036f8f54a58686b02d30dc` |
+| Validation | 4,469 | 5,267,742 | `bf07bd1fb1f956a92ee7788f2a8110e1837f5b880ce432f6cd2a22d05f8ac135` |
+
+## Complete training configuration
+
+| Setting | Value |
+|---|---:|
+| Initialization | scratch |
+| Final checkpoint iteration | 22,004 |
+| Sequence / micro-batch per GPU | 1,024 / 12 |
+| Global / per-GPU accumulation | 40 / 20 micro-steps |
+| DDP world size / effective batch | 2 / 491,520 tokens |
+| Token presentations / passes | 10,815,406,080 / approximately 4.0 |
+| Optimizer / betas | fused AdamW / 0.9, 0.95 |
+| Maximum / minimum LR | 3e-4 / 3e-5 |
+| Schedule / warmup / decay | cosine / 440 / 22,003 iterations |
+| Weight decay / grad clip | 0.1 / 1.0 |
+| Evaluation interval / batches | 1,000 / 100 per split |
+| Precision / compilation / backend | BF16 / `torch.compile` / NCCL |
+| Seed | 1337 plus DDP-rank offset |
+| Hardware | 2× NVIDIA GeForce RTX 4090 24 GB |
+| PyTorch / runtime | 2.8.0+cu128 / approximately 12 h 41 min |
+
+## Training-time validation
+
+Loss estimates used 100 randomly sampled batches and are not a deterministic full-validation pass.
+
+| Step | Train loss | Validation loss |
+|---:|---:|---:|
+| 0 | 9.5797 | 9.5732 |
+| 1,000 | 3.4769 | 3.5219 |
+| 2,000 | 3.0943 | 3.1266 |
+| 5,000 | 2.8030 | 2.8569 |
+| 10,000 | 2.6098 | 2.6928 |
+| 15,000 | 2.5469 | 2.6056 |
+| 18,000 | 2.5104 | 2.5790 |
+| 19,000 | 2.4960 | 2.5795 |
+| 20,000 | 2.4705 | **2.5600** |
+| 21,000 | 2.4877 | 2.5653 |
+| 22,000 | **2.4545** | 2.5775 |
+
+The released weights are the final checkpoint. The increase after step 20,000 is within a noisy sampled protocol and is not definitive evidence of overfitting.
+
+## English zero-shot benchmarks
+
+All tasks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, BF16, 1,024 maximum context, and no sample limit. Random/NumPy/PyTorch/few-shot seeds were 0/1234/1234/1234. No requests were truncated.
+
+| Benchmark | Primary | Additional | Samples |
+|---|---:|---:|---:|
+| BLiMP | acc 0.7669701493 | — | 67,000 |
+| LAMBADA OpenAI | acc 0.2780904328 | perplexity 53.6675225106 | 5,153 |
+| HellaSwag | acc_norm 0.2987452699 | acc 0.2818163712 | 10,042 |
+| PIQA | acc_norm 0.6033732318 | acc 0.6137105550 | 1,838 |
+| SciQ | acc_norm 0.6580000000 | acc 0.7370000000 | 1,000 |
+| ARC-Easy | acc_norm 0.4229797980 | acc 0.4659090909 | 2,376 |
+| ARC-Challenge | acc_norm 0.2414675768 | acc 0.2081911263 | 1,172 |
+
+Raw logs: [`../logs/training.log`](../logs/training.log) and [`../logs/benchmark-english.log`](../logs/benchmark-english.log). Structured results: [`../benchmarks/english.json`](../benchmarks/english.json). Release metadata: [`../release_manifest.json`](../release_manifest.json).
+
+## Licensing and limitations
+
+This is an English, unaligned base model, not an instruction-following assistant. It may hallucinate, generate harmful or biased content, and reproduce training patterns. It is unsuitable for high-risk or production decisions without independent safeguards. The dataset combines sources with different terms; see [`../LICENSE.md`](../LICENSE.md) and the dataset card.