Upload folder using huggingface_hub

README.md

@@ -0,0 +1,104 @@
+---
+license: mit
+language:
+  - code
+tags:
+  - code-generation
+  - python
+  - from-scratch
+  - pretrained
+library_name: pytorch
+---
+
+# code-llm-435m — a Python code-completion model trained from scratch on one RTX 5090
+
+A 435M-parameter decoder-only Transformer for Python code completion. Data pipeline, tokenizer,
+architecture, training loop, checkpoint merging and evaluation were all written and run by one person
+on one consumer GPU — this repository holds the weights; the code, design record, ablation report and
+failure log live in the GitHub repository linked below.
+
+> **This folder is the pretrained base.** Its weights are the *merged* (weight-averaged) product of the
+> pretraining run: under this training scheme the merged checkpoint, not the last step, is the finished
+> pretrained model — hence the folder name. The instruction-tuned variant is in [`../sft`](../sft).
+
+## Architecture
+
+| | |
+|---|---|
+| Parameters | **434,680,832** (bf16) |
+| Layers | 22 |
+| d_model / heads | 1024 / 16 (head_dim 64) |
+| FFN | SwiGLU, d_ff 4096 |
+| Position | RoPE, theta 500000 |
+| Norm | RMSNorm (pre-norm) |
+| Attention | causal SDPA (no biases anywhere) |
+| Embeddings | untied (input embed + output head) |
+| Vocabulary | 32,000 byte-level BPE, trained on the filtered corpus (≈3.5 chars/token) |
+| Context | 1024 tokens |
+| Precision | bf16 (released file) |
+
+## Training
+
+- **Data**: three Python sources (codeparrot / the_stack / star_coder), six-layer quality filtering,
+  copyright filtering, metadata stripping; concatenated into **one continuous 31.0 B-token stream**
+  (no dataset boundaries, no optimizer resets — an earlier version was silently retrained on the same
+  8 B tokens twice by a resume bug, and the design change makes that class of bug impossible).
+- **Schedule**: WSM — constant learning rate, then a 30,000-step cooldown, then **weighted averaging
+  of the last 10,000 checkpoints** (this file is that merged model, not the last step).
+- **Hardware**: a single RTX 5090, 32 GB, Blackwell sm_120.
+
+## What it can and cannot do
+
+**Can**: complete short Python functions when given a signature and the opening indentation.
+On a human-graded benchmark (10 docstring-free tasks × 5 seeds, scored 0/1/2, max 100) this model
+scores **66/100**; the earlier 353M version — same architecture, same GPU — scores **6/100**. The
+difference is the data pipeline, not the architecture. Sample, verbatim (temperature 0.2, seed 0;
+this is a partial-credit example, not a showcase):
+
+```python
+def quicksort(arr):
+    if len(arr) <= 1:
+        return arr
+    else:
+        pivot = arr[0]
+        left = [x for x in arr if x < pivot]
+        right = [x for x in arr if x == pivot]      # <- wrong: should be > pivot
+        return quicksort(left) + [pivot] + quicksort(right)
+```
+
+The recursion, the base case and the partition are there; one comparison operator is wrong, so the
+function drops elements. That is what "66/100" looks like at this scale — the structure is learned
+before the detail is, which is exactly why the benchmark is human-graded rather than pass/fail.
+
+**Cannot**: follow instructions — this is a completion model, not a chat model (supervised
+fine-tuning is documented separately in the GitHub repo). It is blind to docstrings: at this scale a
+from-scratch model reads a docstring as the end of the function, so standard HumanEval
+docstring prompts score ≈0 and docstring-free prompts are used instead. At 18 B tokens the same
+pipeline still scored 0/50 on a five-algorithm suite — implementation ability appears between
+18 B and 31 B tokens, which the ablation report documents rather than hides.
+
+## Files
+
+| File | What |
+|---|---|
+| `model.safetensors` | bf16 weights, 157 tensors — verified bit-identical to the training checkpoint after the fp32→bf16 cast |
+| `config.json` | architecture config exactly as stored in the training checkpoint |
+| `tokenizer.json` | byte-level BPE, 32,000 tokens |
+| `SHA256SUMS.txt` | artifact hash |
+
+Loading it requires the model class from the training repository (`src/train.py`, class `CodeLLM`
+with `ModelConfig(**config.json)`); `torch.load` of a state dict built by hand will not reproduce the
+forward pass described above.
+
+## Intended use and limits
+
+Research and education: understanding what a few-hundred-million-parameter model actually learns when
+trained end to end on real data. Not for production code generation, not for instruction following,
+not a substitute for a competent developer. Trained only on code; no personal data. Outputs may
+reproduce patterns (and licensing quirks) from the training corpus despite copyright filtering.
+
+## Links
+
+Everything else — `DESIGN.md` (decisions and why), `CHALLENGES.md` (every bug and contaminant),
+the ablation report, the fine-tuning log with corrections, and every evaluated generation — is in the
+GitHub repository. MIT licensed.

SHA256SUMS.txt

@@ -0,0 +1 @@
+d2c030b0fbd8c9a6a227b96e704a8c538c3754f468b2ebd2d9970f880b8c7b0d  model.safetensors

arch_facts.json

@@ -0,0 +1,29 @@
+{
+ "params_total": 434680832,
+ "n_tensors": 157,
+ "dtype": "bf16",
+ "vocab_size": null,
+ "d_model": null,
+ "n_blocks": 22,
+ "tensor_keys_sample": [
+  "blocks.0.attn.out_proj.weight",
+  "blocks.0.attn.qkv.weight",
+  "blocks.0.attn_norm.scale",
+  "blocks.0.ffn.down.weight",
+  "blocks.0.ffn.gate.weight",
+  "blocks.0.ffn.up.weight",
+  "blocks.0.ffn_norm.scale",
+  "blocks.1.attn.out_proj.weight"
+ ],
+ "source_checkpoint": "the pretrained checkpoint this export was taken from (path omitted in the released copy)",
+ "source_keys": [
+  "optimizer",
+  "scheduler",
+  "config",
+  "training_state",
+  "timestamp",
+  "model"
+ ],
+ "step": null,
+ "created_utc": "2026-09-20T18:14:35Z"
+}

config.json

@@ -0,0 +1,10 @@
+{
+  "vocab_size": 32000,
+  "d_model": 1024,
+  "num_layers": 22,
+  "num_heads": 16,
+  "d_ff": 4096,
+  "max_seq_len": 1024,
+  "rope_theta": 500000.0,
+  "dropout_rate": 0.0
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2c030b0fbd8c9a6a227b96e704a8c538c3754f468b2ebd2d9970f880b8c7b0d
+size 869377400