Upload Boopit 1.58-bit packed weights

README.md

@@ -4,13 +4,6 @@ language:
 - en
 library_name: pytorch
 pipeline_tag: text-generation
-datasets:
-- openbmb/Ultra-FineWeb-L1
-tags:
-- bitnet
-- ternary
-- 1.58-bit
-- boopit
 
 ---
 
@@ -18,11 +11,16 @@ This model was asked to be published under my account, not the creators. The com
 
 # Boopit 1
 
-A **27.3M**-parameter language model with a **4096** context window, trained from scratch with **BitNet b1.58 ternary weights** (weights in {-1,0,1} from step 0) on [openbmb/Ultra-FineWeb-L1](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1).
+A **27.3M**-parameter language model with a **4096** context window and native **BitNet b1.58 ternary weights** (weights in {-1, 0, 1} from step 0).
 
-Packed weights (`model.boopit`) are **under 7MB**.
+This release contains the best held-out-loss checkpoint from pretraining on [openbmb/Ultra-FineWeb-L1](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1):
 
-Stopped at the **best val** checkpoint (step 3400, val 4.480278) rather than the last step.
+- Selected checkpoint: step **3,400**
+- Tokens seen by this checkpoint: **891,289,600**
+- Validation loss: **4.480278**
 
-Chat fine-tune: [ProCreations/boopit-1-chat](https://huggingface.co/ProCreations/boopit-1-chat).
+Training continued beyond this checkpoint, but later checkpoints did not improve held-out loss, so this best checkpoint was preserved and selected for release. The packed weights (`model.boopit`) are under 7 MB.
+
+Architecture: 6 layers, width 512, 8 attention heads, tied embeddings, RMSNorm, GELU MLP, and a byte-level 16,384-token BPE vocabulary.
 
+Chat fine-tune: [ProCreations/boopit-1-chat](https://huggingface.co/ProCreations/boopit-1-chat).

model.boopit

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:24631dd6774ef3cf7cea0a552e5023487d69ad3497863ad130cc6c801a87684c
-size 5470400
+oid sha256:f3a2770f48d55e229b2f9b4555712662d8a9edae78aff6240fa30baaaff73210
+size 5470440