File size: 2,537 Bytes
579d2fe d23602a 579d2fe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | ---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- byte-level
- cascade
- reasoning
- pretraining
library_name: pytorch
---
# Cascade-1B-Logic
Public experimental byte-level causal language model. Training is in progress and checkpoints are uploaded continuously.
## Architecture
- Approximately 1B stored parameters and approximately one third active per token.
- UTF-8 bytes, vocabulary 256, no external tokenizer.
- Four local causal Transformer layers process every byte.
- Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence. Over-cap boundaries are uniformly coalesced across the sequence instead of truncating its tail.
- Three independent eight-layer global routes are trained cyclically, one full batch per route.
- The previous completed global patch conditions the local prediction, preventing future-token leakage.
- Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
- BF16 training on one NVIDIA RTX PRO 6000 Blackwell, with batch size selected by a live memory/throughput probe.
Default context is 1024 bytes. This is a research checkpoint, not an instruction model.
## Data mixture
The stream is not redistributed in this repository:
- 58% [FineWeb-Edu sample-10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- 22% [FineMath finemath-4plus](https://huggingface.co/datasets/HuggingFaceTB/finemath)
- 10% [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)
- 7% [OpenR1-Math-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k)
- 3% [decontaminated Codeforces reasoning traces](https://huggingface.co/datasets/open-r1/codeforces-cots)
Review each upstream dataset card and license before redistribution or commercial use.
## Checkpoints
`checkpoints/step-XXXXXXXX/model.safetensors` contains BF16 model weights. `training_state.json` reports live loss, throughput, token count, batch size, routing and patch statistics.
## Evidence and limits
The precursor 998.7M-parameter prototype reached validation byte loss 1.235 versus 1.360 for a 988.1M dense baseline on the same TinyStories test. It used about 4.15x fewer estimated model FLOPs and ran 2.06x faster wall-clock. This short controlled result motivated full training; it is not evidence of 100x speedup, broad reasoning ability or production readiness.
Training code and architecture are included for auditability. Results may change while training is active.
|