---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- byte-level
- cascade
- reasoning
- pretraining
library_name: pytorch
---

# Cascade-1B-Logic

Public experimental byte-level causal language model. Training is in progress and checkpoints are uploaded continuously.

## Architecture

- Approximately 1B stored parameters and approximately one third active per token.
- UTF-8 bytes, vocabulary 256, no external tokenizer.
- Four local causal Transformer layers process every byte.
- Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence. Over-cap boundaries are uniformly coalesced across the sequence instead of truncating its tail.
- Three independent eight-layer global routes are trained cyclically, one full batch per route.
- The previous completed global patch conditions the local prediction, preventing future-token leakage.
- Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
- BF16 training on one NVIDIA RTX PRO 6000 Blackwell, with batch size selected by a live memory/throughput probe.

Default context is 1024 bytes. This is a research checkpoint, not an instruction model.

## Data mixture

The stream is not redistributed in this repository:

- 58% [FineWeb-Edu sample-10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- 22% [FineMath finemath-4plus](https://huggingface.co/datasets/HuggingFaceTB/finemath)
- 10% [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)
- 7% [OpenR1-Math-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k)
- 3% [decontaminated Codeforces reasoning traces](https://huggingface.co/datasets/open-r1/codeforces-cots)

Review each upstream dataset card and license before redistribution or commercial use.

## Checkpoints

`checkpoints/step-XXXXXXXX/model.safetensors` contains BF16 model weights. `training_state.json` reports live loss, throughput, token count, batch size, routing and patch statistics.

## Evidence and limits

The precursor 998.7M-parameter prototype reached validation byte loss 1.235 versus 1.360 for a 988.1M dense baseline on the same TinyStories test. It used about 4.15x fewer estimated model FLOPs and ran 2.06x faster wall-clock. This short controlled result motivated full training; it is not evidence of 100x speedup, broad reasoning ability or production readiness.

Training code and architecture are included for auditability. Results may change while training is active.