zarko1321/Bitsy500M

Bitsy-500M Custom

Randomly initialized and untrained. Zero language-model training updates.

A complete model folder with actual BF16 safetensors weights, a new byte-level BPE tokenizer, independently written model code, a pretraining runner, and a Colab notebook. It is not a trained chatbot or coding assistant.

Architecture

Property	Value
Unique parameters	509,943,936 (tied embeddings counted once)
Decoder blocks	27
Hidden width	1,280
SwiGLU intermediate width	3,456
Query heads / KV heads	20 / 5
Head width	64
Vocabulary	32,000
Configured context limit	2,048 tokens
Initial weights	BF16, random seed 42

Bitsy Parallel is an independent PyTorch implementation using parallel residual attention and feed-forward branches, separate pre-RMSNorms, learned query/key RMSNorm, adjacent-pair RoPE, GQA, PyTorch SDPA, SwiGLU, and tied input/output embeddings. It does not subclass or wrap a Llama/GPT/Qwen model. These are established building blocks; this package does not claim a new research invention.

The tokenizer and weights are newly initialized for this model. Existing Bitsy 1B weights and cached token IDs cannot be substituted into this model.

Download and use

Extract all three ZIPs into the same location. Allow the Bitsy-500M-Custom folders to merge. Keep all three weight shards and the weight index together. This replacement Part 1 works with the existing Parts 2 and 3. To publish on Hugging Face, create a model repository and upload the files inside this folder to its root. Keep both custom Python model files. Nothing was uploaded to your Hugging Face account automatically.

Install in a fresh Python environment:

pip install -r requirements.txt

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

path = "./Bitsy-500M-Custom"  # or your Hugging Face repository ID
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(
    path, trust_remote_code=True, torch_dtype=torch.bfloat16
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device).eval()
inputs = tokenizer("One day, a little girl found", return_tensors="pt").to(device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Random output is expected until you train it. trust_remote_code=True executes the included custom model code. Review that code before loading. The dependency versions in requirements.txt are the versions used for validation. Unlisted newer Transformers versions have not been validated.

Training on Colab

Open Train_Bitsy_500M_Colab.ipynb in Colab, select a GPU runtime, and run the three cells. Upload the model folder to your Hugging Face repository first. The notebook downloads your model, streams TinyStories, and calls the included runner. It does not need Google Drive. For a private repository, sign into Hugging Face in Colab without putting access tokens in the notebook source.

The code targets a BF16 GPU such as the L4. Actual L4 throughput and VRAM use have not been measured here; start with batch size 1 and context 512. It uses FP32 training parameters, BF16 autocast, AdamW8bit on CUDA, gradient checkpointing, and gradient clipping. The BF16 download is an initial model export, not a low-precision optimizer checkpoint.

Alternatively:

pip install datasets bitsandbytes
python train.py --model ./Bitsy-500M-Custom --output ./bitsy-run-01 \
  --context 512 --batch-size 1 --accum 8 --steps 250

The default run streams roneneldan/TinyStories and evaluates only 16 fixed validation blocks every 100 updates. A small test run is not enough to teach a 500M model fluent English. Monitor validation and sample text as training grows.

For local data, add --data stories.jsonl, with one {"text":"full story"} object per line. UTF-8 .txt is supported too; blank lines separate documents. Local-data runs do not silently borrow a validation dataset: supply evaluation separately in your workflow. The runner performs pretraining, not chat SFT.

Text is packed into fixed-length blocks, with EOS between documents. Padding is not inserted into these blocks. Targets are shifted exactly once in the model; callers supply unshifted labels. The model supports -100 to mask prompt/padding targets for a future chat trainer. Do not train pad positions as real text.

Checkpoints and stages

Each saved training checkpoint contains model/tokenizer files and training_state.pt with optimizer, RNG, completed update, and data cursor. Checkpoint directories are committed only after writing finishes. Keep enough disk space for multiple checkpoints (FP32 weights plus optimizer state).

To resume an interrupted run with the same data and schedule:

python train.py --model ./Bitsy-500M-Custom --output ./bitsy-resumed \
  --resume ./bitsy-run-01/checkpoint-100 \
  --context 512 --batch-size 1 --accum 8 --steps 250

The output folder must be new. All signature settings must match the original run. Resuming skips already consumed blocks; keep the upstream dataset unchanged. The streaming remote dataset is not archived or pinned by this runner, so exact reproduction also depends on unchanged upstream data and runtime libraries. Only load optimizer state files you trust; they use Python pickle.

For a new training stage, use the previous checkpoint as --model, choose a new output folder, and omit --resume. This preserves model weights but starts a new optimizer/schedule/data cursor. Keep the previous checkpoint. An exported HF model without training_state.pt cannot reproduce an exact optimizer resume. The runner retains all checkpoints; manage disk space yourself.

Tokenizer and chat format

Token	ID
<|bos|>	0
<|eos|>	1
<|pad|>	2
<|unk|>	3
<|start|>	4
<|end|>	5
<|system|>	6
<|user|>	7
<|assistant|>	8

The tokenizer adds no automatic BOS/EOS to raw text. The runner adds document EOS explicitly; apply_chat_template handles chat formatting. Messages use start/role/newline/content/end; completed assistant messages also receive EOS. The saved generation config stops on EOS or the end-of-message token. The template supports text system/user/assistant roles, not tool calls or images. It has no assistant-token-mask generation blocks: a future SFT pipeline must implement and verify its target mask explicitly. A chat template alone teaches no conversational skills.

Tokenizer training used about 5.1 MB of seven older English novels, MIT-licensed Lua C source, and original small Python/Lua-style examples. See tokenizer_sources.json for exact URLs, hashes, and usage. This is a modest, literary-biased tokenizer corpus, not a broad modern English/code corpus. Byte coverage preserves Unicode and code whitespace, but code token efficiency is not benchmarked. Tokenizer training performed no model-weight training.

The Gutenberg material used original English texts by authors who died more than 70 years ago; headers and trailers were excluded. The public-domain texts are not distributed here. Lua source is not distributed here either; its license and source reference are retained in THIRD_PARTY_NOTICES.md.

Longer context

The implementation enforces a 2,048-token context limit. It has rotary positions and no learned position-embedding table, but that does not guarantee extrapolation. Future 4K/8K/16K work needs a deliberate RoPE strategy, longer training sequences, memory benchmarking, and held-out retrieval/continuation tests. No 16K capability or extension recipe has been validated.

Compatibility and test limits

Hugging Face custom AutoConfig/AutoModelForCausalLM loading is supported with trust_remote_code=True. See validation_report.json for checks actually run. CPU testing covers causality, cached/full attention equivalence, left-padding, beam generation, loss alignment/masking, a tiny-model overfit, full-size save/reload equality, and full-size BF16 backward gradients.

Unsloth Studio integration was not tested and is not promised. Its model registry and optimized kernels may not accept this custom architecture. Use the included PyTorch/Transformers runner instead of renaming the architecture to Llama or assuming that a tokenizer fix adds model support. CUDA attention, long-sequence memory use, and Colab execution have not been validated here. The notebook is provided as an executable workflow, not a claimed completed GPU test. There are no language, coding, or reasoning benchmark claims.

License

Original package code and newly generated weights/tokenizer are provided under MIT. External data sources keep their own terms. The model is provided without warranty. See LICENSE and THIRD_PARTY_NOTICES.md.