zarko1321/bitsy21321

Eather-500M — custom language model from scratch

A real, randomly initialized ~505M-parameter decoder-only language model, not pretrained weights. Includes tokenizer training, dataset conversion, pretraining, chat SFT, and inference.

Architecture

• 26 transformer blocks; 1280 hidden width; 20 query heads / 5 KV heads (GQA)
• 3584 SwiGLU intermediate width; RoPE; RMSNorm; Flash/SDPA causal attention
• 32,000-token byte-level BPE vocabulary; tied embedding/output weights
• 2,048-token initial context. Context extension requires further training/testing.
• PyTorch implementation; compatible with ROCm PyTorch when properly installed

Install (Linux)

Use Python 3.11+ in an environment with an appropriate GPU-enabled PyTorch build. On AMD RX 7800 XT, install the ROCm PyTorch build appropriate to your exact ROCm, GPU, and OS per official documentation first. Generic pip install torch does not guarantee ROCm support.

python -m venv .venv
source .venv/bin/activate
# Install your appropriate ROCm PyTorch build first.
pip install numpy tokenizers
python model.py --device meta
python -c "import torch; print('GPU available:',torch.cuda.is_available()); print('Backend:',torch.version.hip)"

The full requirements.txt is a conventional generic environment requirement list; it does not configure ROCm.

1. Train a tokenizer

Put licensed or self-created .txt files in data/raw/ (a substantial corpus is necessary).

python tokenizer_train.py --input data/raw --output tokenizer.json

Tokenizer training requires enough text to reach 32,000 learned tokens. For very small toy data, either add more text or lower both --vocab-size and config.json vocab_size to the same value.

2. Pre-tokenize the text

python prepare.py --input data/raw --tokenizer tokenizer.json --output-dir data/pretrain --format text

prepare.py creates train.bin (uint32 token IDs) and mask.bin (uint8 loss mask), reading text in chunks rather than holding the entire corpus in RAM.

3. Pretrain from zero

python train.py --data-dir data/pretrain --out-dir runs/pretrain \
  --seq-len 256 --batch-size 1 --grad-accum 16 \
  --steps 20000 --lr 0.0003 --save-every 1000

Effective tokens/update = seq-len * batch-size * grad-accum. On 16 GB VRAM, start conservatively at 128–256 sequence length and 1 microbatch; gradient checkpointing defaults ON. Actual feasibility and speed depend on ROCm/PyTorch versions, available VRAM, and kernel support. If you get OOM, reduce sequence length and ensure other GPU apps are closed. Optimizer checkpoints are large. The 20,000 steps above are merely an example, not a promise of an intelligent model.

4. Chat fine-tuning

Training data format, one JSON object per line in data/chat.jsonl:

{"messages":[{"role":"user","content":"Hi"},{"role":"assistant","content":"Hello!"}]}

python prepare.py --input data/chat.jsonl --tokenizer tokenizer.json --output-dir data/chat --format chat
python train.py --data-dir data/chat --out-dir runs/chat \
  --resume runs/pretrain/step_0020000.pt --fine-tune \
  --seq-len 256 --batch-size 1 --grad-accum 16 \
  --steps 2000 --lr 0.00002 --save-every 500

Assistant tokens (plus the assistant end-of-turn token) receive SFT loss; user/system tokens are masked. Avoid mixing unrelated conversations without clear boundaries in an SFT dataset; this minimal streaming approach concatenates conversations and can occasionally sample across boundaries.

5. Generate

python generate.py --checkpoint runs/chat/step_0002000.pt --tokenizer tokenizer.json \
  --prompt "Write a Python function to reverse a string" --chat

Or for raw pretraining checkpoints:

python generate.py --checkpoint runs/pretrain/step_0020000.pt --prompt "Once upon a time"

Important limitations

• All weights start random; this download is code, not a trained chatbot. Reaching useful coding and conversational ability needs high-quality data, substantial compute, evaluation, and repeated tuning.
• A 500M model is fairly small for advanced coding. Start with broad text pretraining, then carefully curated Python/Luau and instruction/chat examples.
• This project is an educational/reference trainer, not a production-scale distributed training stack. It does not include KV caching, packing-aware attention, validation/perplexity curves, distributed training, quantization, or model exports.
• Increasing max_seq_len alone does not grant reliable longer context.
• Only train on material you have the rights to use.

Unsloth Studio / Hugging Face compatibility

See UNSLOTH_STUDIO.md and run export_for_unsloth.py to create a fresh randomly initialized standard Llama-format model using your trained 32k Eather tokenizer. This is not a conversion of custom weights; Unsloth Studio training support still depends on your installed version.