Eather-500M and Unsloth Studio (Arch Linux / ROCm)
Important distinction
The original model.py is a custom PyTorch architecture and its checkpoints are not recognized by Unsloth Studio just by supplying tokenizer.json. The export script builds a new randomly initialized, standard Hugging Face Llama-format model with matching key dimensions. It does not convert existing trained original Eather checkpoints. RoPE conventions and module naming differ; do not load raw custom checkpoint tensors as Llama weights.
Studio's training support for a particular local random-weight model, full-parameter pretraining on ROCm, or importing a local path is version-dependent. Producing a valid Transformers model directory does not guarantee Studio will accept it or train it. Check compatibility in the installed Studio version. For guaranteed control over custom model training, use train.py from the original project.
1. Install dependencies in your dedicated Python environment
python -m venv .venv
source .venv/bin/activate
pip install torch transformers tokenizers safetensors sentencepiece
On Arch/AMD ROCm, do not replace an already working ROCm PyTorch build with a CPU-only build; follow your ROCm PyTorch setup for the training environment. For the export script, CPU PyTorch is sufficient.
2. Train or reuse the matching 32k tokenizer
python tokenizer_train.py --input data/raw --output tokenizer.json
Your source text must be large enough to create exactly 32,000 tokens. If you already generated the matching tokenizer, reuse it. Do not switch tokenizers for a pretrained checkpoint.
3. Export HF-compatible tokenizer and new model weights
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-500M-Unsloth
It creates config.json, generation_config.json (if saved by Transformers), tokenizer.json, tokenizer_config.json, special_tokens_map.json, and sharded model-*.safetensors weights. The model is randomly initialized; it is not pretrained.
To quickly validate the integration without allocating a full 500M model:
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-smoke --smoke-test
4. Validate Hugging Face loading
from transformers import AutoTokenizer, AutoModelForCausalLM
folder = './Eather-500M-Unsloth'
tok = AutoTokenizer.from_pretrained(folder, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(folder)
print(type(model).__name__, len(tok), sum(p.numel() for p in model.parameters()))
5. Try Unsloth Studio
unsloth studio update
unsloth studio -p 8888
If your installed Studio accepts local model directories, choose Eather-500M-Unsloth. The exact UI varies. If Studio rejects that path or model, do not keep reinstalling sentencepiece: the Hugging Face folder can already be valid while Studio's train workflow does not support loading/initializing a scratch model. Use an Unsloth-supported model architecture/workflow or train with train.py and add Transformers-compatible export later. An untrained scratch model cannot be made capable by LoRA on a few examples.
Troubleshooting
Couldn't instantiate backend tokenizer:tokenizer.jsonis missing, malformed, or selected model folder is not the output directory. ConfirmAutoTokenizer.from_pretrained('./Eather-500M-Unsloth')works.Unrecognized model in ...: use the output folder, not the original customconfig.json.Out of memory: 500M full-parameter training on a 16GB card is tight, especially with AdamW and activation memory; lower context/microbatch and use gradient checkpointing. Exporting is easier than training.Missing tokenizer: you must supply a tokenizer trained on actual source data; the project ZIP does not include a pretrained tokenizer or pretrained weights.