| # Eather-500M and Unsloth Studio (Arch Linux / ROCm) |
|
|
| ## Important distinction |
|
|
| The original `model.py` is a custom PyTorch architecture and its checkpoints are **not** recognized by Unsloth Studio just by supplying `tokenizer.json`. The export script builds a **new randomly initialized, standard Hugging Face Llama-format model** with matching key dimensions. It **does not convert existing trained original Eather checkpoints**. RoPE conventions and module naming differ; do not load raw custom checkpoint tensors as Llama weights. |
|
|
| Studio's training support for a particular local random-weight model, full-parameter pretraining on ROCm, or importing a local path is version-dependent. Producing a valid Transformers model directory does **not guarantee** Studio will accept it or train it. Check compatibility in the installed Studio version. For guaranteed control over custom model training, use `train.py` from the original project. |
|
|
| ## 1. Install dependencies in your dedicated Python environment |
|
|
| ```bash |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install torch transformers tokenizers safetensors sentencepiece |
| ``` |
|
|
| On Arch/AMD ROCm, **do not replace an already working ROCm PyTorch build with a CPU-only build**; follow your ROCm PyTorch setup for the training environment. For the export script, CPU PyTorch is sufficient. |
|
|
| ## 2. Train or reuse the matching 32k tokenizer |
|
|
| ```bash |
| python tokenizer_train.py --input data/raw --output tokenizer.json |
| ``` |
|
|
| Your source text must be large enough to create exactly 32,000 tokens. If you already generated the **matching** tokenizer, reuse it. Do not switch tokenizers for a pretrained checkpoint. |
|
|
| ## 3. Export HF-compatible tokenizer and new model weights |
|
|
| ```bash |
| python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-500M-Unsloth |
| ``` |
|
|
| It creates `config.json`, `generation_config.json` (if saved by Transformers), `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, and sharded `model-*.safetensors` weights. The model is randomly initialized; it is not pretrained. |
|
|
| To quickly validate the integration without allocating a full 500M model: |
|
|
| ```bash |
| python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-smoke --smoke-test |
| ``` |
|
|
| ## 4. Validate Hugging Face loading |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| folder = './Eather-500M-Unsloth' |
| tok = AutoTokenizer.from_pretrained(folder, use_fast=True) |
| model = AutoModelForCausalLM.from_pretrained(folder) |
| print(type(model).__name__, len(tok), sum(p.numel() for p in model.parameters())) |
| ``` |
|
|
| ## 5. Try Unsloth Studio |
|
|
| ```bash |
| unsloth studio update |
| unsloth studio -p 8888 |
| ``` |
|
|
| If your installed Studio accepts local model directories, choose `Eather-500M-Unsloth`. The exact UI varies. If Studio rejects that path or model, **do not keep reinstalling sentencepiece**: the Hugging Face folder can already be valid while Studio's train workflow does not support loading/initializing a scratch model. Use an Unsloth-supported model architecture/workflow or train with `train.py` and add Transformers-compatible export later. An untrained scratch model cannot be made capable by LoRA on a few examples. |
|
|
| ## Troubleshooting |
|
|
| - `Couldn't instantiate backend tokenizer`: `tokenizer.json` is missing, malformed, or selected model folder is not the output directory. Confirm `AutoTokenizer.from_pretrained('./Eather-500M-Unsloth')` works. |
| - `Unrecognized model in ...`: use the output folder, not the original custom `config.json`. |
| - `Out of memory`: 500M full-parameter training on a 16GB card is tight, especially with AdamW and activation memory; lower context/microbatch and use gradient checkpointing. Exporting is easier than training. |
| - `Missing tokenizer`: you must supply a tokenizer **trained on actual source data**; the project ZIP does not include a pretrained tokenizer or pretrained weights. |
|
|
