File size: 3,895 Bytes
7da1a03 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | # Eather-500M and Unsloth Studio (Arch Linux / ROCm)
## Important distinction
The original `model.py` is a custom PyTorch architecture and its checkpoints are **not** recognized by Unsloth Studio just by supplying `tokenizer.json`. The export script builds a **new randomly initialized, standard Hugging Face Llama-format model** with matching key dimensions. It **does not convert existing trained original Eather checkpoints**. RoPE conventions and module naming differ; do not load raw custom checkpoint tensors as Llama weights.
Studio's training support for a particular local random-weight model, full-parameter pretraining on ROCm, or importing a local path is version-dependent. Producing a valid Transformers model directory does **not guarantee** Studio will accept it or train it. Check compatibility in the installed Studio version. For guaranteed control over custom model training, use `train.py` from the original project.
## 1. Install dependencies in your dedicated Python environment
```bash
python -m venv .venv
source .venv/bin/activate
pip install torch transformers tokenizers safetensors sentencepiece
```
On Arch/AMD ROCm, **do not replace an already working ROCm PyTorch build with a CPU-only build**; follow your ROCm PyTorch setup for the training environment. For the export script, CPU PyTorch is sufficient.
## 2. Train or reuse the matching 32k tokenizer
```bash
python tokenizer_train.py --input data/raw --output tokenizer.json
```
Your source text must be large enough to create exactly 32,000 tokens. If you already generated the **matching** tokenizer, reuse it. Do not switch tokenizers for a pretrained checkpoint.
## 3. Export HF-compatible tokenizer and new model weights
```bash
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-500M-Unsloth
```
It creates `config.json`, `generation_config.json` (if saved by Transformers), `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, and sharded `model-*.safetensors` weights. The model is randomly initialized; it is not pretrained.
To quickly validate the integration without allocating a full 500M model:
```bash
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-smoke --smoke-test
```
## 4. Validate Hugging Face loading
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
folder = './Eather-500M-Unsloth'
tok = AutoTokenizer.from_pretrained(folder, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(folder)
print(type(model).__name__, len(tok), sum(p.numel() for p in model.parameters()))
```
## 5. Try Unsloth Studio
```bash
unsloth studio update
unsloth studio -p 8888
```
If your installed Studio accepts local model directories, choose `Eather-500M-Unsloth`. The exact UI varies. If Studio rejects that path or model, **do not keep reinstalling sentencepiece**: the Hugging Face folder can already be valid while Studio's train workflow does not support loading/initializing a scratch model. Use an Unsloth-supported model architecture/workflow or train with `train.py` and add Transformers-compatible export later. An untrained scratch model cannot be made capable by LoRA on a few examples.
## Troubleshooting
- `Couldn't instantiate backend tokenizer`: `tokenizer.json` is missing, malformed, or selected model folder is not the output directory. Confirm `AutoTokenizer.from_pretrained('./Eather-500M-Unsloth')` works.
- `Unrecognized model in ...`: use the output folder, not the original custom `config.json`.
- `Out of memory`: 500M full-parameter training on a 16GB card is tight, especially with AdamW and activation memory; lower context/microbatch and use gradient checkpointing. Exporting is easier than training.
- `Missing tokenizer`: you must supply a tokenizer **trained on actual source data**; the project ZIP does not include a pretrained tokenizer or pretrained weights.
|