# Eather-500M and Unsloth Studio (Arch Linux / ROCm)

## Important distinction

The original `model.py` is a custom PyTorch architecture and its checkpoints are **not** recognized by Unsloth Studio just by supplying `tokenizer.json`. The export script builds a **new randomly initialized, standard Hugging Face Llama-format model** with matching key dimensions. It **does not convert existing trained original Eather checkpoints**. RoPE conventions and module naming differ; do not load raw custom checkpoint tensors as Llama weights.

Studio's training support for a particular local random-weight model, full-parameter pretraining on ROCm, or importing a local path is version-dependent. Producing a valid Transformers model directory does **not guarantee** Studio will accept it or train it. Check compatibility in the installed Studio version. For guaranteed control over custom model training, use `train.py` from the original project.

## 1. Install dependencies in your dedicated Python environment

```bash
python -m venv .venv
source .venv/bin/activate
pip install torch transformers tokenizers safetensors sentencepiece
```

On Arch/AMD ROCm, **do not replace an already working ROCm PyTorch build with a CPU-only build**; follow your ROCm PyTorch setup for the training environment. For the export script, CPU PyTorch is sufficient.

## 2. Train or reuse the matching 32k tokenizer

```bash
python tokenizer_train.py --input data/raw --output tokenizer.json
```

Your source text must be large enough to create exactly 32,000 tokens. If you already generated the **matching** tokenizer, reuse it. Do not switch tokenizers for a pretrained checkpoint.

## 3. Export HF-compatible tokenizer and new model weights

```bash
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-500M-Unsloth
```

It creates `config.json`, `generation_config.json` (if saved by Transformers), `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, and sharded `model-*.safetensors` weights. The model is randomly initialized; it is not pretrained.

To quickly validate the integration without allocating a full 500M model:

```bash
python export_for_unsloth.py --tokenizer tokenizer.json --output Eather-smoke --smoke-test
```

## 4. Validate Hugging Face loading

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
folder = './Eather-500M-Unsloth'
tok = AutoTokenizer.from_pretrained(folder, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(folder)
print(type(model).__name__, len(tok), sum(p.numel() for p in model.parameters()))
```

## 5. Try Unsloth Studio

```bash
unsloth studio update
unsloth studio -p 8888
```

If your installed Studio accepts local model directories, choose `Eather-500M-Unsloth`. The exact UI varies. If Studio rejects that path or model, **do not keep reinstalling sentencepiece**: the Hugging Face folder can already be valid while Studio's train workflow does not support loading/initializing a scratch model. Use an Unsloth-supported model architecture/workflow or train with `train.py` and add Transformers-compatible export later. An untrained scratch model cannot be made capable by LoRA on a few examples.

## Troubleshooting

- `Couldn't instantiate backend tokenizer`: `tokenizer.json` is missing, malformed, or selected model folder is not the output directory. Confirm `AutoTokenizer.from_pretrained('./Eather-500M-Unsloth')` works.
- `Unrecognized model in ...`: use the output folder, not the original custom `config.json`.
- `Out of memory`: 500M full-parameter training on a 16GB card is tight, especially with AdamW and activation memory; lower context/microbatch and use gradient checkpointing. Exporting is easier than training.
- `Missing tokenizer`: you must supply a tokenizer **trained on actual source data**; the project ZIP does not include a pretrained tokenizer or pretrained weights.