docs/COMMANDS.md

# SAGE Commands

This is the repo's current command reference for data preparation, tokenizer training, model training, serving, browser control, and validation.

## Install

```bash
pip install -r requirements.txt
```

## Run tests

```bash
pytest -q
```

## 1. Create a starter dataset

This repo does not ship a large training corpus. The fastest way to unblock the pipeline is to generate the built-in smoke dataset first:

```bash
python -m data.bootstrap --output-dir data/raw --overwrite
```

That writes JSONL files like:

```text
data/raw/general_web.jsonl
data/raw/code.jsonl
data/raw/math_science.jsonl
data/raw/multilingual.jsonl
data/raw/synthetic.jsonl
```

If you want to use your own corpus, put JSONL records in the same folder with at least a `text` field:

```json
{ "text": "your training sample here" }
```

## 2. Train the tokenizer

The tokenizer trainer now accepts plain text files or JSONL files.

```bash
python -m tokenizer.train_tokenizer \
  --input data/raw/general_web.jsonl data/raw/code.jsonl data/raw/math_science.jsonl data/raw/multilingual.jsonl data/raw/synthetic.jsonl \
  --model-prefix tokenizer/tokenizer \
  --vocab-size 4096 \
  --training-text tokenizer/training_corpus.txt
```

## 3. Validate the tokenizer

```bash
python -m tokenizer.validate_tokenizer tokenizer/tokenizer.model
```

## 4. Build parquet shards

```bash
python -m data.pipeline \
  --tokenizer-model tokenizer/tokenizer.model \
  --output-dir data/processed \
  --shard-size 128
```

For a short smoke run:

```bash
python -m data.pipeline \
  --tokenizer-model tokenizer/tokenizer.model \
  --output-dir data/processed \
  --shard-size 32 \
  --limit-per-source 4
```

The shell helper now points to the real data pipeline:

```bash
bash scripts/run_data_pipeline.sh --tokenizer-model tokenizer/tokenizer.model --output-dir data/processed
```

## 5. Start training

Smoke run:

```bash
python -m train.trainer \
  --model-config configs/model/1b.yaml \
  --schedule-config configs/train/schedule.yaml \
  --train-shards data/processed/shard-00000.parquet \
  --validation-shards data/processed/shard-00001.parquet \
  --output-dir runs/smoke \
  --steps 20 \
  --disable-wandb
```

Longer run:

```bash
python -m train.trainer \
  --model-config configs/model/1b.yaml \
  --schedule-config configs/train/schedule.yaml \
  --train-shards data/processed/shard-00000.parquet data/processed/shard-00001.parquet \
  --validation-shards data/processed/shard-00002.parquet \
  --output-dir runs/sage-1b
```

## 6. Serve the model

GPU/PyTorch server:

```bash
python -m serve.start --host 0.0.0.0 --port 8000
```

CPU control-plane server:

```bash
python -m serve.start --cpu --host 0.0.0.0 --port 8001
```

Helper scripts:

```bash
bash scripts/run_serve.sh
bash scripts/run_serve_cpu.sh
```

## 7. Browser control panel

Open the server root:

```text
http://127.0.0.1:8000/
```

The browser UI now supports:

- login with the random 12-character password printed in the terminal at server startup
- dataset bootstrap preset
- shard-building preset
- tokenizer/train/eval/server presets
- raw shell commands
- live job logs
- direct model chat through `/chat`

## 8. API commands

Health:

```bash
curl http://127.0.0.1:8000/health
```

Generate from token ids:

```bash
curl -X POST http://127.0.0.1:8000/generate \
  -H "Content-Type: application/json" \
  -d "{\"input_ids\": [1, 42, 99], \"max_new_tokens\": 8}"
```

Chat from text:

```bash
curl -X POST http://127.0.0.1:8000/chat \
  -H "Content-Type: application/json" \
  -d "{\"prompt\": \"Explain the training flow in this repo.\", \"max_new_tokens\": 64}"
```

Chat status:

```bash
curl http://127.0.0.1:8000/chat/status
```

## 9. Evaluation

```bash
python -m eval.run_benchmarks
```

Or use the helper:

```bash
bash scripts/run_eval.sh
```

## 10. Hugging Face sync

```bash
python hf_push.py
```