SPACE_RUNBOOK.md

## Hugging Face token (never commit real values)

1. On https://huggingface.co/settings/tokens create a token with **write** access.
2. In the Space: **Settings → Repository secrets** add:
   - **Name:** `HF_TOKEN` (or `HUGGINGFACE_HUB_TOKEN`)
   - **Value:** your token
3. **Restart the Space** so the env is applied.
4. If a token was ever pasted in chat or committed, **revoke** it and create a new one.

## Check disk writes + that HF token is visible to the app

Pings the Space and creates `outputs/simpllll.csv` (overwrites on each call). JSON includes `hf_token_env_set: true/false` (never the secret). After redeploy, run:

```bash
curl -sS "https://hiitsesh-new-gpu-space.hf.space/outputs/write_test"
```

**Why the CSV does not appear on huggingface.co → Files / `outputs/`:** that view is the **git repository** (committed files). The probe file is written on the **running container** only. Use the `download_url` from the JSON, or:

```bash
curl -sS "https://hiitsesh-new-gpu-space.hf.space/outputs/file?path=simpllll.csv"
```

To list **runtime** files: `GET /outputs/ls` (or open it in the browser).

## Train (saves best-by-loss to `outputs/best_by_loss`)

Replace the Space host if yours differs. `model_name` must be URL-encoded (`/` → `%2F`).

```bash
curl -sS -X POST \
  "https://hiitsesh-new-gpu-space.hf.space/train/pilot?max_steps=100&num_generations=8&gradient_accumulation_steps=8&max_completion_length=512&learning_rate=1e-5&logging_steps=5&model_name=Qwen%2FQwen3-1.7B&bf16=true&best_loss_dir=outputs%2Fbest_by_loss"
```

Optional: use a **separate** folder per run, e.g. `best_loss_dir=outputs%2Fbest_1.7b_2026-04-26`.

### Automatic upload when training finishes (recommended)

Create an empty **model** repo on the Hub first (or it will be created on first push). With `HF_TOKEN` set as a Space secret, training can push **after** `trainer.train()` finishes so weights are not lost if the machine restarts.

- `hub_model_repo` — your repo id, URL-encoded (`/` → `%2F`), e.g. `YOUR_USER%2Freleaseops-grpo-1.7b`
- `hub_upload_include` — `best` (default, `outputs/best_by_loss` → `best_by_loss/` on Hub), `final` (trainer output only → `trainer_final/`), or `both`

Example (append to the same pilot URL as `&hub_model_repo=...&hub_upload_include=both`):

```text
&hub_model_repo=YOUR_USER%2Freleaseops-grpo-1.7b&hub_upload_include=both
```

**Why not GitHub for the model files?** Fine for **code**; **multi‑GB checkpoints** hit LFS limits and are painful. Hugging Face Hub (or S3/GCS) is built for model weights. You can still keep your **code** on GitHub and **weights** on the Hub.

## After training: push finetuned weights to the Hub (manual, same as automatic)

Use a **model** repo (recommended) so you can `from_pretrained` later. Replace `YOUR_USER` and repo name.

```bash
curl -sS \
  "https://hiitsesh-new-gpu-space.hf.space/train/push_to_hub?repo_id=YOUR_USER%2Freleaseops-grpo-1.7b-best&repo_type=model&path=best_by_loss"
```

- `path` is relative to `outputs/`, so `best_by_loss` means `outputs/best_by_loss/`.
- To push the full trainer run dir: `path=releaseops-grpo` (if present).

## Download without Hub (ephemeral)

```bash
curl -sS -o best_by_loss.tar.gz "https://hiitsesh-new-gpu-space.hf.space/outputs/archive?path=best_by_loss"
```

## Use the finetuned model later (Python)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "YOUR_USER/releaseops-grpo-1.7b-best"  # or local folder
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, device_map="auto")
```