RUN.md

# Run and verify OUROBOROS Kernel Mint

This Space has three useful execution paths:

- **Space Local mode:** MiniCPM5-1B GGUF runs with llama.cpp inside the Space, then the
  in-process referee checks and times the candidate on the Space GPU.
- **Space Pro mode:** the Qwen3.6-27B smith runs through the Modal backend, then the same
  referee contract is applied before returning a result.
- **Your machine:** clone the Space or the GitHub repo, run the 1B or 27B model yourself,
  then send the generated Triton source through the referee.

A result only matters after the referee compiles the kernel, checks allclose against
PyTorch, and times it against PyTorch eager, `torch.compile`, and
`torch.compile` max-autotune.

## 1. Run the Space locally

Use this path when you want the same UI on your own CUDA machine.

```bash
git clone https://huggingface.co/spaces/build-small-hackathon/ouroboros-kernel-mint
cd ouroboros-kernel-mint
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python app.py
```

Open `http://localhost:7860`, then use **Local (offline)** in the Build or Expert tab.
When not running on Hugging Face ZeroGPU, the `spaces.GPU` decorator becomes a no-op and
the app uses your attached GPU directly.

Useful local knobs:

```bash
LOCAL_GGUF_REPO=YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF
LOCAL_GGUF_FALLBACK_REPO=openbmb/MiniCPM5-1B-GGUF
LOCAL_GGUF_QUANTS=Q5_K_M,Q4_K_M,Q8_0
LOCAL_LLAMA_GPU_LAYERS=-1
LOCAL_LLAMA_CTX=4096
LOCAL_MAX_TOKENS=768
python app.py
```

## 2. Run the 1B GGUF smith directly

This mirrors Space Local mode. It is the simplest way to run the small model without Modal.

```bash
python -m pip install torch triton huggingface_hub \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu130 \
  llama-cpp-python==0.3.28
```

```python
from huggingface_hub import HfApi, hf_hub_download
from llama_cpp import Llama

repo = "YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF"
files = [f for f in HfApi().list_repo_files(repo) if f.lower().endswith(".gguf")]
filename = next((f for f in files if "Q5_K_M" in f), files[0])
gguf = hf_hub_download(repo, filename=filename)

llm = Llama(model_path=gguf, n_ctx=4096, n_gpu_layers=-1)

system = "You are an expert GPU kernel engineer. Output only one fenced python code block."
user = "Write a fused Triton kernel for row-wise softmax. Use stable max-subtraction. Return run(x)."
prompt = (
    f"<|im_start|>system\n{system}<|im_end|>\n"
    f"<|im_start|>user\n{user}<|im_end|>\n"
    "<|im_start|>assistant\n```python\n"
)
out = llm.create_completion(prompt, max_tokens=768, temperature=0.7, top_p=0.97)
print(out["choices"][0]["text"])
```

## 3. Run the 1B PEFT adapter

Use this path when you want the LoRA adapter instead of the GGUF export.

```bash
python -m pip install torch transformers peft accelerate triton
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    "openbmb/MiniCPM5-1B",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-minicpm5-1b")
model.eval()
```

## 4. Run the 27B adapter locally

The 27B smith is the stronger Qwen3.6-27B run behind the 76 verified compiler-beating
kernels. It uses the same prompt contract, but it is not a laptop path. The training run
used Modal H200s and peaked around 110 GB VRAM. For local inference, expect a large GPU,
multi-GPU `device_map="auto"`, or your own quantization setup.

```bash
python -m pip install torch transformers peft accelerate triton
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-27B", trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-27B",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-qwen3.6-27b")
model.eval()
```

If that is too large for your machine, use **Pro** in the Space. It calls the hosted 27B
backend and still runs the same compile/correctness/timing referee before returning a
verdict.

## 5. Prompt contract

The smith is not being used as a general coding assistant. Keep the prompt narrow:

- ask for exactly one operation,
- name the input tensors and tensor order,
- require one fenced Python code block,
- require one `run(...)` entry point,
- require fp32 accumulation for reductions,
- forbid prose outside the code block,
- verify the output before trusting it.

System prompt:

```text
You are an expert GPU kernel engineer. Write a single correct, fast Triton kernel.
Output ONLY one fenced python code block defining run(*inputs) and its @triton.jit
kernel. Accumulate reductions in float32. No prose.
```

User template:

```text
Operation: add_rmsnorm_gelu
Inputs: x, residual, weight. Each row is one transformer hidden state.
Reference: y = gelu(rmsnorm(x + residual, weight)).
Return: one fenced python block with imports, one @triton.jit kernel, and run(x, residual, weight).
Target: correct vs PyTorch first, then faster than torch.compile max-autotune.
```

Example: residual RMSNorm plus GELU.

```text
Write a fused Triton kernel for add_rmsnorm_gelu.
Inputs are x, residual, and weight, all CUDA tensors.
Compute RMSNorm over each row after x + residual, multiply by weight, then apply GELU.
Use fp32 accumulation for the row reduction.
Return exactly one fenced python code block with run(x, residual, weight).
```

Example: stable softmax.

```text
Write a fused Triton kernel for row-wise softmax.
Input x is a CUDA tensor shaped [M, N].
Use the stable max-subtraction form.
Return exactly one fenced python code block with run(x).
Do not include explanation text outside the code block.
```

Example: SwiGLU.

```text
Write a fused Triton kernel for swiglu.
Inputs are gate and up tensors with the same shape.
Compute silu(gate) * up elementwise.
Return exactly one fenced python code block with run(gate, up).
Keep the launch grid simple and contiguous-row friendly.
```

## 6. Verify a generated kernel

Save the model output as `candidate.py`, then run it through the referee. The public helper
signature is:

```python
evaluate_inprocess_full(kernel_src, spec_name, n_shapes=2, n_iters=30)
```

Minimal check:

```bash
git clone https://github.com/ymrohit/ouroboros-kernelsmith.git
cd ouroboros-kernelsmith
python -m pip install torch triton numpy
```

```python
import pathlib
import sys

sys.path.insert(0, "referee")
from harness import evaluate_inprocess_full

kernel_src = pathlib.Path("candidate.py").read_text()
result = evaluate_inprocess_full(kernel_src, "add_rmsnorm_gelu", n_shapes=2, n_iters=30)
print(result.to_dict())
```

Submission-grade output has:

- `status == "ok"`,
- `correct == True`,
- nonzero eager, compile, and max-autotune timings,
- `speedup_maxauto > 1.0` if claiming a compiler-beating result.

## 7. Backend endpoints

The live Space already has these defaults wired in `app.py`:

```text
1B Modal backend:  https://ymrohit--ouroboros-kernel-mint-mint-mint.modal.run
27B Modal backend: https://ymrohit--ouroboros-kernel-mint-pro-mint-mint.modal.run
ops menu:          https://ymrohit--ouroboros-kernel-mint-ops.modal.run
leaderboard:       https://ymrohit--ouroboros-kernel-mint-leaderboard.modal.run
```

The backend can scale to zero, so the first non-local mint of a session can take about
90 seconds while the model wakes up. The recorded replay in the Space is a real earlier
verified mint, not a mockup.