RUN.md

Run and verify OUROBOROS Kernel Mint

This Space has three useful execution paths:

• Space Local mode: MiniCPM5-1B GGUF runs with llama.cpp inside the Space, then the in-process referee checks and times the candidate on the Space GPU.
• Space Pro mode: the Qwen3.6-27B smith runs through the Modal backend, then the same referee contract is applied before returning a result.
• Your machine: clone the Space or the GitHub repo, run the 1B or 27B model yourself, then send the generated Triton source through the referee.

A result only matters after the referee compiles the kernel, checks allclose against PyTorch, and times it against PyTorch eager, torch.compile, and torch.compile max-autotune.

1. Run the Space locally

Use this path when you want the same UI on your own CUDA machine.

git clone https://huggingface.co/spaces/build-small-hackathon/ouroboros-kernel-mint
cd ouroboros-kernel-mint
python -m venv .venv
. .venv/bin/activate
python -m pip install -r requirements.txt
python app.py

Open http://localhost:7860, then use Local (offline) in the Build or Expert tab. When not running on Hugging Face ZeroGPU, the spaces.GPU decorator becomes a no-op and the app uses your attached GPU directly.

Useful local knobs:

LOCAL_GGUF_REPO=YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF
LOCAL_GGUF_FALLBACK_REPO=openbmb/MiniCPM5-1B-GGUF
LOCAL_GGUF_QUANTS=Q5_K_M,Q4_K_M,Q8_0
LOCAL_LLAMA_GPU_LAYERS=-1
LOCAL_LLAMA_CTX=4096
LOCAL_MAX_TOKENS=768
python app.py

2. Run the 1B GGUF smith directly

This mirrors Space Local mode. It is the simplest way to run the small model without Modal.

python -m pip install torch triton huggingface_hub \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu130 \
  llama-cpp-python==0.3.28

from huggingface_hub import HfApi, hf_hub_download
from llama_cpp import Llama

repo = "YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF"
files = [f for f in HfApi().list_repo_files(repo) if f.lower().endswith(".gguf")]
filename = next((f for f in files if "Q5_K_M" in f), files[0])
gguf = hf_hub_download(repo, filename=filename)

llm = Llama(model_path=gguf, n_ctx=4096, n_gpu_layers=-1)

system = "You are an expert GPU kernel engineer. Output only one fenced python code block."
user = "Write a fused Triton kernel for row-wise softmax. Use stable max-subtraction. Return run(x)."
prompt = (
    f"<|im_start|>system\n{system}<|im_end|>\n"
    f"<|im_start|>user\n{user}<|im_end|>\n"
    "<|im_start|>assistant\n```python\n"
)
out = llm.create_completion(prompt, max_tokens=768, temperature=0.7, top_p=0.97)
print(out["choices"][0]["text"])

3. Run the 1B PEFT adapter

Use this path when you want the LoRA adapter instead of the GGUF export.

python -m pip install torch transformers peft accelerate triton

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    "openbmb/MiniCPM5-1B",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-minicpm5-1b")
model.eval()

4. Run the 27B adapter locally

The 27B smith is the stronger Qwen3.6-27B run behind the 76 verified compiler-beating kernels. It uses the same prompt contract, but it is not a laptop path. The training run used Modal H200s and peaked around 110 GB VRAM. For local inference, expect a large GPU, multi-GPU device_map="auto", or your own quantization setup.

python -m pip install torch transformers peft accelerate triton

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-27B", trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-27B",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-qwen3.6-27b")
model.eval()

If that is too large for your machine, use Pro in the Space. It calls the hosted 27B backend and still runs the same compile/correctness/timing referee before returning a verdict.

5. Prompt contract

The smith is not being used as a general coding assistant. Keep the prompt narrow:

• ask for exactly one operation,
• name the input tensors and tensor order,
• require one fenced Python code block,
• require one run(...) entry point,
• require fp32 accumulation for reductions,
• forbid prose outside the code block,
• verify the output before trusting it.

System prompt:

You are an expert GPU kernel engineer. Write a single correct, fast Triton kernel.
Output ONLY one fenced python code block defining run(*inputs) and its @triton.jit
kernel. Accumulate reductions in float32. No prose.

User template:

Operation: add_rmsnorm_gelu
Inputs: x, residual, weight. Each row is one transformer hidden state.
Reference: y = gelu(rmsnorm(x + residual, weight)).
Return: one fenced python block with imports, one @triton.jit kernel, and run(x, residual, weight).
Target: correct vs PyTorch first, then faster than torch.compile max-autotune.

Example: residual RMSNorm plus GELU.

Write a fused Triton kernel for add_rmsnorm_gelu.
Inputs are x, residual, and weight, all CUDA tensors.
Compute RMSNorm over each row after x + residual, multiply by weight, then apply GELU.
Use fp32 accumulation for the row reduction.
Return exactly one fenced python code block with run(x, residual, weight).

Example: stable softmax.

Write a fused Triton kernel for row-wise softmax.
Input x is a CUDA tensor shaped [M, N].
Use the stable max-subtraction form.
Return exactly one fenced python code block with run(x).
Do not include explanation text outside the code block.

Example: SwiGLU.

Write a fused Triton kernel for swiglu.
Inputs are gate and up tensors with the same shape.
Compute silu(gate) * up elementwise.
Return exactly one fenced python code block with run(gate, up).
Keep the launch grid simple and contiguous-row friendly.

6. Verify a generated kernel

Save the model output as candidate.py, then run it through the referee. The public helper signature is:

evaluate_inprocess_full(kernel_src, spec_name, n_shapes=2, n_iters=30)

Minimal check:

git clone https://github.com/ymrohit/ouroboros-kernelsmith.git
cd ouroboros-kernelsmith
python -m pip install torch triton numpy

import pathlib
import sys

sys.path.insert(0, "referee")
from harness import evaluate_inprocess_full

kernel_src = pathlib.Path("candidate.py").read_text()
result = evaluate_inprocess_full(kernel_src, "add_rmsnorm_gelu", n_shapes=2, n_iters=30)
print(result.to_dict())

Submission-grade output has:

• status == "ok",
• correct == True,
• nonzero eager, compile, and max-autotune timings,
• speedup_maxauto > 1.0 if claiming a compiler-beating result.

7. Backend endpoints

The live Space already has these defaults wired in app.py:

1B Modal backend:  https://ymrohit--ouroboros-kernel-mint-mint-mint.modal.run
27B Modal backend: https://ymrohit--ouroboros-kernel-mint-pro-mint-mint.modal.run
ops menu:          https://ymrohit--ouroboros-kernel-mint-ops.modal.run
leaderboard:       https://ymrohit--ouroboros-kernel-mint-leaderboard.modal.run

The backend can scale to zero, so the first non-local mint of a session can take about 90 seconds while the model wakes up. The recorded replay in the Space is a real earlier verified mint, not a mockup.