# Run and verify OUROBOROS Kernel Mint This Space has three useful execution paths: - **Space Local mode:** MiniCPM5-1B GGUF runs with llama.cpp inside the Space, then the in-process referee checks and times the candidate on the Space GPU. - **Space Pro mode:** the Qwen3.6-27B smith runs through the Modal backend, then the same referee contract is applied before returning a result. - **Your machine:** clone the Space or the GitHub repo, run the 1B or 27B model yourself, then send the generated Triton source through the referee. A result only matters after the referee compiles the kernel, checks allclose against PyTorch, and times it against PyTorch eager, `torch.compile`, and `torch.compile` max-autotune. ## 1. Run the Space locally Use this path when you want the same UI on your own CUDA machine. ```bash git clone https://huggingface.co/spaces/build-small-hackathon/ouroboros-kernel-mint cd ouroboros-kernel-mint python -m venv .venv . .venv/bin/activate python -m pip install -r requirements.txt python app.py ``` Open `http://localhost:7860`, then use **Local (offline)** in the Build or Expert tab. When not running on Hugging Face ZeroGPU, the `spaces.GPU` decorator becomes a no-op and the app uses your attached GPU directly. Useful local knobs: ```bash LOCAL_GGUF_REPO=YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF LOCAL_GGUF_FALLBACK_REPO=openbmb/MiniCPM5-1B-GGUF LOCAL_GGUF_QUANTS=Q5_K_M,Q4_K_M,Q8_0 LOCAL_LLAMA_GPU_LAYERS=-1 LOCAL_LLAMA_CTX=4096 LOCAL_MAX_TOKENS=768 python app.py ``` ## 2. Run the 1B GGUF smith directly This mirrors Space Local mode. It is the simplest way to run the small model without Modal. ```bash python -m pip install torch triton huggingface_hub \ --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu130 \ llama-cpp-python==0.3.28 ``` ```python from huggingface_hub import HfApi, hf_hub_download from llama_cpp import Llama repo = "YMRohit/ouroboros-kernelsmith-minicpm5-1b-GGUF" files = [f for f in HfApi().list_repo_files(repo) if f.lower().endswith(".gguf")] filename = next((f for f in files if "Q5_K_M" in f), files[0]) gguf = hf_hub_download(repo, filename=filename) llm = Llama(model_path=gguf, n_ctx=4096, n_gpu_layers=-1) system = "You are an expert GPU kernel engineer. Output only one fenced python code block." user = "Write a fused Triton kernel for row-wise softmax. Use stable max-subtraction. Return run(x)." prompt = ( f"<|im_start|>system\n{system}<|im_end|>\n" f"<|im_start|>user\n{user}<|im_end|>\n" "<|im_start|>assistant\n```python\n" ) out = llm.create_completion(prompt, max_tokens=768, temperature=0.7, top_p=0.97) print(out["choices"][0]["text"]) ``` ## 3. Run the 1B PEFT adapter Use this path when you want the LoRA adapter instead of the GGUF export. ```bash python -m pip install torch transformers peft accelerate triton ``` ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel tok = AutoTokenizer.from_pretrained("openbmb/MiniCPM5-1B", trust_remote_code=True) base = AutoModelForCausalLM.from_pretrained( "openbmb/MiniCPM5-1B", trust_remote_code=True, torch_dtype="auto", device_map="auto", ) model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-minicpm5-1b") model.eval() ``` ## 4. Run the 27B adapter locally The 27B smith is the stronger Qwen3.6-27B run behind the 76 verified compiler-beating kernels. It uses the same prompt contract, but it is not a laptop path. The training run used Modal H200s and peaked around 110 GB VRAM. For local inference, expect a large GPU, multi-GPU `device_map="auto"`, or your own quantization setup. ```bash python -m pip install torch transformers peft accelerate triton ``` ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.6-27B", trust_remote_code=True) base = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen3.6-27B", trust_remote_code=True, torch_dtype="auto", device_map="auto", ) model = PeftModel.from_pretrained(base, "YMRohit/ouroboros-kernelsmith-qwen3.6-27b") model.eval() ``` If that is too large for your machine, use **Pro** in the Space. It calls the hosted 27B backend and still runs the same compile/correctness/timing referee before returning a verdict. ## 5. Prompt contract The smith is not being used as a general coding assistant. Keep the prompt narrow: - ask for exactly one operation, - name the input tensors and tensor order, - require one fenced Python code block, - require one `run(...)` entry point, - require fp32 accumulation for reductions, - forbid prose outside the code block, - verify the output before trusting it. System prompt: ```text You are an expert GPU kernel engineer. Write a single correct, fast Triton kernel. Output ONLY one fenced python code block defining run(*inputs) and its @triton.jit kernel. Accumulate reductions in float32. No prose. ``` User template: ```text Operation: add_rmsnorm_gelu Inputs: x, residual, weight. Each row is one transformer hidden state. Reference: y = gelu(rmsnorm(x + residual, weight)). Return: one fenced python block with imports, one @triton.jit kernel, and run(x, residual, weight). Target: correct vs PyTorch first, then faster than torch.compile max-autotune. ``` Example: residual RMSNorm plus GELU. ```text Write a fused Triton kernel for add_rmsnorm_gelu. Inputs are x, residual, and weight, all CUDA tensors. Compute RMSNorm over each row after x + residual, multiply by weight, then apply GELU. Use fp32 accumulation for the row reduction. Return exactly one fenced python code block with run(x, residual, weight). ``` Example: stable softmax. ```text Write a fused Triton kernel for row-wise softmax. Input x is a CUDA tensor shaped [M, N]. Use the stable max-subtraction form. Return exactly one fenced python code block with run(x). Do not include explanation text outside the code block. ``` Example: SwiGLU. ```text Write a fused Triton kernel for swiglu. Inputs are gate and up tensors with the same shape. Compute silu(gate) * up elementwise. Return exactly one fenced python code block with run(gate, up). Keep the launch grid simple and contiguous-row friendly. ``` ## 6. Verify a generated kernel Save the model output as `candidate.py`, then run it through the referee. The public helper signature is: ```python evaluate_inprocess_full(kernel_src, spec_name, n_shapes=2, n_iters=30) ``` Minimal check: ```bash git clone https://github.com/ymrohit/ouroboros-kernelsmith.git cd ouroboros-kernelsmith python -m pip install torch triton numpy ``` ```python import pathlib import sys sys.path.insert(0, "referee") from harness import evaluate_inprocess_full kernel_src = pathlib.Path("candidate.py").read_text() result = evaluate_inprocess_full(kernel_src, "add_rmsnorm_gelu", n_shapes=2, n_iters=30) print(result.to_dict()) ``` Submission-grade output has: - `status == "ok"`, - `correct == True`, - nonzero eager, compile, and max-autotune timings, - `speedup_maxauto > 1.0` if claiming a compiler-beating result. ## 7. Backend endpoints The live Space already has these defaults wired in `app.py`: ```text 1B Modal backend: https://ymrohit--ouroboros-kernel-mint-mint-mint.modal.run 27B Modal backend: https://ymrohit--ouroboros-kernel-mint-pro-mint-mint.modal.run ops menu: https://ymrohit--ouroboros-kernel-mint-ops.modal.run leaderboard: https://ymrohit--ouroboros-kernel-mint-leaderboard.modal.run ``` The backend can scale to zero, so the first non-local mint of a session can take about 90 seconds while the model wakes up. The recorded replay in the Space is a real earlier verified mint, not a mockup.