| # Kaiju Coder 7 Serving Benchmarks |
|
|
| This file records serving evidence for public download and paid API decisions. |
| The model id must remain `kaiju-coder-7`. |
|
|
| ## Current Live Runtime |
|
|
| - Host: Gojira-B over Tailscale |
| - Local OpenCode base URL: `http://127.0.0.1:18181/v1` |
| - Upstream base URL: `http://100.109.109.14:18084/v1` |
| - Serving stack: vLLM bitsandbytes runtime quantization behind the Kaiju fast |
| proxy |
| - Current verified context: `16384` |
| - Tested high-context target: `32768` |
| - Current container: `qwen36-merged-vllm-18084` |
| - Current caveat: direct raw generation is still slow for multi-file OpenCode |
| work; use the deterministic router/harness for public business-owner demos. |
|
|
| ## Benchmark Command |
|
|
| For current-context latency without restart: |
|
|
| ```bash |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --contexts 12288 \ |
| --prompts identity business_doc code_patch \ |
| --max-tokens 768 \ |
| --timeout 420 |
| ``` |
|
|
| For context restart benchmarking: |
|
|
| ```bash |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --restart \ |
| --contexts 12288 16384 24576 32768 \ |
| --prompts identity business_doc \ |
| --max-tokens 768 \ |
| --timeout 420 \ |
| --ready-timeout 1200 |
| ``` |
|
|
| Use `--contexts 16384` for the current restored Gojira-B endpoint. Use |
| `32768` when explicitly testing the high-context target; it has passed earlier |
| benchmarks but should be re-confirmed after a fresh restart before calling it |
| the live default. |
|
|
| ## Current 12k Direct API Benchmark |
|
|
| Command: |
|
|
| ```bash |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --contexts 12288 \ |
| --prompts identity code_patch \ |
| --max-tokens 256 \ |
| --timeout 300 |
| ``` |
|
|
| Run: `runs/benchmarks/20260603T135017Z-kaiju-coder-7-serving/summary.md` |
|
|
| | Context | Prompt | OK | Seconds | Chars | Chars/s | |
| | --- | --- | --- | ---: | ---: | ---: | |
| | 12288 | identity | True | 2.41 | 26 | 10.788 | |
| | 12288 | code_patch | True | 57.61 | 860 | 14.928 | |
| |
| Interpretation: direct API calls are usable for short tasks, but latency is too |
| high for a paid raw-code API unless outputs are streamed and route-specific |
| limits are enforced. |
| |
| ## 16k Context Benchmark |
| |
| 16k was tested to reduce OpenCode compaction pressure. |
| |
| Commands: |
| |
| ```bash |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --restart \ |
| --contexts 16384 \ |
| --prompts identity \ |
| --max-tokens 128 \ |
| --timeout 300 \ |
| --ready-timeout 1200 |
| |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --contexts 16384 \ |
| --prompts code_patch \ |
| --max-tokens 128 \ |
| --timeout 300 |
| ``` |
| |
| Runs: |
| |
| - `runs/benchmarks/20260603T135651Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T140318Z-kaiju-coder-7-serving/summary.md` |
| |
| | Context | Prompt | OK | Load Wait | Seconds | Chars | Chars/s | |
| | --- | --- | --- | ---: | ---: | ---: | ---: | |
| | 16384 | identity | True | 354.16 | 14.9 | 26 | 1.745 | |
| | 16384 | code_patch | True | n/a | 28.99 | 416 | 14.35 | |
| |
| Interpretation: `16384` is a stable lower-load fallback and still leaves more |
| room above OpenCode's prompt/tool overhead than the original 12k setting. |
| |
| ## 24k And 32k Context Benchmarks |
| |
| 24k and 32k were tested after 16k proved stable. Both loaded and returned the |
| same code-patch latency profile as 16k on the short patch benchmark. |
| |
| Commands: |
| |
| ```bash |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --restart \ |
| --contexts 24576 \ |
| --prompts identity \ |
| --max-tokens 128 \ |
| --timeout 300 \ |
| --ready-timeout 1200 |
|
|
| python3 scripts/benchmark_kaiju_serving.py \ |
| --contexts 24576 \ |
| --prompts code_patch \ |
| --max-tokens 128 \ |
| --timeout 300 |
| |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --restart \ |
| --contexts 32768 \ |
| --prompts identity \ |
| --max-tokens 64 \ |
| --timeout 300 \ |
| --ready-timeout 1200 |
| |
| python3 scripts/benchmark_kaiju_serving.py \ |
| --contexts 32768 \ |
| --prompts code_patch \ |
| --max-tokens 128 \ |
| --timeout 300 |
| ``` |
| |
| Runs: |
| |
| - `runs/benchmarks/20260603T141559Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T142354Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T142439Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T143256Z-kaiju-coder-7-serving/summary.md` |
| |
| | Context | Prompt | OK | Load Wait | Seconds | Chars | Chars/s | |
| | --- | --- | --- | ---: | ---: | ---: | ---: | |
| | 24576 | identity | True | 439.54 | 16.84 | 26 | 1.544 | |
| | 24576 | code_patch | True | n/a | 29.03 | 416 | 14.33 | |
| | 32768 | identity | True | 386.53 | 16.27 | 26 | 1.598 | |
| | 32768 | code_patch | True | n/a | 28.99 | 416 | 14.35 | |
| |
| Interpretation: `32768` is a proven high-context target from this benchmark set, |
| but it is not the currently parked live endpoint after the later |
| quantized-runtime testing. The current Gojira-B/OpenCode profile should stay at |
| `16384` until `32768` is freshly restarted and re-confirmed. Keep `12288` for |
| direct API smoke tests and constrained hardware. |
| |
| Restored-service 32k direct API smoke after vLLM testing: |
| |
| - Run: `runs/benchmarks/20260603T155233Z-kaiju-coder-7-serving/summary.md` |
| - `/v1/models`: `kaiju-coder-7`, max model len `32768` |
| |
| | Context | Prompt | OK | Seconds | Chars | Chars/s | |
| | --- | --- | --- | ---: | ---: | ---: | |
| | 32768 | identity | True | 2.92 | 26 | 8.904 | |
| | 32768 | business_doc | True | 94.28 | 1737 | 18.424 | |
| |
| Interpretation: the restored default endpoint is usable for business-owner |
| document work, but a long proposal response still takes about 94 seconds. Paid |
| routes must stream, cap output, queue carefully, and prefer verified |
| artifact routes over raw open-ended generation. |
| |
| ## OpenCode Customer-Readiness Evidence |
| |
| Final restored-service small OpenCode smoke: |
| |
| ```bash |
| opencode run -m kaiju/kaiju-coder-7 --agent kaiju-coder-7 \ |
| --dir /tmp/kaiju-opencode-32k-final-smoke \ |
| 'Create hello.txt with exactly: Kaiju Coder 7 final 32k ok' |
| ``` |
| |
| Result: passed. OpenCode wrote `hello.txt` with exactly |
| `Kaiju Coder 7 final 32k ok`. |
| |
| Current restored 16k OpenCode smoke after quantized-vLLM testing: |
| |
| ```bash |
| mkdir -p /tmp/kaiju-opencode-fresh-public-smoke |
| opencode run -m kaiju/kaiju-coder-7 --agent kaiju-coder-7 \ |
| --dir /tmp/kaiju-opencode-fresh-public-smoke \ |
| --dangerously-skip-permissions \ |
| 'Create hello.txt with exactly: Kaiju Coder 7 fresh public smoke ok' |
| ``` |
| |
| Result: passed. OpenCode wrote `hello.txt` with exactly |
| `Kaiju Coder 7 fresh public smoke ok` in |
| `/tmp/kaiju-opencode-fresh-public-smoke`, and `/v1/models` returned |
| `kaiju-coder-7` with max model len `16384`. |
| |
| Current restored 16k direct API identity smoke: |
| |
| - Run: `runs/benchmarks/20260603T174545Z-kaiju-coder-7-serving/summary.md` |
| - `/v1/models`: `kaiju-coder-7`, max model len `16384` |
| |
| | Context | Prompt | OK | Seconds | Chars | Chars/s | |
| | --- | --- | --- | ---: | ---: | ---: | |
| | 16384 | identity | True | 2.3 | 26 | 11.304 | |
| |
| Command: |
| |
| ```bash |
| python3 scripts/run_kaiju_opencode_customer_pack.py |
| ``` |
| |
| Latest harnessed product-path result on 2026-06-03: |
| |
| - Run: `runs/opencode-customer-readiness/20260603T185835Z/summary.md` |
| - Mode: `harnessed` |
| - Status: `4/4` passed |
| - Tasks: |
| - `fade-flow-service-site` |
| - `kiyomi-owner-operating-pack` |
| - `paid-api-safety-scaffold` |
| - `release-provenance-safety-review` |
| - Required files written: `28/28` |
| - Forbidden secret-looking tokens: none found by verifier |
| |
| Loop-guarded OpenCode install smoke: |
| |
| - Command: `python3 scripts/install_kaiju_opencode_profile.py`, then |
| `opencode run -m kaiju/kaiju-coder-7 --agent kaiju-coder-7 --dir /tmp/kaiju-opencode-loopguard-smoke --dangerously-skip-permissions 'Create loopguard.txt with exactly: Kaiju Coder 7 loop guard installed'` |
| - Result: passed. OpenCode wrote `loopguard.txt` in the requested directory with |
| exactly `Kaiju Coder 7 loop guard installed` and exited cleanly. |
| - Installed guard: `/Users/richardecholsai7/.config/opencode/kaiju-no-autocontinue.mjs` |
| |
| Raw OpenCode-agent result on 2026-06-03: |
| |
| - Task: `fade-flow-service-site` |
| - Status: timed out after `900s` |
| - Required files written: `0` |
| - Observed Gojira-B decode throughput while running: about `4.4` tokens/sec |
| - Follow-up runner fix: workspaces now run outside the repo and pass `opencode |
| run --dir <workspace>` explicitly. |
| - Structured follow-up run: |
| `runs/opencode-customer-readiness/20260603T135520Z/results.jsonl` |
| timed out after `60s`, wrote `0` files, and recorded `pwd` as the intended |
| temp workspace. |
| - 16k/stricter-agent follow-up runs: |
| - `runs/opencode-customer-readiness/20260603T140650Z/results.jsonl` |
| timed out after `120s`, wrote `0` files, and recorded the intended temp |
| workspace. |
| - `runs/opencode-customer-readiness/20260603T140908Z/results.jsonl` |
| timed out after `120s`, wrote `0` files after adding stricter "write first |
| file immediately" prompt guidance. |
| - Interpretation: the lean OpenCode agent fits and can write small files. |
| Harnessed file-plan delivery passes the customer pack. Current raw multi-file |
| OpenCode generation is still not public/API ready, so public and paid claims |
| must describe the reliable product path as model plus deterministic harness |
| and verifier. |
| |
| ## Recommendation Until Faster Serving Is Proven |
| |
| - Public local release can proceed only with clear speed/hardware caveats. |
| - Paid API should route business-owner deliverables through deterministic |
| harnesses and verifiers, not raw OpenCode multi-file generation. |
| - Quantized candidates and/or a smaller distilled variant are required for |
| broad public OpenCode usability. |
| |
| ## vLLM Serving Probe |
| |
| vLLM was tested as the practical alternative serving path after SGLang. The |
| standard `vllm/vllm-openai:latest` image cannot read the merged checkpoint's |
| `qwen3_5` config. The Gojira nightly image can read it, but needed two launch |
| fixes for this checkpoint: |
| |
| - preinstall `pandas`, because the Qwen3.5 model path imports it in this image |
| - pass `--language-model-only`, because the merged text-serving checkpoint does |
| not include the visual encoder weights expected by the multimodal config |
| |
| Guarded benchmark command: |
| |
| ```bash |
| KAIJU_VLLM_CONTEXT=16384 KAIJU_VLLM_READY_TIMEOUT=900 \ |
| ./scripts/run-gojira-b-vllm-serving-benchmark.sh |
| ``` |
| |
| Run: `runs/benchmarks/20260603T151244Z-kaiju-coder-7-serving/summary.md` |
| |
| | Stack | Context | Prompt | OK | Seconds | Chars | Chars/s | |
| | --- | ---: | --- | --- | ---: | ---: | ---: | |
| | vLLM nightly | 16384 | identity | True | 19.99 | 26 | 1.301 | |
| | vLLM nightly | 16384 | code_patch | True | 28.8 | 416 | 14.444 | |
|
|
| Interpretation: unquantized vLLM now runs Kaiju Coder 7 at 16k, but it was not |
| clearly faster than SGLang on these smoke prompts. This is historical fallback |
| evidence. The later bitsandbytes vLLM path plus fast proxy is the active speed |
| path. Keep the live/default OpenCode profile at 16k until 32k is freshly |
| re-confirmed. |
|
|
| ## vLLM bitsandbytes Runtime-Quantized Candidate |
|
|
| The first working quantized local variant is a runtime bitsandbytes vLLM path. |
| It does not create separate quantized weights yet; it loads the full merged |
| model through vLLM's bitsandbytes loader. |
|
|
| Command: |
|
|
| ```bash |
| KAIJU_VLLM_CONTEXT=16384 \ |
| KAIJU_VLLM_READY_TIMEOUT=1200 \ |
| KAIJU_VLLM_QUANTIZATION=bitsandbytes \ |
| KAIJU_VLLM_LOAD_FORMAT=bitsandbytes \ |
| ./scripts/run-gojira-b-vllm-serving-benchmark.sh |
| ``` |
|
|
| Runs: |
|
|
| - `runs/benchmarks/20260603T153257Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T154450Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T161316Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T165512Z-kaiju-coder-7-serving/summary.md` |
| - `runs/benchmarks/20260603T223337Z-kaiju-coder-7-serving/summary.md` |
|
|
| | Stack | Context | Prompt | OK | Seconds | Chars | Chars/s | |
| | --- | ---: | --- | --- | ---: | ---: | ---: | |
| | vLLM bitsandbytes | 8192 | identity | True | 21.19 | 26 | 1.227 | |
| | vLLM bitsandbytes | 8192 | code_patch | True | 11.31 | 424 | 37.489 | |
| | vLLM bitsandbytes | 16384 | identity | True | 19.51 | 26 | 1.333 | |
| | vLLM bitsandbytes | 16384 | code_patch | True | 11.3 | 416 | 36.814 | |
| | vLLM bitsandbytes | 16384 | business_doc | True | 53.44 | 1610 | 30.127 | |
| | vLLM bitsandbytes | 16384 | identity | True | 19.65 | 26 | 1.323 | |
| | vLLM bitsandbytes | 16384 | code_patch | True | 24.97 | 997 | 39.924 | |
| | vLLM bitsandbytes | 16384 | business_doc | True | 34.46 | 1615 | 46.874 | |
| |
| Gojira-B vLLM logs reported about `17.8 GiB` model memory for the bitsandbytes |
| load at both 8k and 16k, compared with about `50.22 GiB` for the unquantized |
| vLLM load. Code-patch latency improved materially on this smoke prompt. |
| Business-document latency improved versus the restored 32k SGLang business-doc |
| smoke (`53.44s` at 16k vLLM bitsandbytes versus `94.28s` at 32k SGLang). |
| Identity latency remains slower than SGLang. |
| |
| Quantized OpenCode one-file smoke passed after launching vLLM with |
| `--enable-auto-tool-choice` plus `--tool-call-parser qwen3_coder` and running: |
|
|
| ```bash |
| bash scripts/run_kaiju_quantized_opencode_smoke.sh |
| ``` |
|
|
| Result: OpenCode wrote `/tmp/kaiju-opencode-quantized-smoke/hello.txt` with |
| exactly `Kaiju Coder 7 quantized runtime ok`. |
|
|
| Recommendation: use vLLM bitsandbytes behind the local fast proxy as the |
| current public/OpenCode speed path and keep the installed OpenCode profile at |
| 16k unless the 32k target has just been restarted and re-confirmed. Treat |
| SGLang as fallback and historical high-context evidence. vLLM bitsandbytes has |
| direct identity/code/business-doc evidence plus an OpenCode one-file smoke, but |
| it is not a persisted quantized-weights repo. |
|
|
| ## 2026-06-03 Fast Proxy And Website Harness Speed Pass |
|
|
| The current speed profile keeps runtime-quantized vLLM active on Gojira-B port |
| `18084` and routes OpenCode through the local fast proxy at |
| `http://127.0.0.1:18181/v1`. The proxy preserves OpenCode tool-call streaming |
| while forcing `thinking=false`, model id `kaiju-coder-7`, and bounded output |
| budgets. |
|
|
| Active endpoint checks: |
|
|
| - Local fast proxy health: `http://127.0.0.1:18181/health` |
| - Upstream vLLM models: `http://100.109.109.14:18084/v1/models` |
| - Upstream reports `kaiju-coder-7` with `max_model_len=16384` |
|
|
| Fresh direct vLLM benchmark: |
|
|
| - Run: `runs/benchmarks/20260603T223337Z-kaiju-coder-7-serving/summary.md` |
| - Identity: `19.48s` |
| - Code patch: `24.97s`, `997` chars |
| - Business doc: `34.46s`, `1,615` chars |
|
|
| Fresh OpenCode smoke through the local fast proxy: |
|
|
| - Command: `opencode run -m kaiju/kaiju-coder-7 --agent kaiju-coder-7 --dir /tmp/kaiju-vllm-opencode-smoke --dangerously-skip-permissions 'Create fast-vllm.txt with exactly: Kaiju quantized vLLM OpenCode ok'` |
| - Result: passed in about `23.5s`, wrote the exact requested file. |
| - Packaged public verifier after exact-content agent rule: |
| `runs/public-opencode-smoke/20260603T235002Z/summary.md`, `4/4` |
| passed through `http://127.0.0.1:18181/v1`. |
|
|
| Website harness/router speed pass: |
|
|
| - Direct website harness command: `python3 scripts/run_kaiju_website_harness.py --openai-base-url http://100.109.109.14:18084/v1 --model kaiju-coder-7 ...` |
| - Direct website harness result: `runs/harness/website-speed-pass/avery-stone-vllm.html`, `9,257` chars, `7.31s` |
| - Router command: `python3 scripts/run_kaiju_router.py --kind website --openai-base-url http://100.109.109.14:18084/v1 --model kaiju-coder-7 ...` |
| - Router artifact: `runs/router-speed-pass/20260603T223731Z-website-build-a-premium-one-page-website-for-avery-stone-construction-a-reside/index.html` |
| - Router result: passed in `7.20s`; checks covered complete HTML, required sections, external images, responsive CSS, no lorem ipsum, and manifest write. |
| - Router through the installed local proxy: `runs/router-speed-pass/20260603T224328Z-website-build-a-premium-one-page-website-for-bennett-family-dental-in-charlott/index.html` |
| - Proxy router result: passed in `4.67s`; preserved explicit CTA `Schedule a Visit`, inferred `dental`, and passed the same complete-HTML/static checks. |
|
|
| Updated recommendation: for speed-sensitive OpenCode and paid workflow testing, |
| use vLLM bitsandbytes plus the local fast proxy as the active default. Keep |
| SGLang as fallback/historical evidence, not the fastest current path. For |
| websites and business-owner packs, prefer the deterministic router/harness path |
| over raw long-form HTML generation. |
|
|
| Public business-owner demo pack through the active fast proxy: |
|
|
| ```bash |
| python3 scripts/run_kaiju_public_demo_pack.py \ |
| --openai-base-url http://127.0.0.1:18181/v1 \ |
| --model kaiju-coder-7 \ |
| --planner-timeout 90 |
| ``` |
|
|
| Run: `runs/public-demo-pack/20260603T235009Z/summary.md` |
|
|
| | Task | Result | Seconds | Changed files | |
| | --- | --- | ---: | ---: | |
| | Website | Passed | 4.73 | 2 | |
| | Owner AI company pack | Passed | 29.85 | 19 | |
| | Stripe safety plan | Passed | 9.99 | 2 | |
| | CSV parser artifact | Passed | 19.97 | 2 | |
|
|
| Total: `4/4` passed in `64.529s`. |
|
|
| ## Persisted GGUF Q8_0 Candidate |
| |
| The dedicated persisted-quantization pass found that normal AWQ/GPTQ installs |
| are not clean against the Qwen3.5-capable serving stack tonight, while |
| `llama.cpp` conversion support includes `Qwen3_5ForConditionalGeneration`. |
|
|
| Command: |
|
|
| ```bash |
| ./scripts/probe-gojira-b-persisted-quantization.sh |
| ./scripts/run-gojira-b-kaiju-gguf-convert.sh |
| ``` |
|
|
| Result: |
|
|
| - Artifact: |
| `/home/richardecholsai5/kaiju-coder/models/kaiju-coder-7-gguf/kaiju-coder-7-Q8_0.gguf` |
| - Size: `27G` |
| - SHA256: |
| `596a2c227a429c7309db753061d88d71ee3f8a3b48f17e41ba9d81b0f55bdd4e` |
| - Conversion log: |
| `runs/gguf-conversion/20260603T231446Z/gguf-conversion.log` |
| - Runtime status: candidate only; direct GGUF runtime smoke still required |
| before publishing quantized weights. |
|
|
| Interpretation: the next real speed improvement for broad public users is not |
| another prompt tweak. It is a smoked GGUF or GPU-persisted quantized artifact. |
| The fastest currently verified Kaiju Coder 7 path remains vLLM bitsandbytes |
| plus the local fast proxy and deterministic website/business harnesses. |
|
|
