| # Modal Inference Endpoint — Working Config | |
| Fill this in after M3 (deploy) and M4 (smoke test) pass. | |
| This is the contract the app's ModalChatModel adapter targets. | |
| ## Active config | |
| | Field | Value | | |
| |---|---| | |
| | model | NousResearch/Hermes-3-Llama-3.1-8B | | |
| | gpu | L4 | | |
| | tool-call-parser | hermes | | |
| | max-model-len | 8192 | | |
| | endpoint_url | https://yannsay--kobo-analyst-inference-serve.modal.run | | |
| | volumes | huggingface-cache, vllm-cache | | |
| | first_cold_start_s | 221 (3m41s — weights cached in volume, vLLM init only) | | |
| | second_cold_start_s | 202 (3m22s — weights cached in volume, vLLM reinit only) | | |
| ## 32B swap checklist | |
| When ready to upgrade to Qwen2.5-32B-Instruct: | |
| - [ ] Change `MODEL_NAME = "Qwen/Qwen2.5-32B-Instruct"` in inference_app.py | |
| - [ ] Change `gpu="A100-80GB"` in inference_app.py | |
| - [ ] `modal deploy modal-setup/inference_app.py` | |
| - [ ] Re-run `python modal-setup/smoke_test.py` — must print PASS | |
| - [ ] Update this file with new timings | |