modal-setup/config.md

Modal Inference Endpoint — Working Config

Fill this in after M3 (deploy) and M4 (smoke test) pass. This is the contract the app's ModalChatModel adapter targets.

Active config

Field	Value
model	NousResearch/Hermes-3-Llama-3.1-8B
gpu	L4
tool-call-parser	hermes
max-model-len	8192
endpoint_url	https://yannsay--kobo-analyst-inference-serve.modal.run
volumes	huggingface-cache, vllm-cache
first_cold_start_s	221 (3m41s — weights cached in volume, vLLM init only)
second_cold_start_s	202 (3m22s — weights cached in volume, vLLM reinit only)

32B swap checklist

When ready to upgrade to Qwen2.5-32B-Instruct:

• Change MODEL_NAME = "Qwen/Qwen2.5-32B-Instruct" in inference_app.py
• Change gpu="A100-80GB" in inference_app.py
• modal deploy modal-setup/inference_app.py
• Re-run python modal-setup/smoke_test.py — must print PASS
• Update this file with new timings