docs/model_integration.md

Model integration

One llama.cpp router serves two aliases through http://127.0.0.1:8080/v1/chat/completions:

• medgemma-professor: MedGemma 27B Q4_K_M, 8K context, full GPU offload.
• medgemma-localizer: MedGemma 1.5 4B Q4_K_M, 4K context.

runtime/models.local-wsl.ini is the local desktop profile: 6K context and full GPU offload. Docker Compose selects that profile through RAD_TRAINER_LLAMA_PRESET; the Modal backend uses runtime/models.ini.

The router runs with --models-max 2, and both aliases are loaded at startup on the L40S backend profile. Session JSON and rendered image arrays persist under the /data volume so a chat can resume after Modal scales the container down and wakes a new one.

Inference order:

1. X-Raydar classifies the primary PA/AP image, then returns its models to CPU.
2. MedGemma 1.5 4B generates structured observations and bounding boxes.
3. MedGemma 27B produces a structured professor review and remains available for streamed chat.

Every accepted model output records model identity, runtime, status, and latency. Chat additionally records time to first token and throughput when llama.cpp reports token usage. Invalid structured localizer output is shown as an explicit failed model run.

The professor receives the current radiographs, the committed blind read, sanitized study metadata, X-Raydar predictions, MedGemma observations, accepted regions, reference labels for bundled public cases, and recent chat history.