Handle negated pain mentions in injury keyword detection
Lucas
781cd97
"about my calves, nothing hurts" matched the "hurts" keyword, forced
pain_or_injury to "yes", and the muscle word "calves" then synthesized
a phantom gastrocnemius pain issue. Strip negated injury mentions
("no pain", "nothing hurts", "doesn't hurt", "pain free") before the
keyword check; positive mentions elsewhere in the text still force
"yes" ("no pain except my knee hurts").
Sanitize parser payload instead of rejecting on schema deviations
Lucas
6ab2085
Space parses failed intermittently with extra_forbidden validation
errors: the 1.7B model invents extra keys and out-of-range values, and
output varies between identical requests because KV-prefix reuse
shifts prefill batch boundaries enough to flip near-tie greedy picks.
Add a deterministic sanitation pass before Pydantic validation:
- drop unknown keys at every level (logged for diagnosis)
- coerce invalid enums to their unknown-equivalent (severity ->
"unsure", follow-up field/context label -> "other", unknown muscle
-> null for notes-based inference)
- null out-of-range numbers, fill or drop empty required strings
- keep only string entries in missing_fields/follow_up_questions
Strict rejection tests for synonym/extra keys now assert the drop
behavior; the engine still never sees anything outside the schema.
Fix Space prefill slowness: cap prefill threads, warm up at startup
Lucas
689f982
Space logs showed generation completing but taking ~159s, nearly all
prefill (~9 tok/s for 1496 tokens on 8 vCPUs). llama-cpp-python
defaults n_threads_batch (prefill threads) to the host core count,
which containers report instead of the cgroup quota, so prefill
oversubscribed the CPU quota and stalled under CFS throttling.
LLAMA_CPP_N_THREADS only set decode threads.
- Pass n_threads_batch explicitly, defaulting to n_threads; new
LLAMA_CPP_N_THREADS_BATCH env var overrides
- Log os_cpu_count at model load to expose oversubscription
- Warm up the parser in a background thread at app startup: the prompt
is constant up to the trailing check-in text, so prefilling it once
lets llama.cpp's prefix KV-cache reuse skip most prefill on real
requests (locally: 14s warmup, then 8.7s first parse)
- Serialize model access with a lock; llama-cpp-python is not
thread-safe and warmup may overlap a user request
The full Pydantic schema grammar made per-token sampling unusably slow
on Space CPUs, and Qwen3 thinking mode collapsed grammar-constrained
output to bare "{}".
- Send a compact response shape instead of the full JSON schema
(~2.7k -> ~1.4k prompt tokens); applies to llama.cpp and transformers
- Build the ChatML prompt by hand, prefill an empty <think> block and
add /no_think so JSON-constrained generation stays on-distribution
- Constrain output with a small whitespace-free generic JSON grammar
via create_completion instead of the schema grammar
- Add deterministic repairs in parse_model_response: lift top-level
keys misplaced inside check_in, null invalid enum values, prune
missing_fields/follow-ups already answered by the check-in, drop
bare field names from follow_up_questions
- Log finish_reason and token usage, warn on truncation
- Default LLAMA_CPP_MAX_TOKENS to 512
Revert "Disable llama.cpp schema grammar by default"
Lucas
675478d
This reverts commit 5e5caa6edde48c436a1fa4e4f89de31ea7860569.
