Spaces:
Sleeping
fix(upload-extract): Gemini-primary done right — max_tokens 2048→8192 to stop truncation
Browse filesPrevious Gemini swap (7ef3ca3) failed live because max_tokens=2048
truncated the HealthPolicy JSON mid-emission (the schema has 23
nested fields each with a value + source_quote, can easily exceed
2k tokens). json_from_llm_text then couldn't parse the truncated
payload and the schema validation raised, surfacing as
"LLM returned no valid HealthPolicy after primary + fallback retries".
Fix:
- max_tokens: 2048 → 8192 (Gemini 2.5-flash supports it; the
HealthPolicy + quotes typically lands in ~3-5k tokens).
- Gemini-only JSON mode: response_format only sent to the Gemini
primary, not the NIM fallback (NIM has different kwargs).
- Info log on raw-response length so future debugging shows
immediately whether the LLM ran out of output budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend/uploaded_docs.py +28 -15
|
@@ -853,33 +853,46 @@ async def extract_one_for_upload(
|
|
| 853 |
ChatMessage(role="user", content=prompt),
|
| 854 |
]
|
| 855 |
|
| 856 |
-
#
|
| 857 |
-
#
|
| 858 |
-
# NIM
|
| 859 |
-
#
|
| 860 |
-
#
|
| 861 |
-
#
|
| 862 |
-
#
|
| 863 |
-
#
|
| 864 |
-
llm_primary =
|
| 865 |
llm_fallback = get_brain_llm()
|
| 866 |
|
| 867 |
raw = ""
|
| 868 |
policy: Optional[HealthPolicy] = None
|
| 869 |
for attempt, (llm, label) in enumerate(
|
| 870 |
-
[(llm_primary, "
|
| 871 |
):
|
| 872 |
try:
|
| 873 |
attempt_timeout = 180 if attempt == 0 else 120
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 874 |
res = await asyncio.wait_for(
|
| 875 |
-
llm.chat(
|
| 876 |
-
messages=messages,
|
| 877 |
-
temperature=0.0,
|
| 878 |
-
max_tokens=2048,
|
| 879 |
-
),
|
| 880 |
timeout=attempt_timeout,
|
| 881 |
)
|
| 882 |
raw = res.text
|
|
|
|
|
|
|
|
|
|
|
|
|
| 883 |
data = json_from_llm_text(raw)
|
| 884 |
# Force-fill identity fields (REQUIRED by the schema, the
|
| 885 |
# LLM frequently emits null for these because they're not
|
|
|
|
| 853 |
ChatMessage(role="user", content=prompt),
|
| 854 |
]
|
| 855 |
|
| 856 |
+
# Gemini-primary (user directive 2026-05-27) — same brain the
|
| 857 |
+
# live chat uses (ADR-040), native JSON-mode via
|
| 858 |
+
# response_mime_type=application/json. NIM stays as the
|
| 859 |
+
# transport-level fallback for: GOOGLE_API_KEY missing, 5xx
|
| 860 |
+
# outage, or quota exhaustion. Key fix vs the earlier broken
|
| 861 |
+
# swap: max_tokens bumped 2048 → 8192 so the full HealthPolicy
|
| 862 |
+
# JSON (23 nested fields + verbatim quotes) doesn't get
|
| 863 |
+
# truncated mid-emission, which was the actual failure mode.
|
| 864 |
+
llm_primary = GoogleGeminiLLM(timeout=180.0)
|
| 865 |
llm_fallback = get_brain_llm()
|
| 866 |
|
| 867 |
raw = ""
|
| 868 |
policy: Optional[HealthPolicy] = None
|
| 869 |
for attempt, (llm, label) in enumerate(
|
| 870 |
+
[(llm_primary, "gemini-2.5-flash"), (llm_fallback, "nim-fallback")]
|
| 871 |
):
|
| 872 |
try:
|
| 873 |
attempt_timeout = 180 if attempt == 0 else 120
|
| 874 |
+
chat_kwargs = {
|
| 875 |
+
"messages": messages,
|
| 876 |
+
"temperature": 0.0,
|
| 877 |
+
"max_tokens": 8192,
|
| 878 |
+
}
|
| 879 |
+
# Native JSON mode on the Gemini path — forces a single
|
| 880 |
+
# JSON object response that json_from_llm_text can parse
|
| 881 |
+
# without prose-stripping. Absorbed by **kwargs on the
|
| 882 |
+
# NIM provider (it doesn't honor response_format the same
|
| 883 |
+
# way; rely on the EXTRACT_SYSTEM prompt's JSON-only
|
| 884 |
+
# instruction there).
|
| 885 |
+
if label.startswith("gemini"):
|
| 886 |
+
chat_kwargs["response_format"] = {"type": "json_object"}
|
| 887 |
res = await asyncio.wait_for(
|
| 888 |
+
llm.chat(**chat_kwargs),
|
|
|
|
|
|
|
|
|
|
|
|
|
| 889 |
timeout=attempt_timeout,
|
| 890 |
)
|
| 891 |
raw = res.text
|
| 892 |
+
_log.info(
|
| 893 |
+
"[upload-extract] %s returned %d chars; parsing JSON…",
|
| 894 |
+
label, len(raw or ""),
|
| 895 |
+
)
|
| 896 |
data = json_from_llm_text(raw)
|
| 897 |
# Force-fill identity fields (REQUIRED by the schema, the
|
| 898 |
# LLM frequently emits null for these because they're not
|