rohitsar567 Claude Opus 4.7 (1M context) commited on
Commit
52b0b5d
·
1 Parent(s): fd30ceb

fix(upload-extract): Gemini-primary done right — max_tokens 2048→8192 to stop truncation

Browse files

Previous Gemini swap (7ef3ca3) failed live because max_tokens=2048
truncated the HealthPolicy JSON mid-emission (the schema has 23
nested fields each with a value + source_quote, can easily exceed
2k tokens). json_from_llm_text then couldn't parse the truncated
payload and the schema validation raised, surfacing as
"LLM returned no valid HealthPolicy after primary + fallback retries".

Fix:
- max_tokens: 2048 → 8192 (Gemini 2.5-flash supports it; the
HealthPolicy + quotes typically lands in ~3-5k tokens).
- Gemini-only JSON mode: response_format only sent to the Gemini
primary, not the NIM fallback (NIM has different kwargs).
- Info log on raw-response length so future debugging shows
immediately whether the LLM ran out of output budget.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. backend/uploaded_docs.py +28 -15
backend/uploaded_docs.py CHANGED
@@ -853,33 +853,46 @@ async def extract_one_for_upload(
853
  ChatMessage(role="user", content=prompt),
854
  ]
855
 
856
- # ROLLBACK (2026-05-27 — late session) — the Gemini-primary swap
857
- # in commit 7ef3ca3 broke live extraction (both Gemini and the
858
- # NIM-on-fallback raised, surfacing as "LLM returned no valid
859
- # HealthPolicy after primary + fallback retries"). Reverting to
860
- # NIM as primary (the proven-working path that yielded grade=C
861
- # score=65 on Sarvah Param earlier today). Gemini will be
862
- # re-attempted as a follow-up after proper debugging of its
863
- # JSON-mode response shape against the EXTRACT prompt.
864
- llm_primary = get_brain_llm()
865
  llm_fallback = get_brain_llm()
866
 
867
  raw = ""
868
  policy: Optional[HealthPolicy] = None
869
  for attempt, (llm, label) in enumerate(
870
- [(llm_primary, "nim-primary"), (llm_fallback, "nim-fallback")]
871
  ):
872
  try:
873
  attempt_timeout = 180 if attempt == 0 else 120
 
 
 
 
 
 
 
 
 
 
 
 
 
874
  res = await asyncio.wait_for(
875
- llm.chat(
876
- messages=messages,
877
- temperature=0.0,
878
- max_tokens=2048,
879
- ),
880
  timeout=attempt_timeout,
881
  )
882
  raw = res.text
 
 
 
 
883
  data = json_from_llm_text(raw)
884
  # Force-fill identity fields (REQUIRED by the schema, the
885
  # LLM frequently emits null for these because they're not
 
853
  ChatMessage(role="user", content=prompt),
854
  ]
855
 
856
+ # Gemini-primary (user directive 2026-05-27) — same brain the
857
+ # live chat uses (ADR-040), native JSON-mode via
858
+ # response_mime_type=application/json. NIM stays as the
859
+ # transport-level fallback for: GOOGLE_API_KEY missing, 5xx
860
+ # outage, or quota exhaustion. Key fix vs the earlier broken
861
+ # swap: max_tokens bumped 2048 8192 so the full HealthPolicy
862
+ # JSON (23 nested fields + verbatim quotes) doesn't get
863
+ # truncated mid-emission, which was the actual failure mode.
864
+ llm_primary = GoogleGeminiLLM(timeout=180.0)
865
  llm_fallback = get_brain_llm()
866
 
867
  raw = ""
868
  policy: Optional[HealthPolicy] = None
869
  for attempt, (llm, label) in enumerate(
870
+ [(llm_primary, "gemini-2.5-flash"), (llm_fallback, "nim-fallback")]
871
  ):
872
  try:
873
  attempt_timeout = 180 if attempt == 0 else 120
874
+ chat_kwargs = {
875
+ "messages": messages,
876
+ "temperature": 0.0,
877
+ "max_tokens": 8192,
878
+ }
879
+ # Native JSON mode on the Gemini path — forces a single
880
+ # JSON object response that json_from_llm_text can parse
881
+ # without prose-stripping. Absorbed by **kwargs on the
882
+ # NIM provider (it doesn't honor response_format the same
883
+ # way; rely on the EXTRACT_SYSTEM prompt's JSON-only
884
+ # instruction there).
885
+ if label.startswith("gemini"):
886
+ chat_kwargs["response_format"] = {"type": "json_object"}
887
  res = await asyncio.wait_for(
888
+ llm.chat(**chat_kwargs),
 
 
 
 
889
  timeout=attempt_timeout,
890
  )
891
  raw = res.text
892
+ _log.info(
893
+ "[upload-extract] %s returned %d chars; parsing JSON…",
894
+ label, len(raw or ""),
895
+ )
896
  data = json_from_llm_text(raw)
897
  # Force-fill identity fields (REQUIRED by the schema, the
898
  # LLM frequently emits null for these because they're not