vivekchakraverty Claude Fable 5 commited on
Commit
d4ab656
·
1 Parent(s): e41b244

Fix empty-content crash with reasoning models (GLM-5.2 default)

Browse files

Reasoning models spend tokens thinking before writing the visible answer,
and those tokens count against max_tokens. The seed-keyword step's
max_tokens=400 left GLM-5.2 (the default model) no room to answer: the
API returned content=None with finish_reason=length, and re.search(None)
crashed every default-settings plan generation with 'Unexpected error:
expected string or bytes-like object'.

- app.py: raise the seed-keyword budget 400 -> 2000 (verified sufficient
against the live API with the real prompt).
- llm.py chat(): retry once with +3000 headroom when content is empty at
finish_reason=length, and raise a clear LLMError instead of returning
None if content is still empty - covers every call site centrally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Files changed (2) hide show
  1. app.py +4 -1
  2. modules/llm.py +27 -4
app.py CHANGED
@@ -81,7 +81,10 @@ Product/service: {product_description}
81
  hf_token=hf_token,
82
  model=model,
83
  messages=[{"role": "user", "content": prompt}],
84
- max_tokens=400,
 
 
 
85
  temperature=0.3,
86
  )
87
  match = re.search(r"\[.*\]", raw, re.DOTALL)
 
81
  hf_token=hf_token,
82
  model=model,
83
  messages=[{"role": "user", "content": prompt}],
84
+ # The answer itself is ~100 tokens, but reasoning models (the default
85
+ # GLM-5.2 included) think before answering and that counts against
86
+ # max_tokens — 400 left no room and made content come back empty.
87
+ max_tokens=2000,
88
  temperature=0.3,
89
  )
90
  match = re.search(r"\[.*\]", raw, re.DOTALL)
modules/llm.py CHANGED
@@ -51,13 +51,25 @@ def chat(
51
  raise LLMError("Please enter your Hugging Face access token.")
52
 
53
  client = InferenceClient(api_key=hf_token.strip())
54
- try:
55
- response = client.chat_completion(
 
56
  model=model,
57
  messages=messages,
58
- max_tokens=max_tokens,
59
  temperature=temperature,
60
  )
 
 
 
 
 
 
 
 
 
 
 
61
  except HfHubHTTPError as exc:
62
  status = getattr(exc.response, "status_code", None)
63
  if status == 401:
@@ -77,4 +89,15 @@ def chat(
77
  except Exception as exc: # network errors, timeouts, etc.
78
  raise LLMError(f"LLM request failed: {exc}") from exc
79
 
80
- return response.choices[0].message.content
 
 
 
 
 
 
 
 
 
 
 
 
51
  raise LLMError("Please enter your Hugging Face access token.")
52
 
53
  client = InferenceClient(api_key=hf_token.strip())
54
+
55
+ def _call(budget: int):
56
+ return client.chat_completion(
57
  model=model,
58
  messages=messages,
59
+ max_tokens=budget,
60
  temperature=temperature,
61
  )
62
+
63
+ try:
64
+ response = _call(max_tokens)
65
+ choice = response.choices[0]
66
+ # Reasoning models (GLM-5.2, DeepSeek-V3.2, ...) spend tokens thinking
67
+ # before emitting the visible answer, and those tokens count against
68
+ # max_tokens — a budget that fits the answer can still come back with
69
+ # content=None and finish_reason="length". Retry once with headroom.
70
+ if not choice.message.content and choice.finish_reason == "length":
71
+ response = _call(max_tokens + 3000)
72
+ choice = response.choices[0]
73
  except HfHubHTTPError as exc:
74
  status = getattr(exc.response, "status_code", None)
75
  if status == 401:
 
89
  except Exception as exc: # network errors, timeouts, etc.
90
  raise LLMError(f"LLM request failed: {exc}") from exc
91
 
92
+ content = choice.message.content
93
+ if not content:
94
+ detail = (
95
+ " (it ran out of tokens while reasoning)"
96
+ if choice.finish_reason == "length"
97
+ else ""
98
+ )
99
+ raise LLMError(
100
+ f"The model ({model}) returned an empty response{detail}. "
101
+ "Try again or pick a different model."
102
+ )
103
+ return content