Spaces:
Running on Zero
Running on Zero
atakan Claude Sonnet 5 commited on
Commit ·
6902397
1
Parent(s): 96a6888
test: Try float16 instead of bfloat16 for the PyTorch CUDA path
Browse filesSwitching decoding strategy (greedy vs temperature=0.2) didn't fix the
wrong-coefficient bug from the previous two commits -- still wrong every
time, just differently wrong. That points away from decoding and toward
something in the PyTorch/CUDA numerics itself. bf16's mantissa is the one
thing this path doesn't share with the GGUF/MLX backends that got the
correct answer from the same weights, so testing fp16 as the next candidate.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
controlai_agent/orchestrator.py
CHANGED
|
@@ -654,7 +654,15 @@ class ControlAIAgent:
|
|
| 654 |
print(f"Loading PyTorch model: {hf_id} (threads: {num_threads})...")
|
| 655 |
self.hf_tokenizer = AutoTokenizer.from_pretrained(hf_id, trust_remote_code=True)
|
| 656 |
has_cuda = torch.cuda.is_available()
|
| 657 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 658 |
# SDPA is a large, free speedup over the "eager" attention
|
| 659 |
# default -- meaningful for the long tool-schema prompt prefix.
|
| 660 |
attn_impl = "sdpa" if has_cuda else None
|
|
|
|
| 654 |
print(f"Loading PyTorch model: {hf_id} (threads: {num_threads})...")
|
| 655 |
self.hf_tokenizer = AutoTokenizer.from_pretrained(hf_id, trust_remote_code=True)
|
| 656 |
has_cuda = torch.cuda.is_available()
|
| 657 |
+
# float16, not bfloat16: live-tested bf16 against this exact
|
| 658 |
+
# model on the zeta/wn step-response prompt and it reproducibly
|
| 659 |
+
# derived the wrong closed-loop coefficients (2.56/2.1952-ish
|
| 660 |
+
# instead of the correct 1.68/1.96) across both greedy and
|
| 661 |
+
# temperature=0.2 sampling -- the one thing GGUF/MLX (which got
|
| 662 |
+
# this right from the same weights) don't share with bf16 is
|
| 663 |
+
# its coarser 7-bit mantissa. Testing fp16's extra precision as
|
| 664 |
+
# the fix for that specific divergence.
|
| 665 |
+
dtype = torch.float16 if has_cuda else torch.float32
|
| 666 |
# SDPA is a large, free speedup over the "eager" attention
|
| 667 |
# default -- meaningful for the long tool-schema prompt prefix.
|
| 668 |
attn_impl = "sdpa" if has_cuda else None
|