atakan Claude Sonnet 5 commited on
Commit
6902397
·
1 Parent(s): 96a6888

test: Try float16 instead of bfloat16 for the PyTorch CUDA path

Browse files

Switching decoding strategy (greedy vs temperature=0.2) didn't fix the
wrong-coefficient bug from the previous two commits -- still wrong every
time, just differently wrong. That points away from decoding and toward
something in the PyTorch/CUDA numerics itself. bf16's mantissa is the one
thing this path doesn't share with the GGUF/MLX backends that got the
correct answer from the same weights, so testing fp16 as the next candidate.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. controlai_agent/orchestrator.py +9 -1
controlai_agent/orchestrator.py CHANGED
@@ -654,7 +654,15 @@ class ControlAIAgent:
654
  print(f"Loading PyTorch model: {hf_id} (threads: {num_threads})...")
655
  self.hf_tokenizer = AutoTokenizer.from_pretrained(hf_id, trust_remote_code=True)
656
  has_cuda = torch.cuda.is_available()
657
- dtype = torch.bfloat16 if has_cuda else torch.float32
 
 
 
 
 
 
 
 
658
  # SDPA is a large, free speedup over the "eager" attention
659
  # default -- meaningful for the long tool-schema prompt prefix.
660
  attn_impl = "sdpa" if has_cuda else None
 
654
  print(f"Loading PyTorch model: {hf_id} (threads: {num_threads})...")
655
  self.hf_tokenizer = AutoTokenizer.from_pretrained(hf_id, trust_remote_code=True)
656
  has_cuda = torch.cuda.is_available()
657
+ # float16, not bfloat16: live-tested bf16 against this exact
658
+ # model on the zeta/wn step-response prompt and it reproducibly
659
+ # derived the wrong closed-loop coefficients (2.56/2.1952-ish
660
+ # instead of the correct 1.68/1.96) across both greedy and
661
+ # temperature=0.2 sampling -- the one thing GGUF/MLX (which got
662
+ # this right from the same weights) don't share with bf16 is
663
+ # its coarser 7-bit mantissa. Testing fp16's extra precision as
664
+ # the fix for that specific divergence.
665
+ dtype = torch.float16 if has_cuda else torch.float32
666
  # SDPA is a large, free speedup over the "eager" attention
667
  # default -- meaningful for the long tool-schema prompt prefix.
668
  attn_impl = "sdpa" if has_cuda else None