fix: Wire up the GGUF deployment path and fix its tokenizer mismatch

llama-cpp-python was never in requirements.txt, so on HF Spaces HAS_LLAMA_CPP
was always False and every request silently fell back to slow bf16 PyTorch
inference under ZeroGPU's 300s budget -- despite the app already containing a
full GGUF loading path for exactly this deployment. Added the dependency and
uploaded a quantized GGUF of the currently-best-benchmarked (sft_v2) fused
checkpoint to the atakankahya/ControlAI-Agent Hub repo.

Also fixed: passing an explicit .gguf/ollama model path loaded a hardcoded
"Qwen/Qwen2.5-3B-Instruct" tokenizer instead of this model's own tokenizer and
chat template -- wrong vocab and tool-call format entirely.

Quantization level was chosen empirically, not assumed: tested Q4_K_M, Q6_K,
Q8_0, and full f16 against this exact fine-tuned model on a step-response
prompt (3 runs each). Q4_K_M mis-routed tool calls in 3/3 runs and produced
wildly inconsistent hallucinated numbers; Q6_K/Q8_0/f16 all called the correct
tool with consistent, verified numbers in 3/3 runs. Q8_0 was fastest of the
three correct options and is near-lossless relative to f16 at roughly half
the size, so it's now the default.

Also corrected a stale "68,000+ chunks" README claim -- the actual local RAG
index has ~9,900.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

README.md

@@ -50,7 +50,7 @@ Standard large language models (LLMs) operate probabilistically without determin
 1. **Deterministic Scientific Sandbox:** Computes continuous/discrete algebraic Riccati equations (CARE/DARE), matrix exponentials, and Bode diagrams using LAPACK, SciPy, and CVXPY.
 2. **4-Stage Mathematical Proof Standard:** Formulates system class, analytical theorems, closed-form derivations, and engineering breakdown limits.
 3. **Dynamic Simulation & Plotting:** Solves nonlinear differential equations and renders verified trajectories directly in the interface.
-4. **Offline RAG Knowledge Engine:** Grounded with 68,000+ chunks indexed across classical and modern control engineering literature.
+4. **Offline RAG Knowledge Engine:** Grounded with 9,900+ chunks indexed across classical and modern control engineering literature.
 
 ---
 
@@ -64,7 +64,7 @@ graph TD
     subgraph Core Engine [Hybrid Verification Engine]
         Brain[Foundation Model - Qwen3-4B]
         LoRA[Theory & Tool Adapter - SFT]
-        RAG[Offline Knowledge Engine - 68,000+ Chunks]
+        RAG[Offline Knowledge Engine - 9,900+ Chunks]
         Tools[Deterministic Numerical Tool Suite]
     end
     

controlai_agent/orchestrator.py

@@ -239,6 +239,11 @@ REPETITION_PENALTY = 1.15
 
 PROJECT_ROOT = Path(__file__).resolve().parent.parent
 
+# Our own fine-tuned model's repo -- every GGUF/Ollama/CUDA backend must load
+# ITS tokenizer and chat template (never a generic base model's), since the
+# tool-call format, special tokens, and vocab are specific to this fine-tune.
+CONTROLAI_HF_REPO = "atakankahya/ControlAI-Agent"
+
 # How many times a single tool may be invoked within one user turn. Two allows
 # a legitimate retry with corrected arguments after an error, while stopping
 # the model from spending its whole step budget re-running the same lookup.
@@ -572,7 +577,7 @@ class ControlAIAgent:
 
         if self.is_ollama:
             self.ollama_model = model_path.split(":", 1)[1] if ":" in str(model_path) else "controlai"
-            self.hf_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct", trust_remote_code=True)
+            self.hf_tokenizer = AutoTokenizer.from_pretrained(CONTROLAI_HF_REPO, trust_remote_code=True)
         elif self.is_gguf:
             if not HAS_LLAMA_CPP:
                 raise ImportError("llama-cpp-python is required to run GGUF models. Install it with: pip install llama-cpp-python")
@@ -582,7 +587,12 @@ class ControlAIAgent:
                 n_ctx=16384,
                 verbose=False,
             )
-            self.hf_tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct", trust_remote_code=True)
+            # A local .gguf path carries no tokenizer/chat-template of its own --
+            # always load ours (never a generic base model's), same as the
+            # auto-downloaded deployment path below.
+            local_fused = PROJECT_ROOT / "models" / "controlai_fused"
+            tokenizer_source = str(local_fused) if local_fused.exists() else CONTROLAI_HF_REPO
+            self.hf_tokenizer = AutoTokenizer.from_pretrained(tokenizer_source, trust_remote_code=True)
         elif self.is_mlx:
             if adapter_path:
                 self.model, self.mlx_tokenizer = mlx_load(model_path, adapter_path=adapter_path)
@@ -590,11 +600,22 @@ class ControlAIAgent:
                 self.model, self.mlx_tokenizer = mlx_load(model_path)
             self.hf_tokenizer = AutoTokenizer.from_pretrained(model_path)
         else:
-            # Universal Linux / Cloud / HuggingFace Spaces backend: Fast 4-bit C++ GGUF
+            # Universal Linux / Cloud / HuggingFace Spaces backend: fast C++ GGUF
             # of OUR OWN fine-tuned ControlAI model (never a generic base model).
-            CONTROLAI_HF_REPO = "atakankahya/ControlAI-Agent"
+            #
+            # Q8_0, not Q4_K_M: measured directly against this exact model on the
+            # "simulate a step response" prompt (3 runs each). Q4_K_M mis-routed
+            # tool calls in 3/3 runs (reached for continuous_lqr/bode_analysis
+            # instead of simulate_step_response, then hallucinated inconsistent
+            # overshoot values -- 8.6%, 43.1%, 25.4%, none near the true ~9.5%).
+            # Q6_K, Q8_0, and full f16 all called the correct tool with
+            # consistent, tool-verified numbers in 3/3 runs apiece, matching
+            # local MLX behavior -- Q8_0 was picked among those three because it
+            # was both the fastest of the three in this test (8-19s vs Q6_K's
+            # 11-25s and f16's 11-22s) and, at 8.5 bits/weight, close enough to
+            # f16 to be considered near-lossless, at roughly half f16's size.
             gguf_repo = os.environ.get("CONTROLAI_GGUF_REPO", CONTROLAI_HF_REPO)
-            gguf_filename = os.environ.get("CONTROLAI_GGUF_FILENAME", "*controlai-q4_k_m.gguf")
+            gguf_filename = os.environ.get("CONTROLAI_GGUF_FILENAME", "*controlai-q8_0.gguf")
 
             self.llama_model = None
             if HAS_LLAMA_CPP:

requirements.txt

@@ -13,6 +13,7 @@ control>=0.9.4
 cvxpy>=1.4.0
 rich>=13.7.0
 rank-bm25>=0.2.2
+llama-cpp-python>=0.3.0
 pypdf>=3.17.0
 pymupdf>=1.23.0
 scikit-learn>=1.3.0