Fix: README

README.md

@@ -9,78 +9,83 @@ tags:
   - qwen
 ---
 
+Q2 — Debugging LLMs (Friendli AI Take-Home)
+Fixed Model Repository
 
-Q2  Debugging LLMs (Friendli AI Take-Home)
-Fixed Model Repo:
https://huggingface.co/syedahmedsoftware/broken-model-fixed
-Original Model:
https://huggingface.co/yunmorning/broken-model
-This repository contains the minimal, production-safe fixes required to make the original model usable behind an OpenAI-compatible /chat/completions API server.
No model weights were modified. Only configuration metadata was corrected.
+https://huggingface.co/syedahmedsoftware/broken-model-fixed
+Fixed Model Repo:
https://huggingface.co/syedahmedsoftware/broken-model-fixed
+Original Broken Model:
https://huggingface.co/yunmorning/broken-model
+This repository contains the minimal, production-safe fixes required to make the original model usable behind an OpenAI-compatible /chat/completions API server.
No model weights were modified. Only configuration metadata was corrected.
+https://huggingface.co/yunmorning/broken-model
+
+This repository contains the minimal, production-safe fixes required to make the original model usable behind an OpenAI-compatible /chat/completions API server.
+
+No model weights were modified.
+Only configuration metadata was corrected.
 
 Problem (a) — Why inference failed
+
 The original repository could not serve chat requests because two required metadata components were missing or incorrect.
-Root Causes
+
+Root causes
 1) Missing chat_template (tokenizer_config.json)
+
 Modern OpenAI-style runtimes rely on:
-python
+
 tokenizer.apply_chat_template(messages)
 
 
-The original model did not define a chat template, so:
-	•	Chat messages could not be rendered into prompts
-	•	/chat/completions servers had no deterministic formatting
-	•	Inference failed or produced undefined behavior
+The original model did not define a chat template, which meant:
 
-2) Missing/incorrect pad_token_id (config.json)
-Production inference uses:
-	•	batched decoding
-	•	attention masking
-	•	padded sequences
-The original config.json did not define a valid pad_token_id, making batching unsafe and causing runtime instability.
+Chat messages could not be rendered into prompts
 
+/chat/completions servers had no deterministic formatting
 
-Minimal Fixes Applied (No Weight Changes)
-File
-Change
-Why
-tokenizer_config.json
-Added ChatML-style chat_template
-Required for OpenAI-style chat formatting
-config.json
-Set pad_token_id to tokenizer’s pad token
-Enables safe batching and attention masks
-generation_config.json
-Normalized pad/eos fields (kept defaults)
-Prevents decoding edge cases
+Inference either failed or produced undefined behavior
 
-No architecture, tokenizer vocab, or model weights were changed.
+2) Missing / incorrect pad_token_id (config.json)
 
-Verification (Remote)
+Production inference relies on:
+
+Batched decoding
+
+Attention masking
+
+Padded sequences
+
+The original config.json did not define a valid pad_token_id, which makes batching unsafe and causes runtime instability.
+
+Minimal fixes applied (no weight changes)
+
+Only metadata was changed:
+
+tokenizer_config.json
+Added a ChatML-style chat_template so chat messages can be rendered correctly.
+
+config.json
+Set pad_token_id to match the tokenizer’s pad token so batching and masking are safe.
 
+generation_config.json
+Normalized pad_token_id and eos_token_id to prevent decoding edge cases.
 
-python - <<'PY'
-from transformers import AutoTokenizer
+No architecture, tokenizer vocabulary, or model weights were modified.
 
-repo_id = "syedahmedsoftware/broken-model-fixed"
-tok = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
+Verification (remote)
+python - <<'PY' from transformers import AutoTokenizer
 
-print("tokenizer:", tok.__class__.__name__)
-print("has_chat_template:", bool(getattr(tok, "chat_template", None)))
-print("pad_token_id:", tok.pad_token_id, "eos_token_id:", tok.eos_token_id)
+repo_id = "syedahmedsoftware/broken-model-fixed" tok = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
 
-messages = [
-  {"role":"system","content":"You are helpful."},
-  {"role":"user","content":"Say hello."}
-]
-prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
-print("✅ REMOTE prompt renders. Preview:")
-print(prompt[:250])
-PY
+print("tokenizer:", tok.class.name) print("has_chat_template:", bool(getattr(tok, "chat_template", None))) print("pad_token_id:", tok.pad_token_id, "eos_token_id:", tok.eos_token_id)
 
-Expected output:
+messages = [ {"role":"system","content":"You are helpful."}, {"role":"user","content":"Say hello."} ] prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) print(" REMOTE prompt renders. Preview:") print(prompt[:250]) PY
 
+Expected output
 tokenizer: Qwen2Tokenizer
 has_chat_template: True
-pad_token_id: 151643 eos_token_id: 151645
-✅ REMOTE prompt renders. Preview:
+pad_token_id: 151643
+eos_token_id: 151645
+
+Rendered prompt:
 <|im_start|>system
 You are helpful.<|im_end|>
 <|im_start|>user
@@ -88,45 +93,54 @@ Say hello.<|im_end|>
 <|im_start|>assistant
 
 
+This confirms that the model now supports OpenAI-style chat formatting.
 
 Problem (b) — Why reasoning_effort has no effect
-Root Cause
-reasoning_effort is not a native Transformers generation parameter.
Unless the serving runtime explicitly maps it to real compute policies, it is silently ignored.
-The base model has no internal mechanism to interpret “effort.”
+Root cause
+reasoning_effort is not a native Transformers generation parameter.
Unless the serving runtime explicitly maps it to real compute policies, it is silently ignored.
+reasoning_effort is not a native Transformers generation parameter.
+
+Unless the serving runtime explicitly maps this field to real compute or decoding policies, it is silently ignored. The base model itself has no internal mechanism to interpret the concept of “effort.”
 
 What is required to make reasoning_effort meaningful
 1) Runtime orchestration (required)
-The server must map reasoning_effort to actual strategies, for example:
-Effort Level
-Runtime Policy
-low
-single pass, greedy
-medium
-temperature + nucleus sampling
-high
-multi-sample + rerank
-very_high
-tree-of-thought / verifier loop
-The runtime must log which policy is applied.
-2) Architectural support (one of)
-	•	Multi-pass verifier loop
-	•	Tree search / self-consistency
-	•	Reflection / critique agent
-	•	Budgeted decoding controller
-	•	Control-token trained model
-Without these, reasoning_effort remains a no-op.
 
-Final Notes
-These fixes:
-	•	Restore deterministic chat formatting
-	•	Enable production-safe batching
-	•	Make the model compatible with OpenAI-style /chat/completions servers
-The model is now deployable in real inference environments.
+The server must translate reasoning_effort into real execution strategies, for example:
+
+low → single pass, greedy decoding
+
+medium → temperature + nucleus sampling
+
+high → multi-sample + rerank
+
+very_high → tree-of-thought or verifier loop
 
+The runtime must explicitly log which policy is applied.
 
+2) Architectural support (at least one)
 
+One or more of the following must exist:
 
+Multi-pass verifier or self-consistency loop
 
+Tree search or branch-and-bound reasoning
 
+Reflection / critique agent
 
+Budget-controlled decoding controller
 
+Control-token trained model
+
+Without these, reasoning_effort remains a no-op.
+
+Final notes
+
+These fixes:
+
+Restore deterministic chat formatting
+
+Enable production-safe batching
+
+Make the model compatible with OpenAI-style /chat/completions servers
+
+The model is now deployable in real inference environments.