CNC AI Benchmark: Syntax vs. Physical Reasoning

1. The Capability Gap (Assignment Q1)

Identified Blind Spot: The gap between syntactic/code understanding and physical-world reasoning in CNC machining.

Standard coding benchmarks evaluate an LLM's ability to memorize syntax or write isolated scripts. However, my lived experience as a CNC Operator shows that CNC programming is inherently physical. A model might know that G01 means linear interpolation (syntax), but completely fail to realize that moving a 10mm tool into an 80mm stock cylinder at rapid speed (G00) will cause a catastrophic physical crash.

This benchmark tests if a code-specialized model can translate standard CNC G-code sequences into physically meaningful machine-state reasoning.

2. Dataset Overview

We created a 30-question benchmark utilizing standard Fanuc-style G-code, avoiding machine-specific macros. The dataset scales in difficulty across 5 categories to test our hypothesis:

1. Individual Code Understanding (e.g., What does G96 mean?)
2. Code Interaction (e.g., How do T0202 and G54 interact?)
3. Modal/State Reasoning (e.g., Tracking feed rates after G00 moves)
4. Sequence Reasoning (e.g., Tracking absolute X/Z positions across blocks)
5. Physical Machining Reasoning (e.g., Calculating RPM from CSS, predicting crashes, understanding tool deflection)

Files in this Bucket:

• cnc_questions.jsonl: The 30 prompts fed to the model (no answers included).
• cnc_answer_key.jsonl: The hidden ground truth and 0-2 point grading rubric.
• qwen_results.jsonl: The raw outputs generated by the Qwen model.
• evaluation.ipynb: The Colab notebook used to run the inference.

3. Evaluation Results & Analysis

The model was evaluated using a 0-2 point rubric. The results perfectly validated the hypothesis: the model performs well on memorized syntax but completely fails at physical world modeling.

Score by Category:

• Category 1 (Individual Code Syntax): 9/12 (75%)
• Category 2 (Code Interaction): 5/12 (41%)
• Category 3 (Modal/State Reasoning): 5/12 (41%)
• Category 4 (Sequence Reasoning): 6/12 (50%)
• Category 5 (Physical Machining Reasoning): 4/12 (33%)

Final Score: 29/60 (48%)

Key Failures:

• The 4-Hour Cut: Given a 50mm cut at F0.2 (per rev) and 1000 RPM (a 15-second cut), the model treated the feed as mm/min and stated the cut would take over 4 hours.
• The Crash: The model failed to recognize that a G00 rapid move into unmachined stock coordinates would cause a machine crash, confidently stating it would just "cut through the stock."
• Rigid Tapping: The model stated the spindle would continue to rotate at the bottom of a G84 tapping cycle. Physically, failing to reverse the spindle would instantly snap the tap inside the part.

4. Path Forward (Assignment Q3)

If models struggle with physical reasoning while excelling at syntax, the path forward requires Multi-modal State Tracking Fine-Tuning. Instead of training models purely on flat text files of G-code, models need to be fine-tuned on datasets that pair G-code blocks with the resulting physical state variables (e.g., absolute X/Z position, active feed rate, spindle RPM, and stock material boundaries). By forcing the model to explicitly predict the machine state at every block, we can bridge the gap between text generation and physical world modeling.

5. Links

• Evaluated Model: Qwen/Qwen2.5-Coder-3B-Instruct
• Google Notebook: Notebook