Physics Understanding of Small Language Models (SLMs)

A striking difference between small language models (SLMs) and large language models (LLMs) is their ability of abstract reasoning, critical thinking and solving physics world problems. While LLMs from GPT, Gemini, DeepSeek and Claude families easily solve the physics problems and generate structured output, SLMs struggle a lot. This is the capability gap I observed during my work while testing multiple SLMs on high-school-level numerical physics problems. In this document, I’ll provide details about the evaluation data preparation, evaluation metrics, SLM choices, and results. I’ll conclude with the path forward, proposing how we can close this gap in terms of data, training methodology and architectural changes.

Goal:

To test the physics understanding of SLMs via structured output generation.

Problem:

Given a high-school-level physics question, extract ‘given’, ‘to-find’, and ‘formula’.

• Given: Implicit and explicit givens mentioned in the question.
• To-find: Physical quantity asked to be found, calculated, or determined.
• Formula: Step-by-step formulas required to solve the problem.

Dataset Preparation:

Following steps were taken while preparing an evaluation dataset:

1. Question extraction: Around 525 questions extracted from Tamil Nadu state board (a State in India) high-school textbooks from the 8th to 12th grade.

2. Sampling: 50 questions across 10 domains (mentioned below) are sampled from the 525 questions.

3. Metadata and ground truth preparation: Metadata includes standard, domain, topic, difficulty, question, ground truth, and ground truth in terms of symbol. Ground truths are generated using Claude Sonnet 5.5 and also verified by Gemini 3.8 Flash.

4. Domains: 5 questions per domain across 10 such domains (total 50 questions)

   1. Electrostatics & Current Electricity
   2. Electromagnetic Induction & AC
   3. Mechanics
   4. Optics
   5. Waves & Acoustics
   6. Thermal Physics
   7. Modern Physics
   8. Units & Measurement
   9. Properties of Matter
   10. Electronics & Communication

5. Example: Following is the sample question from the evaluation dataset.

   "id": "Q01",
   "source_id": "EM-01",
   "domain": "Electrostatics & Current Electricity",
   "topic": "Current Electricity",
   "standard": "Class 10",
   "difficulty": "easy",
   "question": "The resistance of a wire of length 10 m is 2 ohm. If the area of cross section of the wire is 2 × 10^-7 m^2, determine its (i) resistivity (ii) conductance and (iii) conductivity.",
   
   "given": {
     "length": "10 m",
     "resistance": "2 ohm",
     "area_of_cross_section": "2 * 10^-7 m^2"
   },
   
   "to_find": [
     "resistivity",
     "conductance",
     "conductivity"
   ],
   
   "formula": [
     "rho = (R * A) / L",
     "G = 1 / R",
     "sigma = 1 / rho"
   ],
   
   "gold_symbols": {
     "given": {
       "L": "10 m",
       "R": "2 ohm",
       "A": "2 * 10^-7 m^2"
     },
   
     "to_find": [
       "rho",
       "G",
       "sigma"
     ],
   
     "formula": [
       "rho = (R * A) / L",
       "G = 1 / R",
       "sigma = 1 / rho"
     ],
   
     "symbol_map": {
       "L": "length",
       "R": "resistance",
       "A": "area_of_cross_section",
       "rho": "resistivity",
       "G": "conductance",
       "sigma": "conductivity"
     }
   },
   "assumed_constants": {},
   "answer": {
     "resistivity": {
       "value": 4e-08,
       "unit": "ohm m"
     },
     "conductance": {
       "value": 0.5,
       "unit": "S"
     },
     "conductivity": {
       "value": 25000000.0,
       "unit": "S/m"
     }
   }
   

2. SLM Selection:

As mentioned in the application, I used SLMs with sizes ranging from 0.6B to 6B. I began with both thinking-on and thinking-off modes where applicable; however, the thinking model showed multiple issues (discussed below). Therefore, final comparison is made across chosen models without thinking mode enabled.

I used the following models from different model families.

1. Qwen3: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B
2. Qwen3.5: Qwen3.5-0.8B, Qwen3.5-2B, Qwen3.5-4B
3. Gemma3: Gemma-3-1B-IT, Gemma-3-4B-IT
4. Gemma4: Gemma-4-E2B-IT, Gemma-4-E4B-IT
5. Phi4: Phi-4-mini-instruct
6. SmolLM: SmolLM-3B

Reasons behind selecting these models.

1. The providers of the Qwen and Gemma models claim that their models are good at maths and reasoning
2. Models from Qwen3.5 and Gemma4 are the latest models.
3. Almost all of these models (Qwen, Gemma, Phi) are coming from world-renowned AI labs which have access to larger datasets, humongous compute resources, and larger models from the same family that perform reasonably well on multiple reasoning and maths benchmarks.
4. SmolLM is chosen because it is directly from the Hugging Face team, who have access to large amounts of data.
5. To check the scaling law in a small size regime, different size models from the same family are chosen.

---

3. Comprehensive Physics Understanding Benchmark: 12 SLMs Comparison

Evaluation done using TypeSafe model's (Jev 1.13) with full Given, To-Find, and Formula soundness verification on 50 questions across 10 physics domains.

I've reported multiple metrics here, but to focus on, consider the exact matched slots (for given and to-find) and physically sound formula slot for 'Formula' field.

Prompt Used:

I used prompt with general instruction for given, to-find and formula with three examples. I used repetation penalty of 1.1 and kept all other hyper-parameters as default.

SYSTEM_PROMPT = """You are an expert physics information extraction engine.

Extract physical quantities, target variables, and step-by-step formulas from the problem text.

Rules:
1. Extraction Scope (Given):
   - Explicit: Numerical quantities directly stated with units (e.g., "800 turns" -> N = 800).
   - Implicit: Boundary conditions from physical keywords (e.g., "starts from rest" -> u = 0 m/s; "at its highest point" -> v = 0 m/s; "perpendicular" -> theta = 90 deg).
   - Strict Anti-Hallucination: Do NOT inject unmentioned constants unless explicitly given in the text.
   - Ratios & Multiples: State directly (e.g., "speed triples" -> v2 = 3 * v1). Do not compute answers.

2. Extraction Scope (To find):
   - Extract only the final target quantities asked to calculate or determine.
   - Do NOT include intermediate steps, values, '?', or units. Output only the symbol of each target (e.g., 'h', 'R_eq').

3. Symbol Rules (apply to Given, To find, and Formula):
   - Canonical Symbols ONLY: Use standard physics symbols everywhere (e.g., 'm', 'v', 'u', 'a', 'F', 'p', 'K', 'theta'). Never write full words ('time', 'pressure', 'velocity') or snake_case names ('final_velocity', 'number_of_turns').
   - Use the same symbol for the same quantity in Given, To find, and Formula.
   - Distinguish different quantities of the same kind with digits or subscripts (e.g., 'R1', 'R2', 'theta1', 'theta2', 'v_1', 'R_eq').

4. Formula Construction Rules (Formula):
   - Step-by-Step Prerequisite Order: If calculating the final formula requires an intermediate quantity not directly given (e.g., a radius needed for an area), output the intermediate formula FIRST, followed by the dependent formula.
   - Standard Operators: Use standard ASCII operators: '+' (addition), '-' (subtraction), '*' (multiplication), '/' (division), and '^' (power). Never use LaTeX or words like 'times'.
   - Grouping Parentheses: Strictly enclose compound expressions, numerators, and denominators in parentheses: '(a * b) / (c + d)'.
   - Algebraic Equations Only: Never write descriptive phrases like 'is proportional to'. Express all ratios and scaling laws as mathematical equations.

5. Sign Conventions:
   - Use the Cartesian sign convention for distances in optics, and give signed values in Given.
   - Retardation or deceleration is a negative acceleration.
   - Work done by a force opposing the displacement is negative.
   - Express electromagnetic induction in terms of magnitude unless the problem asks for direction.

6. Output Format:
   Every field must appear on its own separate line under section headers:
   Given:
   symbol_1 = value unit
   symbol_2 = value unit
   To find:
   symbol_1
   symbol_2
   Formula:
   formula_1
   formula_2

Examples:

Problem:
A stone is thrown vertically upward with a speed of 20 m/s. Taking g = 10 m/s², find the maximum height it reaches.
Output:
Given:
u = 20 m/s
g = 10 m/s²
v = 0 m/s
To find:
h
Formula:
v^2 = u^2 - (2 * g * h)

Problem:
Two resistors of 4 ohm and 6 ohm are connected in series across a 20 V battery. Find the equivalent resistance, the current drawn from the battery and the power dissipated.
Output:
Given:
R1 = 4 ohm
R2 = 6 ohm
V = 20 V
To find:
R_eq
I
P
Formula:
R_eq = R1 + R2
I = V / R_eq
P = V * I

Problem:
A rectangular coil of dimensions 20 cm × 10 cm has 500 turns. It is placed perpendicular to a uniform magnetic field of 0.4 T. If the coil is flipped by 180° about an axis perpendicular to the field in 0.2 seconds, calculate the average electromotive force (emf) induced in the coil.
Output:
Given:
l = 0.2 m
b = 0.1 m
N = 500
B = 0.4 T
theta1 = 0 deg
theta2 = 180 deg
t = 0.2 s
To find:
e
Formula:
A = l * b
e = (2 * N * B * A) / t
"""

What is Physicaly sound formula slot?

Most of the time, LLMs do not output the exact formulas found in the ground truth; however, the formulas they provide are still correct. 'Physically sound' means that any formula generated by an LLM—whether written symbolically or described in words—is scientifically accurate and can be used to solve the given physics problem.

---

General Observations:

1. Models with size more than 2B showed higher accuracy (more than 80%) on given and to-find fields. (Reason: given and to-find are alsmost always mentioned in the question text itself, and language models don't need to take extra efforts/thinking to extract them.)
2. None of the model could pass 80% accuracy on physically sound slots with Qwen3.5-4B showing highest accuracy (78.5%).

Further analysis done only on formula field because it is the most challenging field for language models and also the objective of this evaluation.

The following types of analysis were performed:

1. How accuracy (physically soundness) increases with increase in model size.
2. Domain based accuracy analysis across all the SLMs.
3. How problem difficulty (easy, medium, and hard) affects the accuracy of the models.
4. To understand the effect of different architectural and training innovations, I also studied how release date affects the accuracy.

4. Formula Field Scaling Analysis

Understanding and generating physically sound mathematical equations requires physics understanding, (also symbolic understanding, i.e. linking symbols to their physical definitions) and multi-step reasoning. So, this is not just language modeling where SLMs are already good at (summarization, grammar correction, sentiment analysis, etc.).

To analyze how model capacity influences formula generation, I studied:

1. In-Family Scaling: Tracking performance as parameters increase within the Qwen3, Qwen3.5, Gemma3, and Gemma4 families.
2. Cross-Family Scaling: Plotting all 12 SLMs (from 0.6B to 4.0B parameters) to identify scaling laws.

4.1 In-Family Scaling (Qwen3, Qwen3.5, Gemma3, Gemma4)

In-Family Observations:

1. All Qwen3, Qwen3.5, Gemma3, and Gemma4 family models show improvement in accuracy (physically soundness) with increasing size from sub-1B to 4B.
2. Performance trajectories:
   • Qwen3 improves from 5.6% at 0.6B to 39.2% at 1.7B (a 7.0× increase), reaching 61.7% at 4B (a 11× increase).
   • Qwen3.5 improves from 9.3% at 0.8B to 45.8% at 2B (a 4.9× increase), reaching the benchmark high of 78.5% at 4B (a 8.5x increase).
   • Gemma-3 improves from 13.1% at 1B to 41.1% at 4B (a 3.1x increase).
   • Gemma-4 improves from 50.5% at 2B-effective to 63.5% at 4B-effective (a 1.3x increase).
3. With increase in size, hallucination rate dropped. In sub-1B models, over 92% to 94% of predicted formulas are redundant or hallucinated equations, whereas at 4B, unnecessary formulas fall to 33% – 45%.

---

4.2 Cross-Family Scaling Across All 12 Models

When all 12 SLMs are plotted based on their physically sound slots from 0.6B to 4.0B parameters, three distinct capability regimes emerge:

1. Sub-1B "Failure Regime" (≤ 1.0B):
   • Models: Qwen3-0.6B, Qwen3.5-0.8B, Gemma-3-1B-IT.
   • Performance: 2% to 10% physically sound questions; >81% extraneous equations.
   • Observation: Sub-1B models frequently generate disconnected symbols and hallucinated arithmetic relations.
   • Diagnosis: Sub-1B models lack sufficient parameter capacity to retain physical formula definitions while simultaneously generating structured JSON/markdown extraction.
2. The 1.5B – 3.8B "Transition Plateau" (40% – 54%):
   • Models: Qwen3-1.7B (39.3%), Gemma-3-4B-IT (41.1%), Qwen3.5-2B (45.8%), Gemma-4-E2B-IT (50.5%), SmolLM3-3B (54.2%), Phi-4-mini-instruct (60.8%).
   • Performance: These models identify standard single-step formulas (e.g., V = IR, ρ = RA/L), but struggle with multi-step chained equations.
3. 4B "Frontier Tier" (60% – 80%):
   • Models: Qwen3-4B (61.7%), Gemma-4-E4B-IT (63.5%), and Qwen3.5-4B (78.5%).
   • Performance: Scaling to 4B parameters does not consistently produce strong formula reasoning performance. Qwen3-4B and Gemma-4-E4B-IT remain at only 61.7% and 63.5%, respectively. This indicates limitations in generating physically sound formula sequences even at this model size. In contrast, Qwen3.5-4B is a clear outlier, achieving 78.5%, outperforming the other 4B models. Thus, the results suggest that model capability and training matter more than parameter count alone for sound multi-step formula reasoning.

---

4.3 Physically Sound Formula Slots Ordered Progression (Size Ascending, Release Date Ascending)

Evaluating models on "Physically Sound Questions" treats questions as binary pass/fail—if a question requires three formulas and a model generates two sound formulas but one extraneous/incorrect one, the entire question is marked as incorrect. Therefore, examining "Physically Sound Formula Slots" (out of 107 ground truth slots across all 50 questions) provides better view of model's physics knowledge.

Below, all 12 models are ordered strictly by model parameter size (smallest first), with ties broken by release date (oldest first):

Key Insights from Slot-Level Ordered Analysis:

Architectural Leap in the 4B Cluster (Temporal Progression):

• When parameter size is held constant at 4.0B, sorting by release date shows continuous generational leaps in models' physics reasoning ability:
  • Mar 12, 2025 (Gemma-3-4B-IT): 44 / 107 slots (41.12%)
  • Apr 29, 2025 (Qwen3-4B): 66 / 107 slots (61.68%)
  • Mar 2, 2026 (Qwen3.5-4B): 84 / 107 slots (78.50%)
  • Apr 2, 2026 (Gemma-4-E4B-IT): 68 / 107 slots (63.55%)
• This proves that while model size provides the computational capacity, architecture and data evolution drive major capability jumps:
  • Qwen generational jump (4B): Qwen3-4B (61.68%) → Qwen3.5-4B (78.50%) represents a +16.82% absolute gain.
  • Gemma generational jump (4B): Gemma-3-4B (41.12%) → Gemma-4-E4B (63.55%) delivers a +22.43% absolute gain.

---

4.4 Domain-Wise Formula Soundness Breakdown (Mastery vs Failure Domains)

To find whether SLMs have domain-specific mastery or uniform physical reasoning gaps, Physically Sound Formula Slots across all 10 physics domains were evaluated (5 questions per domain, 107 total slots).

Key Domain Insights:

1. Top-Performing Domains:

   • Electronics & Communication (53.3% Avg), Mechanics (50.8% Avg), and Waves & Acoustics (49.2% Avg) are the most robust domains across all 12 SLMs.
   • Standard mechanics and semiconductor problems have formulas (e.g., v = u + at, F = ma, V = IR) that appear extensively in pre-training corpora with low symbol ambiguity.
   • Qwen3.5-4B, Qwen3-4B, and Gemma-4-E4B-IT achieve 70% to 82% soundness on these domains.

2. Hardest "Failure" Domains:

   • Electrostatics & Current Electricity (35.4% Avg), Properties of Matter (38.6% Avg), and Thermal Physics (39.2% Avg) showed lowest accuracy.
   • These domains require multi-variable substitutions (e.g., surface tension γ = F / (2L), terminal velocity vt = 2r²(ρ - σ)g / (9η), latent heat, drift velocity I = n e A vd). Models frequently confuse coefficients, miss constant multipliers, or invert ratios.

3. Domain-Specific Breakthroughs at 4B Scale:

   • Modern Physics: Qwen3.5-4B achieves 100% (8/8 slots) sound formula recall on Modern Physics (photoelectric effect E = hν - Φ, de Broglie λ = h/p), while Qwen3-4B and Gemma-4-E4B-IT both achieve 75.0% (6/8 slots), whereas sub-1B models scored 0% to 38%.
   • Gemma-4-E4B-IT Cross-Domain Balance: Gemma-4-E4B-IT achieves ≥ 70% soundness across 5 domains: Waves & Acoustics (80.0%), Electromagnetic Induction & AC (77.8%), Modern Physics (75.0%), Properties of Matter (72.7%), and Electronics & Communication (70.0%).

---

4.5 Formula Soundness Across Question Difficulties (Easy, Medium, Hard)

The 50 evaluation questions span three different difficulty levels:

• Easy (18 questions, 36 formula slots): Single-concept, direct standard formula recall (e.g., V = IR, F = ma).
• Medium (24 questions, 50 formula slots): Multi-step derivations, parameter rearrangement, or intermediate quantity calculations.
• Hard (8 questions, 21 formula slots): Multi-concept, non-trivial algebraic steps, or specialized physical constants.

Key Difficulty Observations:

1. The "Hard" Question Collapse in Sub-3B Models:
   • On Hard questions, every model below 2B scores under 15% (Qwen3-0.6B at 4.8%, Qwen3.5-0.8B at 0.0%, Qwen3-1.7B at 14.3%).
2. 4B Models Break the Hard Barrier:
   • Qwen3.5-4B maintains consistency across difficulty: 86.1% on Easy, 76.0% on Medium, and 71.4% on Hard (15 / 21 slots; 6 / 8 questions).
   • Gemma-4-E4B-IT Did well on easy but performance dropped significantly on harder questions, scoring 75.0% Easy, 60.0% Medium, and 52.4% Hard (11 / 21 slots; 4 / 8 questions), a dramatic advancement over Gemma-3-4B-IT (14.3% Hard slots).
   • Qwen3-4B follows with 75.0% Easy, 58.0% Medium, and 47.6% Hard.

Problems with the Thinking Models:

After testing various thinking versions of the above model (wherever applicable, by turning thinking mode on), we found multiple issues. Most important was models going out of track while thinking for a long time. In many cases models generated hallucinated formulas, repeated outputs, stopped generating text in the middle of thinking, etc. Therefore, I did not tested thinking models. Also, generating response using thinking mode is time consuming.

Conclusion:

Overall, small language models still struggle with physics reasoning, especially on complex and multi-step problems. However, newer models are showing clear improvements, even within the 4B parameter range. Some questions that were difficult for 2025 models are now being solved by newer 2026 models. However, no model yet performs consistently well across all difficulty levels and physics domains.

---

5. Future Direction

Results suggests that, even 4B models (except Qwen3.5) struggle to achieve high accuracy on physically sound formula. At the same time, over the period of 1 years, there is improvement in 4B model itself. This implies that, architectural innovation, data reciepe matters more than just compute and model size.

5.1 Architecture:

1. Recently I came across the paper : "Less is More: Recursive Reasoning with Tiny Networks" (https://arxiv.org/abs/2510.04871). This paper talks about Hierarchical Reasoning Model (HRM), which uses two small neural networks recursively at different frequencies to solve Sudoku, Maze, and ARC-AGI problems and beats much larger models. I believe that we can use this architecture for physics problem solving.
2. Physics problem solving is not just language modelling, it requires physics understanding. Instead of generating text token-by-token, a language-based JEPA which operates entirely in concept space seems promising. (https://arxiv.org/abs/2509.14252)
3. Thinking happens at different level, slow and fast, and there are new architectures which mimic this. For solving physics problems, we need lower level representation for symbols, their relations and higher level representations for concept understanding and problem solving. We can use HRM model for text and symbol level representations. (https://arxiv.org/abs/2506.21734)

5.2 Finetuning/Distillation:

1. I tested the knowledge distillation (logit and ground-truth based) from Gemma3:27B-IT to Gemma3:1b-IT, but results are not promising. I found that though there is high top-1 (91%), top-5 (98%) and top-10 (99%) token agreement between student and teacher, their internal representations looks completely different. Highest Centered Kernel Alignment (CKA) was 0.30 across 26 student layers. Also, maximum attention overlap between student and teacher model was about 50%.
2. Therefore, logit and ground-truth based distillation is not sufficient, and we need to consider attention maps and hidden states to calculate the loss. But with limited compute, I could not perform this distillation. Below is the diagram of the distillation process:

            TEACHER
               │
   ┌───────────┼────────────┐
   │           │           │
 logits     attention   hidden states
   │           │           │
   ▼           ▼           ▼
KL + CE    Attn Loss    Feature/CKA Loss
   │           │           │
   └───────────┼────────────┘
               ▼
         STUDENT TRAINING

5.3 Training Data Recipe:

1. Simplest way to improve the accuracy on physics problems is by increasing the proportion of physics data at different stages of training.
2. Also, we can train entire SLM on physics data only during Pretraining, SFT-IFT, and DPO. But this will require a large amount of data. Here, HRM architecture can help which can be train using less data.

---

5.4 Conclusion & Outlook:

In conclusion, this study demonstrates that small language models continue to struggle with complex physics problems. However, recent advancements in 4B-parameter architectures show slight progress. The findings indicate that model size alone is not the only factor; rather, architectural design and data quality play a important role. Distillation experiments shows that simply matching output tokens is not sufficient for transferring deep conceptual understanding, as the internal representations remain mostly different. Therefore, future efforts should focus on combining Hierarchical Reasoning Models, with representation-level distillation and physics-rich training data. This approach offers a practical pathway toward developing lightweight models that can reason accurately about the physical world.

---

Analysis done by: Bhushan Deshpande (bhushan1729dsp@gmail.com)

Date: 2026-10-09

Link to bucket: https://huggingface.co/buckets/bhushan1729/physics_world_understanding