Symbolic-3b-GGUF : GGUF

This model was finetuned and converted to GGUF format using Unsloth.

Example usage:

• For text only LLMs: llama-cli -hf JPQ24/Symbolic-3b-GGUF --jinja
• For multimodal models: llama-mtmd-cli -hf JPQ24/Symbolic-3b-GGUF --jinja

Available Model files:

• llama-3.2-3b-instruct.Q4_K_M.gguf

Ollama

An Ollama Modelfile is included for easy deployment. This was trained 2x faster with Unsloth

Symbolic is an instruction-tuned, quantized variant of the Llama 3.2 3B architecture. It is optimized for structured output generation, roleplay, precise target-audience adaptation, and clean code output.

This model demonstrates a significant performance shift compared to the vanilla baseline, offering highly competitive commonsense reasoning scores that punch well above its parameter class, while trading off some complex logic capabilities due to quantization and alignment constraints.

---

Evaluation Results

We evaluated this model using the tinyBenchmarks framework (utilizing Item Response Theory IRT++ calculations to estimate full benchmark performance) via lm-evaluation-harness.

Benchmark / Task	Llama-3.2-3B-Symbolic	Official Base 3B Baseline	Official Instruct 3B Baseline	Performance Delta (vs. Instruct)
tinyMMLU (0-shot vs 5-shot)	63.25% (0.6325)	63.40%	63.40%	-0.15% (Maintained)
tinyHellaswag (10-shot)	75.31% (0.7531)	69.80%	69.80%	+5.51% (Improved)
tinyArc (25-shot)	55.31% (0.5531)	69.10%	78.60%	-23.29% (Degraded)

Performance Analysis

1. Linguistic Reasoning & Commonsense (HellaSwag): At 75.31%, Symbolic outperforms both its vanilla 3B baseline and several larger models (e.g., Gemma 2 2B at 73%). It operates near the performance tier of much larger 7B to 8B parameter models (such as Llama 3.1 8B at ~76.8%).
2. Academic Knowledge (MMLU): At 63.25% under a strict 0-shot constraint, the model retains virtually all of its core factual knowledge compared to the baseline's 5-shot score of 63.40%.
3. Logic & Hard Reasoning (ARC): The model suffered a notable performance drop to 55.31% on ARC-Challenge. This represents a classic tradeoff: both low-bit quantization and specialized instruction-tuning often degrade multi-step, scientific logic retrieval.

---

Qualitative Strengths & Limitations

Qualitative evaluation testing against the vanilla baseline reveals distinct behavioral shifts:

Strengths

• Superior Persona & Audience Adaptation: Symbolic exhibits strong cognitive scaling. When asked to explain complex topics (e.g., inflation) to different audiences, it successfully navigates simple analogies for children and highly precise macroeconomic strategies for domain experts without introducing factual contradictions.
• Formatting and Structure Adherence: Symbolic exhibits high compliance with structural constraints (e.g., exact bullet counts, specific hashtag limits, and paragraph configurations).
• Cleaner, Pythonic Code Synthesis: The model produces clean, idiomatic code structures. It avoids unnecessary library imports (such as using standard string manipulation over the re module for basic parsing) and shows an advanced semantic understanding of syntax-level bugs (like the difference between =+ and += in Python AST).

Known Limitations

• Negative Constraints Susceptibility: Like many 3B models, high-difficulty negative constraints (e.g., "Write an essay about X but do not use the word Y or Z in paragraph 2") can severely alter token-probability distributions, sometimes causing the model to stall or refuse to generate a response.
• Physical State Tracking: In multi-step logical planning puzzles (e.g., the Wolf, Goat, and Cabbage river crossing puzzle), the model's working memory is physically constrained by its 3B parameter envelope. While it retains highly coherent grammar, it can lose track of sequential state variables in long-form generation.

Addendum: Empirical Evaluation of Specialized Capabilities & Limitations

To further map the boundary limits of this specialized 3B parameter model, we conducted targeted evaluations across three core domains: Common Sense / Discrete Logic, Knowledge Explanation & Reading Comprehension, and Creative Inventions.

Below is an empirical analysis of the behavioral strengths and limitations observed during these tests.

Key Strengths

1. Grounded Reading Comprehension & Text-Constraint Adherence

   • Observation: When presented with a technical paragraph about machining titanium, the model successfully extracted the exact thermodynamic cause of tool wear without introducing external knowledge or hallucinated details.
   • Takeaway: The model demonstrates strong utility for Retrieval-Augmented Generation (RAG) tasks where strict adherence to context and minimization of hallucination are required.

2. Calibrated Persona Translation & Metaphor Mapping

   • Observation: The model translated the complex quantum mechanics concept of Schrödinger's Cat into a simplified, age-appropriate metaphor (a hidden birthday present) for a 10-year-old child. It successfully maintained the structural integrity of the analogy and used it to answer subsequent comprehension questions accurately.
   • Takeaway: This indicates highly effective instruction tuning for educational tools, interactive assistants, and multi-turn persona-driven chats.

3. Discrete Interval & Sequence Reasoning

   • Observation: The model correctly solved the classic "3-pill interval" riddle (taking a pill every half hour), identifying that there are only two intervals over three events to output 60 minutes rather than the common 90-minute error.
   • Takeaway: The model successfully navigates discrete, step-by-step mathematical reasoning when the logical structure is sequence-bound.

4. Structured Creative Synthesis

   • Observation: When designing a futuristic underwater transit system, the model produced a highly organized proposal. It demonstrated strong semantic mapping by leveraging actual technologies, such as Ocean Thermal Energy Conversion (OTEC), to solve setting-specific constraints.

---

Key Limitations

1. Shallow Physical Commonsense Simulation
   • Observation: During a creative task to invent a hybrid toaster/paper-shredder, the model successfully followed all structural constraints but failed to identify the most obvious physical hazard: the high risk of fire from shredded paper contact with heating coils. Instead, it focused on lower-priority risks like paper-shrub ingestion and minor burns.
   • Takeaway: While the model is highly capable of generating creative, structured text, its ability to simulate real-world physical dynamics and chemical cause-and-effect remains superficial.

---

Deployment Recommendations based on Findings

• Optimal Use Cases: Structured extraction (JSON, Markdown), educational explanations, RAG pipelines with strict negative constraints, and guided creative brainstorming.
• Sub-Optimal Use Cases: Safety-critical risk assessments, open-ended physical simulation, or complex multi-step logistical planning.