| --- |
| license: other |
| license_name: lfm1.0 |
| license_link: https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/LICENSE |
| base_model: LiquidAI/LFM2.5-350M |
| tags: |
| - lfm2 |
| - lfm2.5 |
| - liquid |
| - python |
| - math |
| - fine-tune |
| language: |
| - en |
| pipeline_tag: text-generation |
| --- |
| |
| # LFM2.5-350M-Python-Math |
|
|
| A fine-tune of [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) (instruct) focused on **Python code generation** and **math word-problem solving**, while retaining general chat ability through a balanced mixed dataset. |
|
|
| ## Why this exists |
|
|
| The previous 230M fine-tune (`lfm2.5-230m-code-math`) showed strong potential but suffered from catastrophic forgetting (e.g., confusing baking cookies with HTTP cookies, failing negative constraints like "no dairy"). This 350M version addresses those issues by: |
|
|
| 1. **Mixing general chat data** (`yahma/alpaca-cleaned`, 30k samples) to prevent knowledge loss. |
| 2. **Injecting custom fix-it examples** targeting specific failure modes (negative constraints, complete Pygame scripts). |
| 3. **Using longer context** (2048 tokens) so code outputs aren't truncated mid-function. |
| 4. **Reducing epochs to 2** with a lower learning rate (`2e-5`) to prevent overfitting observed in earlier runs. |
|
|
| Fine-tuning started from the **instruct checkpoint** rather than base. Testing confirmed that at 350M scale, starting from base with a mixed dataset still produced alignment failures (refusals, identity confusion, math regression), while the instruct checkpoint with the same dataset produced consistently strong results. |
|
|
| ## Training details |
|
|
| - **Base model:** `LiquidAI/LFM2.5-350M` (instruct) |
| - **Method:** Full fine-tune (96GB VRAM, no LoRA needed) |
| - **Datasets:** |
| - Python Code: [`iamtarun/python_code_instructions_18k_alpaca`](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca) (Python-focused, replacing the multi-language 120k set) |
| - Math: [`openai/gsm8k`](https://huggingface.co/datasets/openai/gsm8k) (main split) |
| - General Chat: [`yahma/alpaca-cleaned`](https://huggingface.co/datasets/yahma/alpaca-cleaned) (30k sample subset) |
| - Custom Fix-It: Hand-crafted examples for negative constraints ("no dairy", "no eggs") and complete runnable Pygame scripts (duplicated 50x for weight) |
| - **Checkpoint selection:** Best by eval_loss |
| - **Sequence length:** 2048 tokens (increased from 1024 to accommodate full scripts) |
| - **Max response chars:** 3500 (prevents code truncation) |
| - **Epochs:** 2 (reduced from 4; overfitting observed past epoch 2 in prior runs) |
| - **Learning rate:** 2e-5 (reduced from 5e-5 for 350M stability) |
| - **Loss:** Completion-only |
| |
| ## What it's good at |
| |
| - **Python Code**: Complete, runnable scripts including Pygame game loops, file I/O, classes, list comprehensions, and algorithmic implementations (e.g., two-pointer palindrome check). No more placeholder `pass` statements or truncated functions. |
| - **Math**: GSM8K-style word problems with step-by-step reasoning annotations (`<<...>>`). Reliable on algebra, percentages, geometry, and multi-step arithmetic. |
| - **General Chat**: Retains coherent conversational ability. Correctly handles negative constraints (e.g., "breakfast without eggs" returns egg-free options). Knows the difference between baking cookies and browser cookies. |
| - **Speed**: At 350M parameters, achieves ~157 t/s generation on laptop CPU (i5-12450H) with Q5_K_S quantization via llama.cpp. |
| |
| ## Known limitations |
| |
| - **Python only**: Trained exclusively on Python code instructions. Other languages were not included in this fine-tune. |
| - **Sentence counting**: May not strictly adhere to "exactly N sentences" constraints. |
| - **Identity**: May occasionally claim to be developed by Google (artifact from Alpaca-Cleaned training data). |
| - **Still 350M parameters**: Do not expect deep multi-step reasoning or long-form creative writing at the level of larger models. |
| - Not evaluated on safety-critical, medical, or legal use cases. |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "hauser458original/lfm2.5-350m-python-math" |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto") |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| |
| messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}] |
| inputs = tokenizer.apply_chat_template( |
| messages, add_generation_prompt=True, return_tensors="pt", return_dict=True |
| ).to(model.device) |
| |
| output = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.5, top_p=0.9) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
| |
| GGUF quantized versions (Q4_K_M, Q5_K_S, Q5_K_M, Q8_0, F16) for llama.cpp/Ollama/LM Studio are available at: [`hauser458original/lfm2.5-350m-python-math-GGUF`](https://huggingface.co/hauser458original/lfm2.5-350m-python-math-GGUF) |
| |
| ## License |
| |
| Inherits the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/LICENSE) from the base model. |
| |
| ## Acknowledgements |
| |
| Built on [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M). See the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404) for details on the base architecture. |
