arabellako22/gifted_maths_lora_model

Gifted Maths LoRA Adapter

A Gemma 4 E2B LoRA adapter for clear, step-by-step mathematics solutions in LaTeX. This repository contains adapter weights, not a standalone model.

Key facts

Field	Value
Developer	arabellako22
Training base	unsloth/gemma-4-E2B-it, loaded in 4-bit
PEFT base alias	unsloth/gemma-4-e2b-it-unsloth-bnb-4bit
Task	Step-by-step mathematics reasoning
Language	English
Method	SFT with response-only loss; LoRA via Unsloth, TRL, and PEFT
License	Gemma license

Use the PEFT base alias above for standard PeftModel loading.

Training system prompt

You are maths_engine, a local gifted mathematics reasoning model for students.
You solve maths problems step by step, clearly and calmly.
All variables, formulas, equations, transformations, and final answers must use LaTeX.
Use inline math with $...$ and display math with $$...$$.
Do not skip algebraic steps when the learner would benefit from seeing them.
End every solution with a clear line starting with: Final Answer:

Use with Unsloth

pip install -U unsloth transformers peft accelerate bitsandbytes safetensors

import torch
from unsloth import FastModel

ADAPTER_ID = "arabellako22/gifted_maths_lora_model"

model, tokenizer = FastModel.from_pretrained(
    model_name=ADAPTER_ID,
    max_seq_length=1024,
    dtype=None,
    load_in_4bit=True,
)
FastModel.for_inference(model)

system_prompt = """You are maths_engine, a local gifted mathematics reasoning model for students.
You solve maths problems step by step, clearly and calmly.
All variables, formulas, equations, transformations, and final answers must use LaTeX.
Use inline math with $...$ and display math with $$...$$.
Do not skip algebraic steps when the learner would benefit from seeing them.
End every solution with a clear line starting with: Final Answer:"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Solve $3x-7=5x+19$ and show each step."},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=500,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = outputs[0][inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Training data

The notebook loaded 50 original chat examples from Gifted_Maths_Data.txt and added 12 worked problems. Normalization, question-based deduplication, and validation produced 61 rows.

Topics include algebra, number theory, calculus, recurrence relations, probability, geometry, linear systems, modular arithmetic, and areas between curves.

Quality checks required each row to contain one user message followed by one assistant message, a Final Answer: line, and at least one LaTeX delimiter.

Split	Rows	Share
Train	54	88.5%
Validation	7	11.5%

The split used seed 3407; no independent test set was created. The snapshot is stored in the currently private training dataset repository. The provenance and licensing of the original 50 examples were not recorded.

The Gemma 4 chat template was applied to each row. System and user tokens were masked; only assistant responses contributed to loss. Sequence packing was disabled.

Training configuration

Parameter	Value
LoRA rank / alpha / dropout	64 / 64 / 0
Trainable parameters	101,351,424 of 5,224,529,440 (1.94%)
Sequence length	1,024
Batch / accumulation	1 / 8
Epochs / optimizer steps	3 / 21
Learning rate	2e-5, cosine schedule
Optimizer	adamw_8bit
Warm-up / weight decay	0.08 / 0.01
Hardware	NVIDIA A100-SXM4-80GB, BF16
Runtime	201.34 seconds

Evaluation

Metric	Recorded value
Aggregate training loss	1.01385
Validation losses	1.89183 → 1.85228 → 1.85170

Validation loss improved slightly but remained above training loss. A seven-example validation set, without an independent test set, is insufficient to establish general mathematical accuracy.

Smoke tests checked adapter loading, Unsloth merging, GGUF conversion, upload, and re-download. The verified merged model returned 2160 for the coefficient of $x^4$ in $(2x-3)^6$; the Q4_K_M GGUF returned $x=\frac{1}{2}$ or $x=-4$ for $2x^2+7x-4=0$. These are artifact checks, not benchmark results.

Related artifacts

Artifact	Repository
Dataset snapshot	arabellako22/gifted_maths_training_dataset
Verified 16-bit merge	arabellako22/gifted_maths_unsloth_remerged_full_gemma_4
Verified Q4_K_M / Q5_K_M GGUF	arabellako22/gifted_maths_gemma_4_gguf_v2
Legacy GGUF export (not recommended)	arabellako22/Gifted-Maths-Gemma-4_legacy

Recorded GGUF sizes are 3.428 GB (Q4_K_M) and 3.630 GB (Q5_K_M). Use the verified Unsloth merge as the conversion source.

Intended use

• Worked examples and local mathematics-assistant experiments
• Clear intermediate algebra and LaTeX-formatted solutions
• Research or teaching experiments with a small domain-specific adapter

Limitations

• Very small dataset; memorization and brittle solution patterns are likely
• Only seven validation rows and no independent test or standardized benchmark
• Original 50 examples have undocumented provenance and license
• Generated algebra, arithmetic, proofs, notation, or LaTeX may be wrong
• “Gifted” describes the project goal, not a validated learner classification

Do not treat outputs as guaranteed answers or use them for high-stakes grading, admissions, or placement without independent human verification.

Artifact notes

• Adapter: adapter_model.safetensors
• LoRA rank 64, alpha 64, dropout 0
• PEFT 0.19.1
• Recorded report: training_report.json