kshitijthakkar/lfm-finetuned

LFM2.5-1.2B-Instruct Fine-tuned on FineTome-100k

A LoRA fine-tune of LiquidAI/LFM2.5-1.2B-Instruct on mlabonne/FineTome-100k for improved instruction following and conversational ability.

Trained using Unsloth SFT on Hugging Face Jobs.

Training Details

Parameter	Value
Base model	LiquidAI/LFM2.5-1.2B-Instruct (1.17B params)
Method	SFT with LoRA via Unsloth
Dataset	mlabonne/FineTome-100k (80k train / 20k eval)
Hardware	NVIDIA A10G on HF Jobs
Epochs	1
Batch size	2 x 4 gradient accumulation = 8 effective
Learning rate	2e-4
Max sequence length	2048
Total steps	9,991

LoRA Configuration

Parameter	Value
Rank (r)	16
Trainable parameters	11,108,352 / 1,181,448,960 (0.94%)

Training Metrics (from TensorBoard)

Step	Loss	Grad Norm	Learning Rate	Epoch
1,000	0.6984	0.349	1.80e-4	0.10
2,000	0.6898	0.298	1.60e-4	0.20
3,000	0.6696	0.266	1.40e-4	0.30
4,000	0.6694	0.523	1.20e-4	0.40
5,000	0.6697	0.356	1.00e-4	0.50
6,000	0.6766	0.367	8.00e-5	0.60
7,000	0.6574	0.426	6.00e-5	0.70
8,000	0.6562	0.387	4.00e-5	0.80
9,000	0.6673	0.516	2.00e-5	0.90

Final training loss: 0.6562 (at step 8,000). Loss decreased from 0.6984 to 0.6562 over the course of training (~6% reduction).

Usage

With PEFT + Transformers

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2.5-1.2B-Instruct")
model = PeftModel.from_pretrained(base_model, "kshitijthakkar/lfm-finetuned")
tokenizer = AutoTokenizer.from_pretrained("kshitijthakkar/lfm-finetuned")

messages = [
    {"role": "user", "content": "Explain the theory of relativity in simple terms."}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With Unsloth (faster inference)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="kshitijthakkar/lfm-finetuned",
    max_seq_length=2048,
)
FastLanguageModel.for_inference(model)

Intended Use

• General instruction following and conversational tasks
• Fine-tuned on a curated subset of high-quality instruction data (FineTome-100k)

Limitations

• Trained for 1 epoch; further training may improve results
• Inherits limitations from the base LFM2.5-1.2B-Instruct model
• Performance on domain-specific tasks may vary

Training Infrastructure

Trained on Hugging Face Jobs using the Unsloth SFT training script with an NVIDIA A10G GPU.