LFM2.5-350M-code_gen-finetuned

Fine-tuned version of LiquidAI/LFM2.5-350M for Python code generation from natural language instructions.

Generates clean, PEP 8 compliant Python code with docstrings and error handling.

Model Details

Attribute	Value
Base Model	LiquidAI/LFM2.5-350M
Parameters	350M (full), ~2M trainable (LoRA)
Architecture	Hybrid (10 LIV convolution + 6 GQA blocks)
Context Length	32,768 tokens
Fine-tuning Method	QLoRA (4-bit NF4)
LoRA Rank	16
LoRA Alpha	32
Dataset	TokenBender/code_instructions_122k_alpaca_style
Epochs	3
Learning Rate	2e-4
Optimizer	paged_adamw_32bit

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model
tokenizer = AutoTokenizer.from_pretrained("LiquidAI/LFM2.5-350M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "LiquidAI/LFM2.5-350M",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(model, "rohitnagareddy/LFM2.5-350M-code_gen-finetuned")

# Generate code
messages = [
    {"role": "system", "content": "You are an expert Python programmer. Write clean, efficient, and well-documented code."},
    {"role": "user", "content": "Write a function that finds the longest palindromic substring"},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.9, repetition_penalty=1.3, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))

Training Details

• LoRA Target Modules: q_proj, k_proj, v_proj, o_proj
• LoRA Dropout: 0.05
• Gradient Accumulation: 4 steps
• Scheduler: Cosine with 3% warmup
• Max Gradient Norm: 1.0
• Weight Decay: 0.01
• Quantization: 4-bit NF4 with double quantization

Intended Use

Python code generation from natural language descriptions. Useful for:

• Quick function prototyping
• Code completion assistance
• Learning Python patterns
• Boilerplate generation

Limitations

• 350M parameters — generates simple-to-medium complexity code
• May produce syntactically correct but logically flawed solutions for complex algorithms
• Not a replacement for human code review
• Knowledge cutoff: mid-2024

Author

Fine-tuned by rohitnagareddy

License

MIT License