Walid692/agentforge-mistral-7b-qlora

AgentForge — QLoRA Fine-tuned Mistral 7B

Fine-tuning of Mistral-7B-Instruct-v0.2 with QLoRA (PEFT) on conversational AI data.

Model Description

This model is part of the AgentForge portfolio project — a production-grade multi-agent LLM system with RAG, agents, and fine-tuning.

• Base model: mistralai/Mistral-7B-Instruct-v0.2
• Fine-tuning method: QLoRA (4-bit quantization + LoRA)
• Dataset: HuggingFaceH4/ultrachat_200k (500 samples)
• Training: Google Colab T4 GPU

Training Results

Epoch	Train Loss	Val Loss
1	-	1.4584
2	1.2800	1.4662
3	1.2800	1.4771

Training Details

• LoRA rank: 8
• LoRA alpha: 16
• Target modules: q_proj, v_proj
• Epochs: 3
• Batch size: 2
• Learning rate: 2e-4

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Walid692/agentforge-mistral-7b-qlora")
model = AutoModelForCausalLM.from_pretrained("Walid692/agentforge-mistral-7b-qlora")

prompt = "[INST] What is RAG and how does it work? [/INST]"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Author

Ahmed Walid BOUANZOUL — AI Engineer

• GitHub: AhmedWalidbou
• HuggingFace: Walid692
• Project: AgentForge