StudyAbroadGPT-7B-LoRa-Kaggle

Parameter-efficient LoRA-adapted Mistral-7B-Instruct-v0.3 fine-tuned on synthetic study-abroad conversational data for domain-specific academic advising guidance.

Status: Generation and lightweight qualitative evaluation complete ✅ | Manual blinded scoring pending ⏳ | Factuality audit pending ⏳

🔗 Project Ecosystem

Resource	Link
Dataset	millat/StudyAbroadGPT-Dataset
Training Code	codermillat/StudyAbroadGPT
Dataset Generation	codermillat/study-abroad-dataset
Evaluation Artifacts	LoRA Paper evaluation workspace
Research Paper	arXiv:2504.15610
Author ORCID	0009-0005-7198-9893

📊 Model Details

Architecture

Component	Specification
Base Model	mistralai/Mistral-7B-Instruct-v0.3
Base Model Size	7 billion parameters
Quantization	4-bit NF4 (via Unsloth)
Fine-Tuning Method	LoRA (Low-Rank Adaptation)
LoRA Rank (r)	16
LoRA Alpha (α)	32
Scaling Factor	α / r = 2.0

Trainable Parameters

• Total Model Parameters: ~7B
• LoRA Trainable Parameters: ~4.7M (0.07% of model)
• LoRA Adapters: Applied to Q, K, V, O projections (attention) + FFN layers (gate, up, down)

Quantization Details

• Method: 4-bit NF4 quantization via Unsloth
• Benefit: Reduces model memory from ~14GB (fp16) to ~8GB
• Trade-off: Minimal impact on model quality with significant memory savings
• Hardware Requirement: 16GB+ VRAM GPU (e.g., Tesla T4, P100)

🎯 Training Details

Training Data

• Dataset: millat/StudyAbroadGPT-Dataset
• Training Samples: 2,274 conversations
• Test Samples: 402 conversations
• Average Turns/Conversation: 5.2
• Topics Covered: Admissions, scholarships, visas, accommodation, cultural adaptation

Training Configuration

Parameter	Value
Batch Size	2 per device
Gradient Accumulation	4 steps
Learning Rate	2e-4
Warmup Ratio	0.03
Number of Epochs	4
Max Sequence Length	2048 tokens
Optimizer	adamw_8bit (8-bit optimizer)
Max Gradient Norm	0.3
Learning Rate Scheduler	Linear
Effective Batch Size	8 (2 × 4)

Hardware & Resources

Setting	Value
GPU Tested	Tesla T4 (Kaggle), Tesla P100 (Kaggle)
RAM	16GB
Training Time (T4)	~3-4 hours
Training Time (P100)	~1-2 hours
Monitoring	Weights & Biases (WandB)

🚀 Usage

Option 1: Using Unsloth (Recommended for Inference)

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    max_seq_length=2048,
    dtype=torch.float16,
    load_in_4bit=True,
)

# Prepare for inference
FastLanguageModel.for_inference(model)

prompt = "What documents do I need for a UK student visa?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.0,  # Deterministic
        do_sample=False,
        top_p=1.0
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Option 2: Using Transformers Library

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged"
)

prompt = "How much should I budget for accommodation in London?"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.0,
    do_sample=False
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Option 3: Using LoRA Adapter (Training)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype="auto",
    device_map="auto"
)

# Load LoRA adapters
model = PeftModel.from_pretrained(
    base_model,
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle"
)

tokenizer = AutoTokenizer.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3"
)

# For continued training, unfreeze LoRA parameters
model.training_mode = True

Option 4: Local Inference with Ollama/GGUF

The merged weights can be quantized to GGUF format for local inference:

# Clone the model
git clone https://huggingface.co/millat/StudyAbroadGPT-7B-LoRa-Kaggle

# Convert to GGUF (requires llama.cpp tools)
python convert.py studyabroadgpt-merged/

# Run locally with Ollama
ollama run studyabroadgpt-7b-merged

📊 Lightweight Evaluation Results

From companion 50-sample deterministic evaluation run:

Generation Metrics

Metric	Base Model	LoRA Model	Δ
Avg Response Length (chars)	1151.88	1178.74	+26.86
Avg Response Tokens	252.26	254.72	+2.46
Avg Generation Time (sec)	20.92	23.98	+3.06
Truncation Rate	0.96	0.96	0.0

Domain-Specific Term Coverage

Term	Base	LoRA	Change
University	48%	42%	-6%
Tuition	12%	22%	+10%
Scholarship	20%	16%	-4%
Visa	8%	6%	-2%
IELTS	6%	10%	+4%

Format Quality

Format Indicator	Base	LoRA
Bullet/List Usage	92%	96%
Caveat Phrase Usage	2%	0%

Interpretation: Lightweight qualitative findings only. Manual blinded scoring and factuality audit still pending before claiming quantified improvements.

⚠️ Important Limitations

What This Model Is

✅ A domain-adapted assistant for study-abroad queries
✅ Suitable for research and experimentation
✅ Parameter-efficient alternative to full fine-tuning
✅ Demonstrated on low-resource hardware (T4 GPU)

What This Model Is NOT

❌ Not a policy authority — Do not use as single source of truth
❌ Not factually verified — Outputs may contain inaccuracies or hallucinations
❌ Not production-ready — Requires validation layer before operational use
❌ Not comprehensive — May miss edge cases or regional variations
❌ Not a replacement for official guidance — Always verify with official sources

Recommended Usage

✅ Safe to use for:

• Educational chatbot prototyping
• Research on domain adaptation
• Fine-tuning experiments
• Data augmentation for related tasks

❌ NOT safe to use for:

• Direct immigration advice
• Official policy interpretation
• Time-sensitive information (visas, deadlines)
• High-stakes decision making without expert review

🔄 Model Variants

Merged vs Adapter-Only

Variant	Format	Size	Use Case
/merged	Full merged weights	~14GB	Inference, GGUF conversion
Adapter-only	LoRA weights	~30MB	Further training, fine-tuning

Both are available in this repository.

📚 Citation

If you use this model or findings:

Model Card

@model{StudyAbroadGPT-7B-LoRa-Kaggle,
  title={StudyAbroadGPT-7B-LoRa-Kaggle: LoRA-Adapted Mistral-7B for Study Abroad Guidance},
  author={Hosen, Md Millat},
  year={2025},
  howpublished={\url{https://huggingface.co/millat/StudyAbroadGPT-7B-LoRa-Kaggle}},
  note={HuggingFace Model Hub}
}

Research Paper

@article{hosen2025lora,
  title={A LoRA-Based Approach to Fine-Tuning LLMs for Educational Guidance in Resource-Constrained Settings},
  author={Hosen, Md Millat},
  journal={arXiv preprint arXiv:2504.15610},
  year={2025},
  doi={10.48550/arXiv.2504.15610}
}

🔐 License

Apache 2.0

📊 Performance Benchmarks

Comparison with Base Model

On a 50-sample deterministic evaluation:

• Response length: +2.3% increase
• Generation time: +14.6% slower (acceptable trade-off for domain specialization)
• Topic relevance: Maintained domain terminology coverage

Hardware Compatibility

Device	Status	Notes
NVIDIA T4 (16GB)	✅ Tested	Kaggle
NVIDIA P100 (16GB)	✅ Tested	Kaggle
NVIDIA A100 (40GB)	Should work	Not tested
CPU Only	❌ Not recommended	Too slow
Mac M1/M2	⚠️ Requires setup	MPS acceleration possible

🤝 Contributing & Feedback

• Bug Reports: Open an issue on training repo
• Model Improvements: Submit PRs with new training runs or data
• Questions: Check companion evaluation artifacts or GitHub discussions
• Evaluation: See LoRA Paper workspace

📧 Support

• 📖 Documentation: See WANDB.md for monitoring details
• 🔧 Setup Help: See training repo README
• 📋 Architecture: See architecture.md
• 📰 Full Methods: See arXiv:2504.15610

---

Model Version: 1.0
Release Date: May 2025
Last Updated: May 2025
Training Framework: Unsloth + Transformers
Base Model Lineage: Mistral-7B → Instruct-v0.3 → 4-bit Quantized → LoRA Fine-tuned