metadata
license: mit
language:
- en
base_model:
- mistralai/Mistral-7B-v0.1
pipeline_tag: text-generation
tags:
- academic-writing
- mistral
- qlora
- fine-tuning
- arxiv
- llm
- uc-davis
Mistral-7B Fine-Tuned for Academic Style (QLoRA)
This is a parameter-efficient fine-tuning of mistralai/Mistral-7B-v0.1 using QLoRA on 500K academic abstracts. It was built for the ECS 271 final project at UC Davis.
Intended Use
The model is designed to generate formal academic paragraphs given a paper title, useful for research drafts, educational AI tools, and academic-style assistants.
Training Details
- Base model:
mistralai/Mistral-7B-v0.1 - Method: QLoRA (low-rank adapter)
- Prompt format: "Write an academic paragraph given the title: ..."
- Dataset: 500K arXiv abstracts
- Epochs: 1
- GPU: RTX 5070 Ti (~60 hours)
Limitations
- Generic or templated outputs
- No citation support
- Frequent hallucinations
Example
Prompt:
"Write an academic paragraph given the title: LoRA for In-Context Learning"
Output:
"We present LoRA (Loosely Regularized Adapters), a novel approach to fine-tune large language models for in-context learning tasks. Unlike traditional methods, LoRA updates only a small number of trainable parameters, achieving comparable performance while reducing training costs."
More Info
Main Project: Github