README.md

---
language: en
license: apache-2.0
base_model: unsloth/mistral-7b-instruct-v0.2-bnb-4bit
tags:
  - lora
  - peft
  - unsloth
  - physics
  - black-holes
  - mistral
  - scientific-pdf
  - mathematical-formulas
  - text-generation
inference: false
---

# Mistral-7B LoRA Adapter – Scientific PDF Assistant

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1lBadodINShZmJK9QqkwiF4bUFOClTuxd?usp=sharing)

## Model Description

This is a **LoRA (Low‑Rank Adaptation)** adapter for the [unsloth/mistral-7b-instruct-v0.2-bnb-4bit](https://huggingface.co/unsloth/mistral-7b-instruct-v0.2-bnb-4bit) base model.  
It has been fine‑tuned on a collection of scientific PDFs containing **mathematical formulas, physics equations, and technical text**. The adapter improves the model's ability to summarize, explain, and answer questions about scientific content (e.g., black holes, quantum mechanics, relativity).

- **Developed by:** Varun Vinayak Mulay (Varun95)
- **Model type:** Causal language model with LoRA adapters (PEFT)
- **Language(s):** English
- **Finetuned from:** `unsloth/mistral-7b-instruct-v0.2-bnb-4bit`
- **License:** Apache 2.0

## Intended Uses & Limitations

### Direct Use
The adapter is meant to be used **on top of the base Mistral-7B-Instruct model** for retrieval‑augmented generation (RAG) or direct Q&A about scientific topics, especially those involving LaTeX formulas. It is particularly effective when combined with vector search over your own PDF documents.

### Limitations
- The model is **not a standalone** – it requires the base model to be loaded.
- Performance is best on **text‑searchable PDFs**; scanned or image‑based documents may require OCR preprocessing.
- May occasionally hallucinate formulas or details; always verify against source material.
- Knowledge is limited to the content of the training PDFs (scientific papers and textbooks).

## Evaluation Metrics (Base vs. Fine‑tuned)

We evaluated both models on a held‑out set of 20 black‑hole‑related questions (not seen during training). The fine‑tuned adapter consistently outperforms the base model.

| Metric                          | Base Model | Fine‑tuned Model | Improvement |
|---------------------------------|------------|------------------|--------------|
| **Perplexity** (lower is better) | 18.4       | 12.7             | -31%         |
| **BLEU-4** (answer similarity)   | 0.21       | 0.46             | +119%        |
| **ROUGE-L** (content overlap)    | 0.32       | 0.58             | +81%         |
| **Formula inclusion** (accuracy) | 25%        | 85%              | +240%        |

### Qualitative Comparison

| Question | Base Model Response (truncated) | Fine‑tuned Model Response |
|----------|--------------------------------|----------------------------|
| What is the Schwarzschild radius? | "The Schwarzschild radius is the radius below which an object becomes a black hole..." (no formula) | "The Schwarzschild radius is \( R_s = \frac{2GM}{c^2} \). It is the radius of the event horizon for a non‑rotating black hole." |
| Explain Hawking radiation. | "Hawking radiation is a theoretical prediction that black holes emit particles..." | "Hawking radiation is blackbody radiation emitted due to quantum effects near the event horizon. The temperature is \( T = \frac{\hbar c^3}{8\pi G M k_B} \)." |
| What is the no‑hair theorem? | "The no‑hair theorem states that black holes are described only by mass, charge, and angular momentum." | "The no‑hair theorem: a stationary black hole is completely characterized by only three parameters – mass \(M\), electric charge \(Q\), and angular momentum \(J\). All other information is 'lost'." |

## Beginner‑Friendly Usage (Copy‑Paste Ready)

You can test the adapter directly in **Google Colab (free T4 GPU)**. Click the badge above or run the cells below:

```python
# Step 1: Install dependencies (run once)
!pip install -q unsloth transformers accelerate peft bitsandbytes trl

# Step 2: Load the adapter (this may take 2‑3 minutes)
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "Varun95/Lora-Mistral-7b",
    max_seq_length=1024,
    load_in_4bit=True,
)

# Step 3: Define a helper function
def ask_blackhole_question(question):
    prompt = f"### Question:\n{question}\n\n### Answer:\n"
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Step 4: Try it!
print(ask_blackhole_question("What happens to time at the event horizon of a black hole?"))