README.md

---
license: apache-2.0
datasets:
- ShivomH/Medical-and-Mental-Health
language:
- en
base_model:
- meta-llama/Llama-3.2-3B-Instruct
new_version: ShivomH/Elixir-MentalHealth-3B
pipeline_tag: text-generation
library_name: transformers
tags:
- experimental
- failed_attempt
- mental_health
- medical
---

⚠️ **Important Disclaimer**

Elixir-v1 is an **experimental fine-tuned model** and is **not suitable for medical, clinical, or therapeutic use**.
The model may generate **hallucinations, fabricated details (names, emails, contacts), and unreliable medical/mental health information**.

This release is intended **only for research and educational purposes**.
Do **not** use this model for real-world health advice or decision-making.

## Model Summary

Elixir-v1 is an experimental fine-tuned version of Llama-3.2-3B-Instruct, trained on a mixed dataset of mental health and general medical dialogues.
The goal of this fine-tune was to create a supportive, domain-focused assistant capable of handling both medical knowledge and empathetic mental health conversations.

While the model produces domain-relevant responses, it should be considered experimental and not production-ready due to noticeable limitations and inconsistencies (see below).

## Training Dataset

~75,000 rows collected from 4 different Hugging Face datasets (mental health + medical QA/conversations).

### Data mixture:

- 42k mental health dialogues (single-turn and multi-turn)
- 33k medical Q&A

### Dataset Sources:

* [FunPang/medical_dataset](https://huggingface.co/datasets/FunPang/medical_dataset)
* [jerryjalapeno/nart-100k-synthetic](https://huggingface.co/datasets/jerryjalapeno/nart-100k-synthetic)
* [fadodr/mental_health_therapy](https://huggingface.co/datasets/fadodr/mental_health_therapy)
* [marmikpandya/mental-health](https://huggingface.co/datasets/marmikpandya/mental-health)

Conversations were reformatted into the Llama chat template (<|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, <|eot_id|>), rather than using the native messages list format.

## Intended Use

* Research exploration in fine-tuning small-scale instruction models.
* Educational purpose to understand how dataset quality and formatting affect fine-tuning outcomes.

## Not intended for:

* Clinical, diagnostic, or therapeutic use.
* Providing real medical or psychological advice.
* Deployment in production environments.

## Known Limitations & Issues

* Hallucinations: Sometimes generates unrealistic or fabricated details.
* Unreliable factual grounding: May provide outdated or incorrect medical information.
* Data leakage style outputs: Tends to generate random names, email addresses, or phone numbers due unfiltered dataset during training.
* Verbose responses: Occasionally responds with unnecessarily long answers.
* Data quality concerns: Dataset included inconsistencies and lacked careful filtering.

## Ethical Considerations

* Elixir-v1 was never intended for professional or clinical use.
* Outputs should not be trusted for diagnosis, treatment, or emergency situations.
* Users are encouraged to seek licensed medical and mental health professionals for real-world concerns.

## Lessons Learned

* Dataset quality and formatting matter more than quantity.
* Mixing multiple sources without thorough cleaning introduces artifacts.
* Following proper schema (messages list format) is crucial for stability.
* Transparency in documenting shortcomings can prevent misuse.

## Future Work

* Curating a cleaner dataset (removing names, contacts, artifacts).
* Using the messages format instead of raw Llama template for alignment.
* Adding stricter safety/ethical filtering during training.
* Exploring reinforcement-based approaches to reduce hallucinations.

## Author

Developed by Shivom Hatalkar as a first LLM fine-tuning experiment on Llama-3.2-3B-Instruct.

## How to load the model

Install Dependencies
```Python
!pip install transformers accelerate bitsandbytes
```

Inference
```Python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_NAME = "ShivomH/Elixir-Health-Llama3B"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# 🔹 Simple inference helper
def chat_with_elixir(user_input, system_prompt="""You are a compassionate and knowledgeable wellness assistant.
You specialize in Mental health support, mindfulness techniques, and emotional wellness
Provide clear, concise, and actionable responses. Avoid overly long explanations, and offer direct advice or suggestions. When giving lists, display each item on a new line.
Be warm, professional, and gentle in your tone, and always encourage users to seek professional help if serious health concerns arise."""):
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_input},
    ]

    # Use Llama-3 chat template
    formatted = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    inputs = tokenizer(formatted, return_tensors="pt").to(model.device)

    # Set eos_token_id to <|eot_id|>
    eos = tokenizer.eos_token_id

    output = model.generate(
        **inputs,
        max_new_tokens=300,
        eos_token_id=eos,
        pad_token_id=tokenizer.eos_token_id,
        do_sample=True,
        temperature=0.5,
        top_p=0.8
    )

    # Decode only the new tokens
    response = tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    return response.strip()

# 🔹 Test Elixir
print("=== Test Conversation ===")
print(chat_with_elixir("Been feeling extremely anxious lately. Suggest me some safe coping strategies."))
```