help2opensource/Qwen3-4B-Instruct-2507_mental_health_cbt

🧠 Model Card: help2opensource/Qwen3-4B-Instruct-2507_mental_health_cbt

🔍 Overview

This model is a 4-bit quantized version of the Qwen/Qwen3-4B-Instruct large language model fine-tuned using LoRA (Low-Rank Adaptation) to generate CBT (Cognitive Behavioral Therapy) responses for mental health applications. The model follows the prompt format:
### Dialogue:\n{dialogue}\n### Use the CBT technique: {technique}\n### Plan: {plan}\n### Assistant:

It is designed to assist in generating structured, evidence-based therapeutic interventions for individuals working with cognitive-behavioral techniques.

---

🎯 Use Cases

• Mental Health Support: Provide users with CBT strategies (e.g., cognitive restructuring, behavioral activation).
• Therapeutic Planning: Generate actionable plans based on patient dialogues.
• Clinical Training: Simulate therapist responses for training purposes.

---

📚 Training Data

The model is trained on the LangAGI-Lab/cactus dataset, which includes:

• Dialogues: Real-world conversations between patients and therapists.
• CBT Techniques: Predefined techniques (e.g., "challenging negative thoughts").
• Plans: Step-by-step therapeutic plans to address specific issues.

The dataset is split into training and test sets (90/10). Each example includes:

{
    "dialogue": str,          # Patient-Therapist conversation
    "cbt_technique": str,     # CBT technique to apply
    "cbt_plan": str           # Step-by-step therapeutic plan
}

---

🧠 Model Architecture

• Base Model: Qwen/Qwen3-4B-Instruct (a 4-billion parameter causal language model).
• Quantization: 4-bit quantized with bitsandbytes for reduced memory usage.
• LoRA Configuration: Fine-tuned using LoRA with the following parameters:
  • Rank (r): 8
  • Alpha (lora_alpha): 32
  • Target Layers: q_proj, v_proj, k_proj, o_proj
  • Dropout Rate: 0.1

---

🛠️ Training Process

• Hardware: GPU with mixed-precision (FP16).
• Batch Size: per_device_train_batch_size=5, gradient_accumulation_steps=5.
• Optimization: AdamW optimizer, learning rate 2e-5, weight decay 0.01.
• Early Stopping: Not included in the code (can be added via EarlyStoppingCallback).
• Evaluation Metrics:
  • BLEU (for n-gram overlap)
  • ROUGE-L (for long-text similarity)

---

📌 Known Limitations

• The model may struggle with highly specialized clinical cases not covered in the training data.
• Generated CBT responses should be reviewed by licensed professionals before use.

---

🛡️ Safety & Ethics

• This model is intended for educational and research purposes only.
• Avoid using it for real-world therapeutic decisions without human oversight.
• Ensure compliance with local laws and ethical guidelines for mental health applications.

---