PhaaNe/gpt_depression

GPT-OSS 20B Depression Detection

Model Description

This model is a fine-tuned version of openai/gpt-oss-20b for depression detection in text. It uses QLoRA (Quantized LoRA) fine-tuning to efficiently adapt the large language model for binary classification of depression indicators in text.

Model Details

• Base Model: openai/gpt-oss-20b (20 billion parameters)
• Fine-tuning Method: QLoRA (Quantized Low-Rank Adaptation)
• Task: Binary text classification (depression vs non-depression)
• Training Data: 6,006 instruction-formatted samples
• Validation Data: 1,000 samples
• Test Data: 3,245 samples

Training Configuration

• LoRA Rank (r): 32
• LoRA Alpha: 64
• LoRA Dropout: 0.1
• Target Modules: q_proj, k_proj, v_proj, o_proj
• Training Epochs: 2.0
• Effective Batch Size: 8
• Learning Rate: 5e-4
• Optimizer: Paged AdamW 8-bit
• Scheduler: Cosine with warmup
• Quantization: 4-bit NF4 with double quantization

Performance

• Training Time: 3 hours 6 minutes (752 steps)
• Final Training Loss: 2.3626
• Best Validation Loss: 2.3047
• Token Accuracy: ~53.6%

Usage

Installation

pip install transformers torch peft accelerate bitsandbytes

Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Load model and tokenizer
model_id = "openai/gpt-oss-20b"
adapter_id = "PhaaNe/gpt_depression"

# Configure quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load base model
tokenizer = AutoTokenizer.from_pretrained(model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)

# Prepare input
text = "I feel hopeless and nothing seems to matter anymore. I cant find joy in anything."
instruction = "Analyze this text for depression indicators. Respond depression or non-depression:"
prompt = f"{instruction}

{text}

"

# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=10,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id,
    )

# Decode response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
prediction = response[len(prompt):].strip()
print(f"Prediction: {prediction}")

Dataset

The model was trained on a depression detection dataset with the following characteristics:

• Total Samples: 10,251
• Training: 6,006 samples (58.6%)
• Validation: 1,000 samples (9.8%)
• Test: 3,245 samples (31.7%)
• Class Distribution:
  • Depression: 62.5%
  • Non-depression: 37.5%

Intended Use

This model is designed for research and educational purposes in mental health text analysis. It can be used to:

• Identify potential depression indicators in text
• Support mental health research
• Assist in preliminary screening (not for clinical diagnosis)
• Analyze social media or forum posts for mental health insights

Limitations

• Not for Clinical Diagnosis: This model should not be used as a substitute for professional mental health assessment
• Bias: May reflect biases present in the training data
• Context: Performance may vary across different text domains and populations
• Language: Primarily trained on English text

Ethical Considerations

• Use responsibly and with appropriate human oversight
• Consider privacy implications when analyzing personal text
• Do not use for discriminatory purposes
• Complement, dont replace, professional mental health services

Citation

If you use this model in your research, please cite:

@misc{gpt_depression_2024,
  title={GPT-OSS 20B Depression Detection},
  author={PhaaNe},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/PhaaNe/gpt_depression}
}

License

This model is released under the Apache 2.0 License.