File size: 4,628 Bytes
b4e5d15 277136a b4e5d15 277136a b4e5d15 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 |
---
license: apache-2.0
base_model: openai/gpt-oss-20b
tags:
- depression-detection
- mental-health
- text-classification
- qlora
- peft
- psychology
- healthcare
language:
- en
pipeline_tag: text-classification
library_name: transformers
---
# GPT-OSS 20B Depression Detection
## Model Description
This model is a fine-tuned version of [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) for depression detection in text. It uses QLoRA (Quantized LoRA) fine-tuning to efficiently adapt the large language model for binary classification of depression indicators in text.
## Model Details
- **Base Model**: openai/gpt-oss-20b (20 billion parameters)
- **Fine-tuning Method**: QLoRA (Quantized Low-Rank Adaptation)
- **Task**: Binary text classification (depression vs non-depression)
- **Training Data**: 6,006 instruction-formatted samples
- **Validation Data**: 1,000 samples
- **Test Data**: 3,245 samples
## Training Configuration
- **LoRA Rank (r)**: 32
- **LoRA Alpha**: 64
- **LoRA Dropout**: 0.1
- **Target Modules**: q_proj, k_proj, v_proj, o_proj
- **Training Epochs**: 2.0
- **Effective Batch Size**: 8
- **Learning Rate**: 5e-4
- **Optimizer**: Paged AdamW 8-bit
- **Scheduler**: Cosine with warmup
- **Quantization**: 4-bit NF4 with double quantization
## Performance
- **Training Time**: 3 hours 6 minutes (752 steps)
- **Final Training Loss**: 2.3626
- **Best Validation Loss**: 2.3047
- **Token Accuracy**: ~53.6%
## Usage
### Installation
```bash
pip install transformers torch peft accelerate bitsandbytes
```
### Inference
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
# Load model and tokenizer
model_id = "openai/gpt-oss-20b"
adapter_id = "PhaaNe/gpt_depression"
# Configure quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
# Load base model
tokenizer = AutoTokenizer.from_pretrained(model_id)
base_model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
# Prepare input
text = "I feel hopeless and nothing seems to matter anymore. I cant find joy in anything."
instruction = "Analyze this text for depression indicators. Respond depression or non-depression:"
prompt = f"{instruction}
{text}
"
# Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1024)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=10,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
# Decode response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
prediction = response[len(prompt):].strip()
print(f"Prediction: {prediction}")
```
## Dataset
The model was trained on a depression detection dataset with the following characteristics:
- **Total Samples**: 10,251
- **Training**: 6,006 samples (58.6%)
- **Validation**: 1,000 samples (9.8%)
- **Test**: 3,245 samples (31.7%)
- **Class Distribution**:
- Depression: 62.5%
- Non-depression: 37.5%
## Intended Use
This model is designed for research and educational purposes in mental health text analysis. It can be used to:
- Identify potential depression indicators in text
- Support mental health research
- Assist in preliminary screening (not for clinical diagnosis)
- Analyze social media or forum posts for mental health insights
## Limitations
- **Not for Clinical Diagnosis**: This model should not be used as a substitute for professional mental health assessment
- **Bias**: May reflect biases present in the training data
- **Context**: Performance may vary across different text domains and populations
- **Language**: Primarily trained on English text
## Ethical Considerations
- Use responsibly and with appropriate human oversight
- Consider privacy implications when analyzing personal text
- Do not use for discriminatory purposes
- Complement, dont replace, professional mental health services
## Citation
If you use this model in your research, please cite:
```bibtex
@misc{gpt_depression_2024,
title={GPT-OSS 20B Depression Detection},
author={PhaaNe},
year={2024},
publisher={Hugging Face},
url={https://huggingface.co/PhaaNe/gpt_depression}
}
```
## License
This model is released under the Apache 2.0 License.
|