File size: 2,331 Bytes
f767984 3283bf4 f767984 3283bf4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 | ---
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- mental-health
- counseling
- lora
- peft
- diffusion-language-model
- LLaDA
---
# BiGraph-Diffuse
LoRA adapter for **BiGraph-Diffuse**, a retrieval-augmented diffusion language model for empathetic mental health counseling.
## Model Overview
This is a [LoRA](https://arxiv.org/abs/2106.09685) adapter fine-tuned on **LLaDA-8B-Instruct**, a discrete diffusion language model. The adapter is trained on counseling dialogues to generate empathetic, psychologically grounded counselor responses.
### Architecture
- **Base Model**: LLaDA-8B-Instruct (discrete diffusion LM)
- **Adapter**: LoRA (rank=32, alpha=64, dropout=0.1)
- **Target Modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`
- **Task**: Causal language modeling with masked diffusion loss
Full architecture includes **BiGraph-RAG**, a bipartite graph retrieval system that augments generation with relevant psychological knowledge. Code available at the [GitHub repo](https://github.com/Chekhov0919/BiGraph-Diffuse).
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load base model
base_model_path = "path/to/LLaDA-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
tokenizer.padding_side = "left"
base_model = AutoModelForCausalLM.from_pretrained(
base_model_path,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Chekhov0919/BiGraph-Diffuse")
model.eval()
# Generate with diffusion
# See GitHub repo for full inference code with BiGraph-RAG integration
```
For the complete inference pipeline with BiGraph-RAG retrieval, refer to the [GitHub repository](https://github.com/Chekhov0919/BiGraph-Diffuse).
## Training
| Setting | Value |
|---------|-------|
| Base Model | LLaDA-8B-Instruct |
| Dataset | CPsyCounD (counseling dialogues) |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.1 |
| Batch size | 2 × 32 (gradient accumulation) |
| Learning rate | 3e-5 |
| Epochs | 5 |
| LR scheduler | Cosine |
| Mask token ID | 126336 |
## Citation
Please stay tuned — citation information will be added upon publication.
## License
MIT
|