metadata
license: mit
language:
- en
pipeline_tag: text-generation
tags:
- mental-health
- counseling
- lora
- peft
- diffusion-language-model
- LLaDA
BiGraph-Diffuse
LoRA adapter for BiGraph-Diffuse, a retrieval-augmented diffusion language model for empathetic mental health counseling.
Model Overview
This is a LoRA adapter fine-tuned on LLaDA-8B-Instruct, a discrete diffusion language model. The adapter is trained on counseling dialogues to generate empathetic, psychologically grounded counselor responses.
Architecture
- Base Model: LLaDA-8B-Instruct (discrete diffusion LM)
- Adapter: LoRA (rank=32, alpha=64, dropout=0.1)
- Target Modules:
q_proj,k_proj,v_proj,o_proj - Task: Causal language modeling with masked diffusion loss
Full architecture includes BiGraph-RAG, a bipartite graph retrieval system that augments generation with relevant psychological knowledge. Code available at the GitHub repo.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Load base model
base_model_path = "path/to/LLaDA-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
tokenizer.padding_side = "left"
base_model = AutoModelForCausalLM.from_pretrained(
base_model_path,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True
)
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Chekhov0919/BiGraph-Diffuse")
model.eval()
# Generate with diffusion
# See GitHub repo for full inference code with BiGraph-RAG integration
For the complete inference pipeline with BiGraph-RAG retrieval, refer to the GitHub repository.
Training
| Setting | Value |
|---|---|
| Base Model | LLaDA-8B-Instruct |
| Dataset | CPsyCounD (counseling dialogues) |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.1 |
| Batch size | 2 × 32 (gradient accumulation) |
| Learning rate | 3e-5 |
| Epochs | 5 |
| LR scheduler | Cosine |
| Mask token ID | 126336 |
Citation
Please stay tuned — citation information will be added upon publication.
License
MIT