---
license: mit
language:
  - en
pipeline_tag: text-generation
tags:
  - mental-health
  - counseling
  - lora
  - peft
  - diffusion-language-model
  - LLaDA
---

# BiGraph-Diffuse

LoRA adapter for **BiGraph-Diffuse**, a retrieval-augmented diffusion language model for empathetic mental health counseling.

## Model Overview

This is a [LoRA](https://arxiv.org/abs/2106.09685) adapter fine-tuned on **LLaDA-8B-Instruct**, a discrete diffusion language model. The adapter is trained on counseling dialogues to generate empathetic, psychologically grounded counselor responses.

### Architecture

- **Base Model**: LLaDA-8B-Instruct (discrete diffusion LM)
- **Adapter**: LoRA (rank=32, alpha=64, dropout=0.1)
- **Target Modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`
- **Task**: Causal language modeling with masked diffusion loss

Full architecture includes **BiGraph-RAG**, a bipartite graph retrieval system that augments generation with relevant psychological knowledge. Code available at the [GitHub repo](https://github.com/Chekhov0919/BiGraph-Diffuse).

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
base_model_path = "path/to/LLaDA-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_path, trust_remote_code=True)
tokenizer.padding_side = "left"

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "Chekhov0919/BiGraph-Diffuse")
model.eval()

# Generate with diffusion
# See GitHub repo for full inference code with BiGraph-RAG integration
```

For the complete inference pipeline with BiGraph-RAG retrieval, refer to the [GitHub repository](https://github.com/Chekhov0919/BiGraph-Diffuse).

## Training

| Setting | Value |
|---------|-------|
| Base Model | LLaDA-8B-Instruct |
| Dataset | CPsyCounD (counseling dialogues) |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.1 |
| Batch size | 2 × 32 (gradient accumulation) |
| Learning rate | 3e-5 |
| Epochs | 5 |
| LR scheduler | Cosine |
| Mask token ID | 126336 |

## Citation

Please stay tuned — citation information will be added upon publication.

## License

MIT