README.md

---
base_model: tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3
license:
- llama3.1
- gemma
language:
- ja
- en
pipeline_tag: text-generation
tags:
  - counseling
  - dialogue-system
datasets:
- UEC-InabaLab/KokoroChat
---

# 🧠 Llama-3.1-KokoroChat-High: Japanese Counseling Dialogue Model

**Llama-3.1-KokoroChat-High** is a large-scale Japanese language model fine-tuned on the **entire KokoroChat dataset**—a collection of over 6,000 psychological counseling dialogues conducted via **role-play between trained counselors**. The model is capable of generating **empathetic and context-aware responses** suitable for mental health-related conversational tasks.

---

## 💡 Overview

- ✅ Fine-tuned on **2,601 dialogues** with client feedback scores between **70 and 98**
- ✅ Data collected through **text-based role-play** by trained counselors 
- ✅ Covers a wide range of topics: depression, family, school, career, relationships, and more
- ✅ Base Model: [`tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3`](https://huggingface.co/tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.3)

---

## ⚙️ Usage Example

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "UEC-InabaLab/Llama-3.1-KokoroChat-High"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Set pad_token_id
if tokenizer.pad_token_id is None:
    tokenizer.pad_token = "[PAD]"
    tokenizer.pad_token_id = tokenizer.convert_tokens_to_ids("[PAD]")

model.config.pad_token_id = tokenizer.pad_token_id

# Build dialogue input
messages = [
    {"role": "system", "content": "心理カウンセリングの会話において、対話履歴を考慮し、カウンセラーとして適切に応答してください。"},
    {"role": "user", "content": "最近、気分が落ち込んでやる気が出ません。"}
]

# Tokenize with chat template
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

attention_mask = inputs.ne(tokenizer.pad_token_id)

# Generate response
outputs = model.generate(
    inputs,
    attention_mask=attention_mask,
    pad_token_id=tokenizer.pad_token_id,
    max_new_tokens=256
)

# Extract only the newly generated tokens
response = outputs[0][inputs.shape[-1]:]
response_text = tokenizer.decode(response, skip_special_tokens=True)

# Print clean response
print(response_text)
```

---

## 🛠️ Fine-Tuning Details

Fine-tuning was performed using **QLoRA** with the following configuration:

- **Quantization**: 4-bit NF4 with bfloat16 computation
- **LoRA target modules**: `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
- **LoRA parameters**:
  - `r = 8`
  - `lora_alpha = 16`
  - `lora_dropout = 0.05`

### Dataset Split

- **Training Data**: 2,601 dialogues with feedback scores between 70 and 98  
- **Train/Validation Split**: 90% train, 10% validation

### Hyperparameter Settings

- **Optimizer**: `adamw_8bit`
- **Warm-up Steps**: `100`
- **Learning Rate**: `1e-3`
- **Epochs**: `5`
- **Batch Size**: `8`
- **Validation Frequency**: every 400 steps

---

## 📄 Citation

If you use this model or dataset, please cite the following paper:

```bibtex
@inproceedings{qi2025kokorochat,
  title     = {KokoroChat: A Japanese Psychological Counseling Dialogue Dataset Collected via Role-Playing by Trained Counselors},
  author    = {Zhiyang Qi and Takumasa Kaneko and Keiko Takamizo and Mariko Ukiyo and Michimasa Inaba},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics},
  year      = {2025},
  url       = {https://github.com/UEC-InabaLab/KokoroChat}
}
```
---

## 🔗 Related

- 📁 **Dataset**:  
  - [KokoroChat on Hugging Face Datasets](https://huggingface.co/datasets/UEC-InabaLab/KokoroChat)  
  - [KokoroChat on GitHub (UEC-InabaLab)](https://github.com/UEC-InabaLab/KokoroChat)
- 🤖 **Model Variants**:  
  - [Llama-3.1-KokoroChat-Low](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-Low): fine-tuned on **3,870 dialogues** with client feedback scores **< 70**
  - [Llama-3.1-KokoroChat-Full](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-Full): fine-tuned on **6,471 dialogues** with client feedback scores **≤ 98**
- 🤖 **Score Prediction Model**:
  - [Llama-3.1-KokoroChat-ScorePrediction](https://huggingface.co/UEC-InabaLab/Llama-3.1-KokoroChat-ScorePrediction): fine-tuned to **predict client feedback scores** based on the full counseling dialogue history.
- 📄 **Paper**: [ACL 2025 Paper](https://aclanthology.org/2025.acl-long.608/)