README.md

---
language:
- en
license: apache-2.0
tags:
- text-classification
- multilabel-classification
- behavioral-coding
- motivational-interviewing
- modernbert
- transformers
base_model: answerdotai/ModernBERT-base
metrics:
- f1
- precision
- recall
- exact_match
- hamming_loss
model-index:
- name: bc-multilabel-classifier
  results:
  - task:
      type: text-classification
      name: Multilabel Text Classification
    metrics:
    - name: Exact Match
      type: exact_match
      value: 0.8563
    - name: Hamming Loss
      type: hamming_loss
      value: 0.0579
    - name: F1 Macro
      type: f1_macro
      value: 0.8666
    - name: F1 Micro
      type: f1_micro
      value: 0.9246
    - name: Adherent F1
      type: f1
      value: 0.7429
    - name: Non-Adherent F1
      type: f1
      value: 0.8932
    - name: Neutral F1
      type: f1
      value: 0.9639
widget:
- text: "That's a great step you're taking to improve your health."
- text: "You really should stop smoking, it's bad for you."
- text: "What do you think about trying to quit?"
---

# Behavioral Coding Multilabel Classifier

## Model Description

This model is a fine-tuned version of [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) for multilabel classification of Motivational Interviewing (MI) behavioral codes. It classifies utterances into three non-mutually-exclusive categories used in behavioral coding of therapeutic conversations.

**Developed by:** Lekhansh

**Model type:** Multilabel Text Classification

**Language:** English

**Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)

**License:** Apache 2.0

## Intended Uses

### Primary Use Case

This model is designed for automated behavioral coding in Motivational Interviewing contexts, predicting three types of MI-consistent and MI-inconsistent behaviors:

- **Adherent:** MI-adherent behaviors (e.g., affirmations, seek collaboration)
- **Non-Adherent:** MI-non-adherent behaviors (e.g., confrontation, persuade without permission)
- **Neutral:** Neutral behaviors (e.g., giving information, questions, reflections)

### Key Features

- **Multilabel Classification:** Utterances can have multiple labels simultaneously
- **Therapeutic Context:** Specifically trained on Motivational Interviewing conversations
- **Context-Aware:** Includes three preceding utterances for context

### Potential Applications

- Automated analysis of therapy session transcripts
- Training and feedback for MI practitioners
- Quality assurance in behavioral health interventions
- Research in therapeutic communication patterns

## Model Performance

### Test Set Metrics

The model was evaluated on a held-out test set of 3,235 coded utterances.

#### Overall Performance

| Metric | Score |
|--------|------:|
| **Exact Match Accuracy** | **85.63%** |
| **Hamming Loss** | **0.0579** |
| **F1 Macro** | **86.66%** |
| **F1 Micro** | **92.46%** |
| **Precision Macro** | 86.53% |
| **Precision Micro** | 93.47% |
| **Recall Macro** | 86.84% |
| **Recall Micro** | 91.48% |

**Exact Match:** Percentage of examples where all labels are predicted correctly
**Hamming Loss:** Average fraction of labels that are incorrectly predicted (lower is better)

#### Per-Label Performance

| Label | F1 Score | Precision | Recall | Accuracy |
|-------|----------|-----------|--------|----------|
| **Adherent** | 74.29% | 74.47% | 74.10% | 90.26% |
| **Non-Adherent** | 89.32% | 87.34% | 91.39% | 98.98% |
| **Neutral** | 96.39% | 97.77% | 95.04% | 93.38% |

### Class Distribution

The training data exhibits class imbalance, addressed through positive class weighting:
- **Neutral:** Most common (majority class)
- **Non-Adherent:** Moderate frequency
- **Adherent:** Least common (minority class)

## Training Details

### Training Data

- **Source:** Multilabel behavioral coding dataset from Motivational Interviewing transcripts
- **Preprocessing:**
  - Excluded utterances marked as "not_coded" (no MI codes assigned)
  - Included context from three preceding utterances
  - Stratified splitting to maintain label distribution
- **Split:** 70% train, 15% validation, 15% test

### Training Procedure

**Hardware:**
- GPU training with CUDA
- Mixed precision (BFloat16) training

**Hyperparameters:**

| Parameter | Value |
|-----------|-------|
| Learning Rate | 6e-5 |
| Batch Size (per device) | 12 |
| Gradient Accumulation | 2 steps |
| Effective Batch Size | 24 |
| Max Sequence Length | 3000 tokens |
| Epochs | 20 (early stopped at epoch 14) |
| Weight Decay | 0.01 |
| Warmup Ratio | 0.1 |
| LR Scheduler | Cosine |
| Optimizer | AdamW |
| Dropout | 0.1 |

**Training Features:**
- **Positive Class Weighting:** BCEWithLogitsLoss with computed pos_weights for each label
- **Early Stopping:** Patience of 3 epochs on validation F1 macro
- **Gradient Checkpointing:** Enabled for memory efficiency
- **Flash Attention 2:** For efficient attention computation
- **Best Model Selection:** Based on validation F1 macro score

**Loss Function:** Binary Cross-Entropy with Logits Loss (BCEWithLogitsLoss) with per-label positive class weights

### Model Architecture

The model uses a custom architecture on top of ModernBERT:
```
ModernBERT-base (encoder)
  → [CLS] token extraction
  → Dropout (0.1)
  → Linear layer (hidden_size → 3)
  → Sigmoid activation (applied during inference)
```

## Usage

### Direct Use

```python
import torch
from transformers import AutoTokenizer, AutoModel
import torch.nn as nn

# Define the model class
class MultiLabelBERTModel(nn.Module):
    def __init__(self, model_name, num_labels=3, dropout=0.1):
        super().__init__()
        self.bert = AutoModel.from_pretrained(model_name)
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(self.bert.config.hidden_size, num_labels)
        self.num_labels = num_labels

    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        pooled_output = outputs.last_hidden_state[:, 0, :]  # [CLS] token
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)
        return logits

# Load model and tokenizer
model_name = "Lekhansh/bc-multilabel-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize model architecture
model = MultiLabelBERTModel(model_name, num_labels=3)

# Load trained weights
# Note: You'll need to load the weights from the saved model
model.eval()

# Prepare input
text = "That's a wonderful goal you've set for yourself."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=3000)

# Get predictions
with torch.no_grad():
    logits = model(inputs['input_ids'], inputs['attention_mask'])
    probs = torch.sigmoid(logits)
    predictions = (probs > 0.5).int()

# Interpret results
labels = ['adherent', 'non_adherent', 'neutral']
print(f"Text: {text}")
print("\nPredictions:")
for i, label in enumerate(labels):
    if predictions[0][i]:
        print(f"  ✓ {label} (confidence: {probs[0][i]:.2%})")
```

### Batch Prediction with Confidence Scores

```python
def predict_multilabel(texts, model, tokenizer, threshold=0.5):
    """
    Predict multiple labels for each text with confidence scores.

    Args:
        texts: List of input texts
        model: The multilabel classification model
        tokenizer: The tokenizer
        threshold: Probability threshold for positive prediction (default: 0.5)

    Returns:
        List of dicts with predictions and probabilities
    """
    inputs = tokenizer(
        texts,
        return_tensors="pt",
        truncation=True,
        max_length=3000,
        padding=True
    )

    with torch.no_grad():
        logits = model(inputs['input_ids'], inputs['attention_mask'])
        probs = torch.sigmoid(logits)

    labels = ['adherent', 'non_adherent', 'neutral']
    results = []

    for i in range(len(texts)):
        predictions = (probs[i] > threshold).int()
        result = {
            'text': texts[i],
            'labels': {},
            'probabilities': {}
        }

        for j, label in enumerate(labels):
            result['labels'][label] = bool(predictions[j])
            result['probabilities'][label] = float(probs[i][j])

        results.append(result)

    return results

# Example usage
utterances = [
    "I hear you saying that you want to change but you're not sure how.",
    "You need to stop making excuses and just do it.",
    "How many cigarettes do you smoke per day?"
]

results = predict_multilabel(utterances, model, tokenizer)
for r in results:
    print(f"\nText: {r['text'][:60]}...")
    print("Predicted labels:")
    for label in ['adherent', 'non_adherent', 'neutral']:
        status = "✓" if r['labels'][label] else "✗"
        conf = r['probabilities'][label]
        print(f"  {status} {label}: {conf:.2%}")
```

### Custom Threshold Tuning

```python
# Adjust threshold for precision/recall trade-off
def predict_with_custom_threshold(text, model, tokenizer, thresholds):
    """
    Predict with different thresholds for each label.

    Args:
        thresholds: Dict with keys 'adherent', 'non_adherent', 'neutral'
    """
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=3000)

    with torch.no_grad():
        logits = model(inputs['input_ids'], inputs['attention_mask'])
        probs = torch.sigmoid(logits)

    labels_list = ['adherent', 'non_adherent', 'neutral']
    predictions = {}

    for i, label in enumerate(labels_list):
        threshold = thresholds.get(label, 0.5)
        predictions[label] = {
            'predicted': bool(probs[0][i] > threshold),
            'probability': float(probs[0][i]),
            'threshold': threshold
        }

    return predictions

# Example: Higher threshold for adherent (higher precision)
custom_thresholds = {
    'adherent': 0.6,
    'non_adherent': 0.5,
    'neutral': 0.5
}

result = predict_with_custom_threshold(
    "What are your thoughts on reducing your drinking?",
    model,
    tokenizer,
    custom_thresholds
)
```


## Limitations and Bias

### Limitations

1. **Domain Specificity:** Trained on Motivational Interviewing data; may not generalize to other therapeutic modalities
2. **Context Dependency:** Performance may vary with utterances lacking proper conversational context
3. **Class Imbalance:** Lower performance on "adherent" label due to class imbalance in training data
4. **Multilabel Complexity:** Some utterances may have ambiguous or overlapping codes
5. **Context Length:** Maximum 3000 tokens; longer texts will be truncated
6. **Language:** Trained on English text only

### Potential Biases

- Training data may reflect biases from the original coding framework and human coders
- Performance may vary across different MI contexts (e.g., substance use vs. health behavior change)
- Cultural and linguistic variations in therapeutic communication may affect predictions
- The model may be more accurate on populations/contexts similar to training data

### Recommended Use

- Use as a screening tool or preliminary analysis, not as definitive behavioral coding
- Validate predictions with human expert review, especially for critical applications
- Consider adjusting prediction thresholds based on your use case (precision vs. recall trade-off)
- Be aware that multilabel predictions may sometimes conflict with clinical judgment

## Technical Specifications

### Model Architecture

- **Base:** ModernBERT-base (encoder-only transformer)
- **Custom Head:** Dropout (0.1) + Linear layer (hidden_size → 3 labels)
- **Activation:** Sigmoid (for independent label probabilities)
- **Attention:** Flash Attention 2 implementation
- **Parameters:** ~110M (inherited from base model + classification head)
- **Precision:** BFloat16

### Compute Infrastructure

- **Training:** Single GPU with CUDA
- **Inference:** CPU or GPU compatible
- **Memory:** ~500MB model size

### Label Format

```python
# Output format
{
  "adherent": 0 or 1,
  "non_adherent": 0 or 1,
  "neutral": 0 or 1
}

# Example: An utterance can have multiple labels
# "I hear that you're struggling, and I believe you can overcome this."
# → adherent=1, non_adherent=0, neutral=0
```

## Environmental Impact

Training was conducted using mixed precision to optimize resource usage. Exact carbon footprint was not measured.

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{lekhansh2025bcmultilabel,
  author = {Lekhansh},
  title = {Behavioral Coding Multilabel Classifier for Motivational Interviewing},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Lekhansh/bc-multilabel-classifier}}
}
```

## References

For more information on Motivational Interviewing behavioral coding:
- Miller, W. R., & Rollnick, S. (2013). *Motivational Interviewing: Helping People Change* (3rd ed.)
- Moyers, T. B., et al. (2016). *Motivational Interviewing Treatment Integrity Coding Manual 4.2.1*

## Model Card Authors

Lekhansh

## Model Card Contact

[drlekhansh@gmail.com]