README.md

---
language: en
license: cc-by-4.0
library_name: transformers
pipeline_tag: token-classification
tags:
  - clause-segmentation
  - discourse
  - situation-entities
  - roberta
base_model: roberta-base
---

# DiSCo Clause Segmenter

A `roberta-base` token-classification model that splits English text into Elementary Discourse Unites, roughly **clauses**. Each token is tagged with a 3-way label and clause spans are recovered by matching the pattern `B I* E`:

| id | label | meaning |
|----|-------|---------|
| 0 | B | clause beginning |
| 1 | I | inside clause |
| 2 | E | clause end |

This is the segmentation component of the DiSCo pipeline; feed its clause output to the companion [`BabakScrapes/disco-se-classifier`](https://huggingface.co/BabakScrapes/disco-se-classifier) for Situation Entity typing.

## Usage

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tok = AutoTokenizer.from_pretrained("roberta-base", add_prefix_space=True)
model = AutoModelForTokenClassification.from_pretrained("BabakScrapes/disco-clause-segmenter").eval()

words = "There was bad weather at the airport and so our flight got delayed".split()
enc = tok(words, is_split_into_words=True, return_tensors="pt", truncation=True, max_length=512)
with torch.inference_mode():
    token_preds = model(**enc).logits.argmax(-1)[0].tolist()
# align token_preds back to words (majority vote per word), then cut clauses on B...E
```

See `pipeline.py` in the DiSCo release for the full word-alignment + `B I* E` clause-recovery logic.

## Training data

Trained on the **public SitEnt corpus** (Friedrich et al. 2016/2017), converted to `B/I/E` token labels, with a deterministic split (seed 42, 90/10 train/validation). The construction recipe (`construct_clause_corpus.ipynb`) is distributed with the DiSCo code/corpus release, not in this model repo. 

## Performance

In-domain SitEnt held-out evaluation (seed-42 10% partition; 16,515 tokens, 1,848 gold clause spans):

| Metric | Value |
|--------|-------|
| Token macro-F1 | .753 (B .703 / I .917 / E .639) |
| Token accuracy (= micro-F1) | .861 |
| Gold clauses with ≥50% predicted-overlap coverage | 95.5% |
| Predicted vs gold clause counts | 2,149 vs 1,848 (slight over-segmentation) |

The segmenter recovers clause *regions* reliably; the span-overlap numbers are the operationally relevant summaries because the downstream pipeline consumes whole clauses rather than exact boundary tokens. Macro-F1 is depressed by the sparse B/E classes.

## Limitations

- English only.
- Tends to slightly over-segment, which is benign for downstream SE classification.

## Citation

Hemmatian, B. (2022). *Taking the High Road: A Big Data Investigation of Natural Discourse in the Emerging U.S. Consensus about Marijuana Legalization*. Brown University. And the DiSCo corpus paper (forthcoming, *Behavior Research Methods*). SitEnt: Friedrich, Palmer & Pinkal (2016).