Mozat commited on
Commit
e1204b7
·
verified ·
1 Parent(s): 1041cc2

add README

Browse files
Files changed (1) hide show
  1. README.md +102 -0
README.md ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ library_name: transformers
4
+ base_model: distilbert-base-uncased
5
+ tags:
6
+ - text-classification
7
+ - affect
8
+ - emotion
9
+ - distilbert
10
+ language:
11
+ - en
12
+ ---
13
+
14
+ # how-affect-v1 — Bridge-Grounded Affect Detector
15
+
16
+ A DistilBERT-based affect-valence classifier fine-tuned on **non-circular**
17
+ author-narrated affect labels (mined from public-domain novel narration via
18
+ BookNLP), rather than on LLM-generated personality scores.
19
+
20
+ ## Why this exists
21
+
22
+ Production personality / emotion classifiers in companion AI are commonly trained
23
+ **on LLM labels** (e.g. Claude/GPT scores). Evaluation against those same LLM
24
+ labels is circular — the model only learns to imitate the labeling LLM. We
25
+ needed a HOW (affect) detector grounded in **independent human-written signal**
26
+ about how characters speak. Solution: harvest dialogue-tag adverbs + WordNet
27
+ emotion supersenses from BookNLP-processed novels (~1000 books, 25k labeled
28
+ quotes), bind them to the speaker's actual utterances, and train a probe.
29
+
30
+ ## Metrics
31
+
32
+ Held-out test set (5,971 quotes, balanced neg/pos author affect):
33
+
34
+ | Model | Held-out AUC |
35
+ |---|---|
36
+ | Existing circular "emotion" dim (177-dim model trained on Claude scores) | 0.557 |
37
+ | Frozen-embedding probe (sentence-transformer + linear head) | 0.637 |
38
+ | **This model — DistilBERT end-to-end on bridge labels** | **0.678** |
39
+
40
+ Honest ceiling: ~0.68 is real but modest. Narrated affect ("said bitterly")
41
+ often lives in prosody, not lexical content, so text-only affect detection
42
+ has a structural ceiling. A voice/prosody channel is the path to higher AUC.
43
+
44
+ ## Files
45
+
46
+ - `model.pt` — full state-dict: DistilBERT encoder + mean-pool + Linear(hidden→1) head.
47
+ - `metrics.json` — final held-out AUC + baseline comparison.
48
+
49
+ ## Usage
50
+
51
+ The head is custom (DistilBERT + mean-pool + 1-logit), so you can't use
52
+ `AutoModelForSequenceClassification.from_pretrained` directly. Load like this:
53
+
54
+ ```python
55
+ import torch
56
+ from transformers import AutoTokenizer, AutoModel
57
+
58
+ class AffectNet(torch.nn.Module):
59
+ def __init__(self):
60
+ super().__init__()
61
+ self.enc = AutoModel.from_pretrained("distilbert-base-uncased")
62
+ self.head = torch.nn.Linear(self.enc.config.hidden_size, 1)
63
+ def forward(self, ids, mask):
64
+ h = self.enc(input_ids=ids, attention_mask=mask).last_hidden_state
65
+ m = mask.unsqueeze(-1).float()
66
+ pooled = (h * m).sum(1) / m.sum(1).clamp(min=1e-6)
67
+ return self.head(pooled).squeeze(1)
68
+
69
+ tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")
70
+ model = AffectNet()
71
+ model.load_state_dict(torch.load("model.pt", map_location="cpu"))
72
+ model.eval()
73
+
74
+ text = "I can't bear this any longer."
75
+ enc = tok(text, padding="max_length", truncation=True, max_length=48, return_tensors="pt")
76
+ with torch.no_grad():
77
+ valence = torch.sigmoid(model(enc["input_ids"], enc["attention_mask"]))[0].item()
78
+ print(valence) # ~1.0 = negative/distressed affect, ~0.0 = positive
79
+ ```
80
+
81
+ ## Training data (non-circular)
82
+
83
+ Bridge corpus from ~1000 BookNLP-processed novels (`corpus/booknlp_output/`):
84
+ for each character quote, the narration window (±7 tokens around the quote) was
85
+ scanned for emotion supersense spans (`verb.emotion`, `noun.feeling`) and
86
+ manner adverbs anchored to a speech verb ("said *bitterly*"). Quotes mapped
87
+ to net-negative vs net-positive author affect → 17,749 neg / 16,375 pos
88
+ balanced labels (29,852 total used, 23,881 train / 5,971 test).
89
+
90
+ ## Architecture
91
+
92
+ - Base encoder: `distilbert-base-uncased` (~66M params).
93
+ - Head: `Dropout-free Linear(hidden_size, 1)` over mean-pooled token embeddings.
94
+ - Loss: `BCEWithLogitsLoss` on binary affect-valence.
95
+ - Trained 1-2 epochs on CPU (best epoch saved by held-out AUC; early-stopped when AUC stopped improving).
96
+ - Max input length: 48 tokens (quotes are short).
97
+
98
+ ## License
99
+
100
+ Trained on derivatives of public-domain (Project Gutenberg) novels processed
101
+ via BookNLP. The model weights are released for research use; please consult
102
+ your jurisdiction's rules around derivative works for production deployment.