README.md

---
license: mit
language: en
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert-base-uncased
datasets:
- gramajo/nouns-proposals
tags:
- governance
- dao
- nouns
- null-result
---

# Nouns DAO Proposal Predictor (DistilBERT)

Fine-tuned DistilBERT that reads a Nouns DAO proposal's **title + description** and predicts
whether it passed. Trained on [gramajo/nouns-proposals](https://huggingface.co/datasets/gramajo/nouns-proposals)
(982 proposals, pass/fail labels).

## ⚠️ Read this before using the model

**Do not use this model to predict real proposal outcomes.** It is published as a
*negative result* and as a reproducible artifact, not as a working predictor. See the
numbers below and decide for yourself.

## Results

Accuracy is meaningless without a baseline. The bar is **majority-class accuracy** —
what you'd score by ignoring the proposal entirely and always guessing the more common
outcome. A model only tells you something if it clears that bar.

| run | n train | n test | test pass rate | majority baseline | accuracy | lift | AUC | beats baseline? |
|---|---|---|---|---|---|---|---|---|
| A_stratified_random | 785 | 197 | 50.2% | 50.2% | 58.9% | +8.6% | 0.648 | yes |
| B_temporal_breakeven | 785 | 197 | 28.4% | 71.6% | 73.1% | +1.5% | 0.651 | yes |
| C_post_breakeven_only | 147 | 50 | 28.0% | 72.0% | 72.0% | +0.0% | 0.365 | NO |

**What each run asks:**

- **A — stratified random split** over all 982 proposals. Regime held constant.
  *Does the text carry signal at all?*
- **B — temporal holdout** at the BreakEven break. Train on props 1–785, test on 786–982.
  *Does a model trained before the "BreakEven" voting bloc transfer to after it?*
- **C — post-break only**, stratified inside the new regime.
  *Within the new regime, is text predictive?* (n is small — read with caution.)

## The BreakEven regime change

Around proposal ~786, a bloc of large Nouns holders ("BreakEven") began voting down
spend proposals to bring outflows in line with revenue. The pass rate shifts sharply
across that boundary:

- Proposals 1–785: **55.9%** pass
- Proposals 786–982: **28.4%** pass

This is a structural break in the decision rule, and the new rule is largely **not a
function of the proposal text** — it's a function of who is voting and what the treasury
looks like. Run B is therefore a test of transfer across concept drift, not a test of
whether text is informative. Run A is the cleaner test of the latter.

## Training details

- Base model: `distilbert-base-uncased`, max_length 512, batch size 16, lr 3e-5, up to 6–8 epochs
- Early stopping on **eval_loss** (not F1 — F1 rewards a degenerate always-PASS predictor)
- **No class weighting.** An earlier version used balanced class weights derived from the
  pre-break training distribution; on the post-break test set this pushed the model further
  toward predicting PASS, making the drift look worse than it was.
- Seed: 42. Exact split indices are published in `splits.json` in this repo.

## Reproduce it

Everything needed to check these numbers is public:

- Dataset: https://huggingface.co/datasets/gramajo/nouns-proposals
- Exact split IDs: `splits.json` in this repo
- Raw metrics: `results.json` in this repo
- Training notebook: `nouns_predictor_experiments.ipynb` in this repo

## Limitations

- Text-only. No proposer identity, no proposer track record, no requested amount, no
  treasury balance at submission, no voting history — i.e. none of the variables most
  likely to actually determine a Nouns vote.
- 982 examples is small for a transformer. Run C in particular is trained on ~150 examples.
- Labels are binary pass/fail; abstentions, cancellations, and vetoes are not modeled here.
- Single seed. Results have not been averaged over multiple seeds and should be treated as
  indicative, not precise.