---
language: en
license: apache-2.0
library_name: lightgbm
tags:
  - twitter
  - engagement-prediction
  - tabular
  - ranking
---

# will-it-bang

> **Can you predict a tweet's engagement from only what's knowable before it's posted?**

This project says yes — and shows how. We train a LightGBM model on ~110k tweets from the [Twitter Community Archive](https://www.community-archive.org) to forecast the exposure-adjusted engagement rate of a tweet using:

- The tweet text itself
- Structural features (length, URLs, media, mentions, emoji, etc.)
- The author's pre-tweet history of *replies received*, *quotes received*, and *mentions received* (never dump-time totals)
- Reply/quote parent context from both CA and the X API
- Calendar / platform-era signals
- A semantic embedding of the text (MiniLM → PCA + residual engagement CAV)
- Lexical features (TF-IDF → SVD)
- An **account-level prior** capturing the author's typical engagement rate

**Crucially, nothing leaks the future.** No dump-time favorite/retweet counts as features. No current follower counts (they're snapshots, not histories). No undated likes. The model sees only what was knowable at posting time.

---

## Scoreboard

The primary metric is **within-account Spearman** by stratum — given an author's tweets, how well does the model rank them by eventual engagement?

| Metric | Value |
|---|---|
| Within-account Spearman (non-replies, weighted mean) | **0.246** |
| Within-account Spearman (replies, weighted mean) | **0.307** |
| Global Spearman (non-replies) | **0.616** |
| Rate RMSE | **0.588** |
| Training rows / accounts | 88,500 / 868 |
| Validation rows / accounts | 22,125 / 594 |
| Features | 326 |

*Release-v1 checkpoint trained on X-complete full data (~110k / 934 accounts).*

---

## What makes this hard?

**1. Leakage is everywhere.** The naive approach uses a tweet's eventual favorites/retweets as features for that same tweet. Even "just the author's average engagement on previous tweets" is dangerous if that average includes dump-time totals that post-date the candidate. We ban all of that.

**2. The signal is sparse.** Most tweets get little engagement. The model must pick out the ~10% that will outperform an author's own median.

**3. Reply vs. standalone tweets behave differently.** A thoughtful reply to a hot thread has a very different engagement profile than an original post. We model these separately and finally blend them with a within-author ranker.

---

## How it works

### Training pipeline

```
enriched_tweets.parquet (CA dump, ~900MB)
        │
  ┌─────┴──────────┐
  │  prepare.py     │  ─── Feature engineering
  │                  │      - Text → MiniLM → PCA
  │                  │      - TF-IDF → SVD
  │                  │      - Ego history (replies/quotes/mentions received)
  │                  │      - Parent context (CA + X API fallback)
  │                  │      - Calendar / platform eras
  └─────┬──────────┘
        │
  ┌─────┴──────────┐
  │  dataset.parquet │
  └─────┬──────────┘
        │
  ┌─────┴──────────┐
  │  train.py       │  ─── Training
  │                  │      - Exposure-rate target (maturity-corrected)
  │                  │      - Account target encoding (LOO + shrinkage)
  │                  │      - Residual embedding CAV
  │                  │      - Asymmetric MSE loss
  │                  │      - Hybrid level + within-author LambdaRank
  └─────┬──────────┘
        │
  ┌─────┴──────────┐
  │  model.txt      │  ─── LightGBM booster
  │  + transforms   │      + PCA, TF-IDF→SVD, residualizer
  └────────────────┘
```

### Feature groups

| Group | Description | Count |
|---|---|---|
| **Structure** | text_len, has_url, has_media, n_mentions, caps_ratio, is_reply, etc. | ~20 |
| **Calendar** | hour, day-of-week, month, year, platform era (pre-timeline → musk_late) | ~12 |
| **Ego history** | inbound replies/quotes/mentions (last 7d, 30d, all-time, per-tweet rates), crosses with media/reply flags | ~32 |
| **Parent context** | reply/quote parent MiniLM cosine, parent age, parent's own pre-T replies, same-author flag | ~16 |
| **URL structure** | text urls, domain-level features (twitter, youtube, github, etc.) | ~8 |
| **Account prior** | `account_mean_rate` (shrunken LOO prior, caller-supplied) + `log_account_n` | 2 |
| **Residual CAV** | projection of MiniLM residual onto engagement direction (after regressing out baseline) | 2 |
| **Embedding PCA (MiniLM)** | 96-d PCA of `all-MiniLM-L6-v2` | 96 |
| **Text SVD** | 128-d TF-IDF → TruncatedSVD | 128 |

### The target

```
y = log1p(eng / (min(age_days, 30) + 1))

where eng = favorite_count + retweet_count
```

This is an **exposure rate** — the intensity of engagement per unit tweet lifetime, capped at 30 days so dump maturity doesn't distort scores. Retweets (`RT @...`) are excluded from training.

### Leakage rules (non-negotiable)

| Allowed | Banned |
|---|---|
| Pre-T reply/quote/mention arrivals to the author | Dump-time `favorite_count` / `retweet_count` of any tweet |
| Parent tweet content + pre-T parent reply count | Current `all_account.num_followers` (snapshot, leaks future) |
| Calendar / platform era | `account_display_name` (mutable, no as-of-T history) |
| Text and its structure | Undated likes |
| Account-level shrunken mean of *prior* tweets' rates | Username fed into an LM for "reputation" scoring |

---

## What's in this repository

### Inference bundle

| File | Role |
|---|---|
| `model_level.txt` | LightGBM level booster (exposure-rate target) |
| `model_rank_nr.txt` / `model_rank_rp.txt` | Hybrid stratum rankers (non-reply / reply) |
| `emb_pca.joblib` | MiniLM → PCA transform |
| `tfidf.joblib` + `text_svd.joblib` | TF-IDF → TruncatedSVD |
| `res_emb_cav_direction.npy` | Residual engagement CAV direction (384-d) |
| `res_emb_baseline_coef.npy` + `res_emb_baselines.json` | OLS residualizer coefficients |
| `feature_schema.json` | Ordered features, dtypes, defaults, groups |
| `constants.json` | Global priors, encoder ID, exposure parameters |
| `metrics.json` | Full holdout evaluation |
| `predict.py` | Reference scorer (JSON in → score out) |

The bundle excludes: training data, per-account TE maps, emb-kNN train index, CA graph caches. Those are private to the research pipeline.

### Research pipeline (not in this repo)

- `prepare.py` — Full feature engineering from CA parquet dump
- `train.py` — Model training with all ablation knobs
- `analyze.py` — Detailed evaluation with per-stratum breakdowns
- `experiments.md` — Complete experiment log (50+ runs with outcomes)
- `AGENTS.md` — Technical guidance for the data and leakage rules

---

## Quick start: scoring a tweet

```bash
# Install dependencies
pip install lightgbm scikit-learn numpy pandas joblib pyarrow sentence-transformers

# Download the bundle from Hugging Face
# (you're looking at it — clone or download the files above)

# Score a single tweet
cat > tweet.json << 'EOF'
{
  "tweet_id": "123456789",
  "full_text": "just shipped something i've been working on for months. feels good.",
  "is_reply": false,
  "has_media": false,
  "has_url": false,
  "n_mentions": 0,
  "n_hashtags": 0,
  "account_mean_rate": 0.15,
  "log_account_n": 3.5
}
EOF

python predict.py --bundle . --input tweet.json
```

For full context (parent tweets, ego history), populate the remaining schema fields (see `feature_schema.json`). Missing fields default to 0 or the global mean.

### The `account_mean_rate` recipe

This is the most important caller-supplied value. Given the author's earlier tweets with known engagement:

1. For each tweet in the author's history, compute the same exposure rate `y_i`
2. Let `n` = number of historical tweets, `y_bar` = their mean rate
3. Shrink toward the global prior using `m = 20`:

```
account_mean_rate = (n * y_bar + m * global_mean) / (n + m)
log_account_n = log1p(n)
```

If `n = 0`: use `global_mean ≈ 0.151`, `log_account_n = 0`.

---

## Research process

This project ran over 50 experiments across 18 months. The full log is in `experiments.md`, but the headline lessons are:

- **Signal > technique.** The biggest lifts came from denser leakage-safe history (inbound replies/quotes, mentions, parent context from X API) — not from embedding discovery, LM labels, or modeling tricks.
- **Metric selection matters.** Within-author ranking (does the model rank an author's own tweets correctly?) is harder and more useful than cross-author level accuracy.
- **Failure modes are open.** Viral non-replies are still under-predicted. Media/image understanding is weak. Reply dyad features don't yet pierce the noise floor.

---

## License

Apache 2.0

## Citation

```bibtex
@misc{will-it-bang,
  author = {Chris Carroll Smith},
  title = {will-it-bang: Predicting Tweet Engagement from Pre-Publication Features},
  year = {2026},
  howpublished = {\url{https://huggingface.co/chriscarrollsmith/will-it-bang}}
}
```

---

*Built with the [Twitter Community Archive](https://www.community-archive.org) data, LightGBM, and sentence-transformers/all-MiniLM-L6-v2.*