File size: 4,205 Bytes
ff1810f 97b7d5a ff1810f 97b7d5a ff1810f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 | ---
library_name: skops
license: mit
pipeline_tag: tabular-regression
tags:
- scikit-learn
- tabular-regression
- engagement-prediction
- viralst
- poc
model-index:
- name: raviearjun/engagement-predictor
results:
- task:
type: tabular-regression
dataset:
name: viralst-engagement-synthetic-poc
type: synthetic-poc
metrics:
- type: mean_absolute_error
value: 0.01142
- type: root_mean_squared_error
value: 0.01458
- type: r_squared
value: 0.71023
---
# engagement-predictor
A regression model (`scikit-learn` `Ridge` inside a `Pipeline`) that predicts
`engagement_score` for brand-creator campaign content, part of the **Viralst**
application.
## Overview
Given campaign brief attributes (tone, hook type, target audience, creator
tier, objective, etc. — typically extracted from a brief document via an LLM
parser) and planned content execution details (duration, upload time,
caption), the model predicts the expected `engagement_score`
(`(likes + comments + shares) / views`).
It is served through the `POST /engagement/predict` endpoint of the Viralst
`ai/` service.
## Training data and methodology
The model is trained on a **synthetically generated dataset** (2,000 rows,
300 unique briefs, 30 unique brands) produced by
`training/generate_poc_data.py`. Feature-to-target relationships were
deliberately encoded (with variable noise per row) so the training pipeline
— feature engineering, model selection, evaluation, and serving — could be
validated end-to-end ahead of onboarding real campaign data. Results below
characterize how well the model recovers the encoded synthetic relationships,
not real-world campaign performance.
## Training procedure
- Model: `sklearn.linear_model.Ridge` inside a `Pipeline` (`OneHotEncoder`
for single-value categorical features, `StandardScaler` for numeric and
multi-hot features).
- Alpha selected via grid search using `GroupKFold` (grouped by `brief_id`),
so evaluation reflects generalization to unseen briefs rather than
leaking the same brief across train/test folds.
- Serialized with `skops.io.dump` (not raw pickle) for safer loading.
## Evaluation results
```
{
"alpha": 10.0,
"n_rows": 2000,
"n_groups": 300,
"cv_mae": 0.011419824566734465,
"cv_rmse": 0.01458130687031906,
"cv_r2": 0.7102257580449983,
"baseline_mae": 0.021489796999999998,
"baseline_rmse": 0.027087352583401354
}
```
Mean-predictor baseline RMSE: `0.027087352583401354`. The final model outperforms
this baseline by a wide margin on the synthetic dataset described above.
## Feature schema
See `config.json` in this repo for the full list and order of feature
columns the model expects. The canonical definition lives in
`src/engagement_predictor/features.py` in the application repository
(https://github.com/<your-org>/viralst).
## Training dataset
The raw synthetic dataset used to train this model is included in this repo
under `dataset/` (`brands.csv`, `briefs.csv`, `contents.csv` — 30
brands, 300 briefs, 2000 content rows). It was generated by
`training/generate_poc_data.py`; see "Training data and methodology" above
for why it is synthetic and what it does and does not represent.
## How to use
```python
import pandas as pd
import skops.io as sio
from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="raviearjun/engagement-predictor", filename="engagement_ridge.skops")
untrusted = sio.get_untrusted_types(file=path)
model = sio.load(path, trusted=untrusted)
# X must be a DataFrame with columns matching config.json -> feature_columns
prediction = model.predict(X)
```
## Limitations
- Trained on synthetic data; does not reflect real-world engagement
distributions, platform trends, or actual creator/audience behavior.
- Feature quality depends on upstream LLM extraction from brief documents;
prediction quality degrades if that extraction is inaccurate or incomplete.
- Not yet validated against real campaign data from brands outside the
training set.
## Reproducibility
Training environment (see `requirements.txt` in this repo for exact pins):
```
scikit-learn==1.7.2
skops==0.14.0
numpy==1.26.4
pandas==2.3.3
```
|