--- license: mit language: - en base_model: xlm-roberta-base pipeline_tag: text-classification tags: - youtube-comments - bot-detection - media-literacy - xlm-roberta widget: - text: "niche low effort . few likes . no replies . first!!" - text: "niche genuine . many likes . has replies . the part at 3:42 where he almost fell was insane, glad he stuck the landing" --- # Kratt Comment Authenticity Classifier Fine-tuned `xlm-roberta-base` that scores a YouTube comment as **`bot`** or **`authentic`**. Built for Kratt, a media-literacy tool that helps viewers read comment sections more critically — it flags likely bot activity, it never deletes or hides anything. ## ⚠️ Required input format — read this first This model was **not** trained on raw comment text. Every input was built as: ``` niche . . . ``` Concrete example: ``` niche low effort . few likes . no replies . first!! ``` | Piece | Values | Rule | |---|---|---| | `niche_tag` | `genuine`, `copycat`, `low effort` | property of the **video**, hyphens replaced with spaces | | likes bucket | `few likes` (<2), `some likes` (2–9), `many likes` (≥10) | from `like_count` | | replies bucket | `no replies` (0), `has replies` (>0) | from `reply_count` | | comment text | lowercased **except ALL-CAPS tokens** (shouting is a signal); URLs/@mentions/#hashtags replaced with `` / `` / ``; emoji kept as-is | see Preprocessing below | **If you skip this prefix and feed raw comment text, the model will perform far worse than the reported metrics** — the niche and engagement context are load-bearing, not optional metadata. This was a deliberate design choice: `niche_tag` is a video-level property that determines a large part of the label, so the model cannot infer it from the comment text alone. The three placeholder tokens ``, ``, `` are registered as tokenizer special tokens — load the tokenizer from this repo, not a fresh `xlm-roberta-base` tokenizer, or these will be split into sub-word garbage. ## How to use ```python import re from transformers import AutoTokenizer, AutoModelForSequenceClassification import torch REPO = "geraldadli/Kratt" # replace with your HF repo id tokenizer = AutoTokenizer.from_pretrained(REPO) model = AutoModelForSequenceClassification.from_pretrained(REPO) INVISIBLE_RE = re.compile('[​‎‏⁠⁡⁢⁣⁤­]') URL_RE = re.compile(r'(?:https?://|www\.)\S+', re.IGNORECASE) USER_RE = re.compile(r'@[\w.\-]+') TAG_RE = re.compile(r'#\w+') def clean_text(text): s = INVISIBLE_RE.sub('', str(text)) s = URL_RE.sub(' ', s) s = USER_RE.sub(' ', s) s = TAG_RE.sub(' ', s) return ' '.join(t if (len(t) >= 2 and t.isupper()) else t.lower() for t in s.split()) def like_phrase(n): return 'many likes' if n >= 10 else ('some likes' if n >= 2 else 'few likes') def reply_phrase(n): return 'has replies' if n > 0 else 'no replies' def build_input(text, niche_tag, like_count, reply_count): return (f"niche {niche_tag.replace('-', ' ')} . {like_phrase(like_count)} . " f"{reply_phrase(reply_count)} . {clean_text(text)}") model_input = build_input( text="first!!", niche_tag="low-effort", like_count=0, reply_count=0) inputs = tokenizer(model_input, truncation=True, max_length=128, return_tensors="pt") with torch.no_grad(): probs = torch.softmax(model(**inputs).logits, dim=-1)[0] print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)}) ``` ## Labels | id | label | |---|---| | 0 | `bot` | | 1 | `authentic` | ## Training data ~[FILL IN: total rows after filtering] YouTube comments, per-comment-tagged by a local LLM (Qwen2.5-7B-Instruct) into `genuine` / `copycat` / `low-effort` / `ads_spam`, combined with each comment's video-level `niche_tag` (`genuine` / `copycat` / `low-effort`). **Labels are weak/derived, not human-annotated.** A `(niche_tag, comment_tag)` → authenticity-score matrix converts the combination into a binary label (`authentic` if score ≥ 0.5) and a per-sample training weight (`|score − 0.5| × 2`). The core idea: the same comment type means different things in different niches — e.g. a low-effort comment under a genuine-niche video (tutorial, stunt) is usually just a casual human (high authenticity), while the same comment type under a low-effort video (fast-consume clips) matches an observed bot pattern (low authenticity). Ambiguous combinations (e.g. `copycat` in a `genuine` niche, score 0.5) get a training weight near zero and barely influence the model. `ads_spam`-tagged comments were excluded from training — spam detection is handled by a separate rule engine in the product, not this classifier. ## Training procedure - Base model: `xlm-roberta-base` (multilingual, cased — casing matters for the ALL-CAPS signal) - 3 epochs, batch size 16 (grad. accumulation ×2), LR 2e-5, max sequence length 128, `fp16` - **Grouped split by `video_id`** (no video's comments appear in both train and test) with a per-video cap of 500 comments so one large video can't dominate a niche's data or degenerate the split, plus a guard requiring ≥2 videos and 15–25% share in the test set - **Weighted loss**: per-sample weight from the authenticity matrix (above) × class weight, where class weights are computed on the **effective weighted mass** per class (not raw row counts) so the sample weights and class balance don't fight each other in the loss ## Evaluation Test set: grouped by video, 8581 comments across 18 videos. ``` precision recall f1-score support bot 0.72 0.79 0.75 991 authentic 0.77 0.70 0.74 1009 accuracy 0.75 2000 macro avg 0.75 0.75 0.75 2000 weighted avg 0.75 0.75 0.75 2000 ``` Positive class for the confusion matrix is `bot`; a false positive means an **authentic comment flagged as bot** — the costlier error for a media-literacy tool, since it wrongly casts doubt on a real person. ## Limitations - **Weak labels**: ground truth comes from an LLM's per-comment judgment + a hand-picked authenticity matrix, not human annotation. Treat metrics as agreement with that scheme, not absolute truth — a hand-checked sample (`handcheck_sample.csv` in the training notebook) is the real validation step. - **English-only**: non-English comments were filtered out of training via language detection; behavior on other languages is untested. - **Requires the exact input format above** — this is not a general-purpose comment classifier. - Not intended to auto-moderate or delete content — designed to surface a bot-likelihood signal for human review, in line with the project's media-literacy (not censorship) goal. ## Intended use Input to Kratt's `/analyze` pipeline: score each fetched comment, aggregate into a bot-likelihood percentage and evidence breakdown shown to the end user.