# Load model directly
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("hermanda/robeczech-propaganda")
model = AutoModelForSequenceClassification.from_pretrained("hermanda/robeczech-propaganda", device_map="auto")robeczech-propaganda
Fine-tuned ufal/robeczech-base classifier for pro-Kremlin propaganda in Czech/Slovak news-comment text. Comment-only input (no article context).
v2 (current): retrained on human-reviewed Argilla labels (previously: Haiku-teacher-distilled labels). Full finetune, 4 epochs, lr=2e-5, 2x positive-class oversampling.
Evaluation (605-row held-out test set, 69 positives)
| v1 (Haiku-labeled) | v2 (Argilla-labeled, current) | |
|---|---|---|
| F1 | 0.375 | 0.487-0.521 (run-to-run variance; see repo RESEARCH_NOTES.md) |
| Precision | 0.231 | 0.437-0.440 |
| Recall | 1.000 | 0.551-0.638 |
| ROC AUC | 0.883 | 0.878 |
v1 and v2 have near-identical AUC (ranking quality) — v1's low F1 was primarily a threshold-calibration problem, not a weaker model. v2 improves F1 mainly through better label quality and, downstream, an explicitly recalibrated operating threshold (see below).
Score calibration / recommended thresholds
Softmax score is not comparable to v1's — v2 outputs are calibrated very differently (fewer, more confident high scores). Precision/recall at threshold, measured on the same held-out test set:
| Threshold | Precision | Recall | Tier |
|---|---|---|---|
| ≥ 0.9659 | 1.000 | 0.101 | high-confidence propaganda |
| 0.9254 – 0.9659 | ~0.59-0.71 (band) | up to 0.246 cumulative | suspected propaganda |
| < 0.9254 | ~0.09 (band) | — | clean |
Caveat: these thresholds are set from only 605 test rows (69 positives); the 0.9659 cutoff rests on just 7 true positives at that exact operating point, so treat it as directional, not exact. At this calibration, roughly 75% of actual propaganda comments fall into the "clean" tier — this is a high-precision, low-recall configuration by design (optimizing for few false alarms), not a broad detector.
Training and evaluation data
Real human-reviewed labels from an internal Argilla annotation queue (~11.4k training rows after excluding the fixed test set, 2x oversampled to ~12.7k). Comment-only input (article/parent-comment context was tested and consistently hurt performance at this scale).
Framework versions
- Transformers 5.x, PyTorch 2.x
- Downloads last month
- 47
Model tree for hermanda/robeczech-propaganda
Base model
ufal/robeczech-base
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="hermanda/robeczech-propaganda")