File size: 2,051 Bytes
a498c14 55af022 a498c14 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 |
---
license: apache-2.0
language:
- en
metrics:
- name: exact_match
value: 0.9777
verified: false
pipeline_tag: text-classification
library_name: sklearn
datasets:
- Lucanix/meta-paper-classify
---
# WITTR or Wait, Is That The References?
A lightweight Naive Bayes classifier
designed to detect whether a given text should be **filtered out** from an academic paper
before being used for **language-model pretraining** or **RAG (Retrieval-Augmented Generation)**.
---
## 🧠 Concept
WITTR is trained on text from academic and research-style corpora.
It distinguishes between two main categories:
- ✅ **0** – meaningful content such as main academic paragraphs, analysis, or discussion
- ❌ **1** – metadata or non-content text such as author names, URLs, DOIs, references, publication years, or institutional names
The goal is to provide an **automatic corpus-cleaning step**
that keeps only the informative text suitable for model training.
---
## ⚙️ Model Details
- **Framework:** scikit-learn
- **Architecture:** Multinomial Naive Bayes
- **Vectorization:** TF–IDF
- **Language:** English academic text
- **Accuracy:** ≈ 0.9777
- **Intended Use:** academic text preprocessing, corpus filtering before LLM or RAG pipelines
---
## 📦 Files
| File | Description |
|------|--------------|
| `wittr_naive.pkl` | Trained Naive Bayes classifier |
| `wittr_naive_vectorizer.pkl` | TF–IDF vectorizer (must be used with the model) |
| `wittr_naive.py` | Simple python script |
| `README.md` | This documentation |
---
## 🚀 Usage Example
```python
from huggingface_hub import hf_hub_download
import joblib
repo = "Lucanix/wittr"
clf = joblib.load(hf_hub_download(repo, "wittr_naive.pkl"))
vectorizer = joblib.load(hf_hub_download(repo, "wittr_naive_vectorizer.pkl"))
text = ["18 Bales G. S. & and Chrzan, D. C. Dynamics of irreversible island growth during submonolayer epitaxy. *Phys. Rev. B* **50**, 6057–6067 (1994)."]
X = vectorizer.transform(text)
print(clf.predict(X)) # → [1]
``` |