README.md

---
language:
- en
license: mit
tags:
- text-classification
- html-analysis
- article-extraction
- xgboost
- web-scraping
datasets:
- Allanatrix/articles
metrics:
- accuracy
- f1
library_name: xgboost
---

# Article Extraction Outcome Classifier

A fast, lightweight classifier that categorizes web article extraction outcomes with 90% accuarcy

## Model Description

This model predicts whether HTML extraction succeeded, failed, or returned a non-article page. It combines rule-based heuristics for speed with XGBoost for accuracy on ambiguous cases.

## Classes

| Class | Description |
|-------|-------------|
| `full_article_extracted` | Complete article successfully extracted |
| `partial_article_extracted` | Article partially extracted (incomplete) |
| `api_provider_error` | External API/service failure |
| `other_failure` | Low-confidence failure (catch-all) |
| `full_page_not_article` | Page is not an article (nav, homepage, etc.) |

## Performance

~90% accuracy on a large, real-world test set, with strong performance on dominant classes

| Class                     | Precision | Recall | F1-score | Support |
| ------------------------- | --------- | ------ | -------- | ------- |
| full_article_extracted    | 0.91      | 0.84   | 0.87     | 1,312   |
| partial_article_extracted | 0.76      | 0.63   | 0.69     | 92      |
| api_provider_error        | 0.95      | 0.93   | 0.94     | 627     |
| other_failure             | 0.41      | 0.28   | 0.33     | 44      |
| full_page_not_article     | 0.92      | 0.97   | 0.94     | 11,821  |
| **Accuracy**              | —         | —      | **0.90** | 13,852  |
| **Macro Avg**             | 0.79      | 0.73   | 0.72     | 13,852  |
| **Weighted Avg**          | 0.90      | 0.90   | 0.90     | 13,852  |


```python
import numpy as np
import torch
from sklearn.preprocessing import StandardScaler

# Load model
artifacts = torch.load("artifacts.pt")
scaler = artifacts["scaler"]
model = artifacts["xgb_model"]
id_to_label = artifacts["id_to_label"]

# Extract features (26 features from HTML prefix)
def extract_features(html: str, max_chars: int = 64000) -> dict:
    prefix = html[:max_chars].lower()
    
    features = {
        "length_chars": len(html),
        "prefix_len": len(prefix),
        "ws_ratio": sum(c.isspace() for c in prefix) / len(prefix) if prefix else 0,
        "digit_ratio": sum(c.isdigit() for c in prefix) / len(prefix) if prefix else 0,
        "punct_ratio": sum(c in ".,;:!?" for c in prefix) / len(prefix) if prefix else 0,
        # Keyword counts
        "cookie": prefix.count("cookie") + prefix.count("consent"),
        "subscribe": prefix.count("subscribe") + prefix.count("newsletter"),
        "legal": prefix.count("privacy policy") + prefix.count("terms of service"),
        "error": prefix.count("error") + prefix.count("timeout") + prefix.count("rate limit"),
        "nav": prefix.count("home") + prefix.count("menu") + prefix.count("navigation"),
        "article_kw": prefix.count("published") + prefix.count("reading time"),
        "meta_article_kw": prefix.count("og:article") + prefix.count("article:published"),
        # Tag counts
        "n_p": prefix.count("<p"),
        "n_a": prefix.count("<a"),
        "n_h1": prefix.count("<h1"),
        "n_h2": prefix.count("<h2"),
        "n_h3": prefix.count("<h3"),
        "n_article": prefix.count("<article"),
        "n_main": prefix.count("<main"),
        "n_time": prefix.count("<time"),
        "n_script": prefix.count("<script"),
        "n_style": prefix.count("<style"),
        "n_nav": prefix.count("<nav"),
    }
    
    # Density features
    kb = len(prefix) / 1000.0
    features["link_density"] = features["n_a"] / kb if kb > 0 else 0
    features["para_density"] = features["n_p"] / kb if kb > 0 else 0
    features["script_density"] = features["n_script"] / kb if kb > 0 else 0
    features["heading_score"] = features["n_h1"] * 3 + features["n_h2"] * 2 + features["n_h3"]
    
    return features

# Predict
features = extract_features(html_string)
NUM_COLS = ["length_chars", "prefix_len", "ws_ratio", "digit_ratio", "punct_ratio",
            "cookie", "subscribe", "legal", "error", "nav", "article_kw", "meta_article_kw",
            "n_p", "n_a", "n_h1", "n_h2", "n_h3", "n_article", "n_main", "n_time",
            "n_script", "n_style", "n_nav", "link_density", "para_density", 
            "script_density", "heading_score"]

X = np.array([features[col] for col in NUM_COLS]).reshape(1, -1).astype(np.float32)
X_scaled = scaler.transform(X)
prediction = model.predict(X_scaled)[0]

print(f"Outcome: {id_to_label[prediction]}")
```

### Optional: Rule-Based Fast Path

For 80%+ of cases, you can skip the model entirely:

```python
def apply_rules(features: dict) -> str | None:
    """Returns class label or None if ambiguous."""
    if features["error"] >= 3:
        return "api_provider_error"
    
    if features["meta_article_kw"] >= 2 and features["n_p"] >= 10:
        return "full_article_extracted"
    
    if features["nav"] >= 5 and features["n_p"] < 5 and features["link_density"] > 20:
        return "full_page_not_article"
    
    return None  # Use ML model

# Try rules first
rule_result = apply_rules(features)
if rule_result:
    print(f"Outcome (rule-based): {rule_result}")
else:
    # Fall back to model
    prediction = model.predict(X_scaled)[0]
    print(f"Outcome (model): {id_to_label[prediction]}")
```

## Training Data

- **Dataset:** `Allanatrix/articles` (194,183 HTML pages)
- **Labeled samples:** 138,523 (LLM-labeled)
- **Labeling method:** Distillation from large language models
  - **Primary teacher:** GPT-5
  - **Secondary / adjudicator:** Qwen
- **Train/Val/Test split:** 110,819 / 13,852 / 13,852
- **Class distribution:** ~85% non-articles, ~10% full articles, ~4% errors, ~1% partial articles

## Model Details

- **Algorithm:** XGBoost (GPU-trained)
- **Features:** 26 hand-crafted features (HTML structure, keyword counts, density metrics)
- **Training:** 500 boosting rounds with early stopping
- **Hardware:** Single GPU (CUDA)
- **Training time:** ~6 minutes

### Features Used

- **Content statistics:** length, whitespace ratio, digit and punctuation ratios
- **Keyword signals:** error messages, article indicators, navigation text
- **HTML structure:** paragraph, link, heading, script, style, and nav tag counts
- **Density metrics:** links/KB, paragraphs/KB, scripts/KB, heading score

## Limitations

- Only analyzes the first 64KB of HTML (important metadata must appear early)
- Labels are generated by LLMs rather than direct human annotation
- Some classes (e.g. `other_failure`) have limited representation
- Optimized primarily for English-language web pages

## Intended Use

**Primary use cases:**
- Quality control for article extraction pipelines
- Monitoring extraction API health and failure modes
- Fast filtering of non-article pages before downstream processing
- Analytics on extraction success and failure rates

**Not suitable for:**
- Language detection
- Content quality assessment
- Paywall detection
- Full content extraction

## Model Card Authors

Allanatrix