File size: 9,580 Bytes

---
library_name: transformers
license: cc-by-nc-4.0
tags:
- html
- document-embedding
- feature-extraction
pipeline_tag: feature-extraction
---

# HTML-LM

HTML-LM is a compact encoder model designed to generate general-purpose embeddings for HTML web pages, capturing both textual content and HTML structure. The embeddings can be used as inputs to lightweight downstream models for various classification and regression tasks.

HTML-LM representations can be reused across a **wide range of downstream applications**, supporting tasks such as **classification, regression, clustering, and broader web-document understanding**. Specifically, we use them for:

* **Explicit content classification** — determining whether a webpage contains explicit or adult content.
* **Article-page detection** — identifying whether a webpage primarily contains article or editorial content.
* **Product-page detection** — determining whether a webpage represents a product or e-commerce listing.
* **Page clustering** — grouping webpages with similar content and structural characteristics.
* **Page-quality regression** — estimating the overall quality and usefulness of a webpage.
* **Web-spam detection** — estimating the degree of spam or low-quality content on a webpage.

## Model details

| Property        | Value          |
| --------------- | -------------- |
| Architecture    | [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) |
| Parameters      | 154M           |
| Hidden size     | 768            |
| Layers          | 22             |
| Attention heads | 12             |
| Context length  | 8192 (trained with 4096) |
| Vocabulary      | 57K            |

The model was trained from scratch for one pass over the training corpus using a combination of:

* Masked Language Modeling (MLM)
* [Bag-of-Words prediction](https://arxiv.org/abs/2401.11248) from `[CLS]`
* Contrastive distillation from [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) and [Seznam SeLLMa 8B](https://blog.seznam.cz/2024/10/diana-hlavacova-sellma-aneb-jak-v-seznamu-krotime-drave-jazykove-modely/)

## Training data

HTML-LM was trained on **100M HTML documents** sampled from the Seznam.cz crawl database.

The corpus contains approximately:

* 53% Czech domains
* 34% primarily English-language domains
* 9% other European domains
* 4% other domains

## HTML preprocessing

> ⚠️ HTML-LM expects HTML documents to be preprocessed in a specific way, which is handled by the bundled AutoProcessor.from_pretrained("Seznam/html-lm").

The included `HTMLLMProcessor` performs the following preprocessing steps:

1. Parsing the document into a DOM.
2. Removing scripts, styles, comments, and irrelevant subtrees.
3. Preserving text and selected structurally meaningful HTML tags.
4. Removing HTML attributes.
5. Simplifying the DOM hierarchy.
6. Normalizing whitespace.

> **Performance note:** The bundled processor may be slow out of the box for high-throughput or large-scale workloads. For improved performance, we recommend using it with `torch.utils.data.DataLoader` and setting `num_workers > 1`. See the example below.

## Usage

```bash
pip install transformers torch beautifulsoup4 lxml
```

```python
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Seznam/html-lm"

# <class 'transformers_modules.hf.processor.HTMLLMProcessor'>
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)
# <class 'transformers.models.modernbert.modeling_modernbert.ModernBertModel'>
model = AutoModel.from_pretrained(model_id).to("cuda")
model.eval()

html_pages = [
    "<html><body><h1>Example</h1><p>Some text.</p></body></html>",
]

inputs = processor(
    html_pages,
    padding=True,
    truncation=True,
    max_length=4096,
    return_tensors="pt",
)

with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"].to("cuda"),
        attention_mask=inputs["attention_mask"].to("cuda")
    )

# [CLS] representation = document embedding
document_embeddings = outputs.last_hidden_state[:, 0, :]

print(document_embeddings.shape)
# torch.Size([1, 768])

print(document_embeddings)
# tensor([[-2.0882e-01,  9.9287e-01, -1.0417e+00,  9.4297e-01, -8.4005e-01,
#          1.1509e+00,  8.4631e-01, -3.2688e-01,  7.1889e-01,  3.8875e-02,
#          ... 
#          -2.9921e-01, -1.0583e+00,  1.4555e+00]], device='cuda:0')
```

## Performance

The models were evaluated using the following methodology: we first froze each model and extracted embeddings from the task-specific datasets. These embeddings were then used to train and evaluate lightweight, task-specific MLPs across five downstream tasks. The performance was then aggregated using NMM (Normalized Metric Mean), which represents the mean normalized improvement over a random baseline.

| Model                   | Parameters |      NMM ↑ |
| ----------------------- | ---------: | ---------: |
| ModernBERT base         |       149M |     0.4835 |
| text-embedding-3-small  |          — |     0.6544 |
| jina-embeddings-v3      |       570M |     0.6466 |
| Qwen3-Embedding-8B      |         8B |     0.6571 |
| SeLLMa 8B               |         8B |     0.6103 |
| **HTML-LM Base**        |   **154M** | **0.7001** |

HTML-LM achieves the highest aggregated score in this evaluation despite being substantially smaller than the evaluated large embedding models.

## Examples

### Preprocessing Example

`HTMLLMProcessor` exposes `preprocess_html(html: str)` for obtaining the cleaned HTML before tokenization.

```python
from transformers import AutoProcessor

model_id = "Seznam/html-lm"

# <class 'transformers_modules.hf.processor.HTMLLMProcessor'>
processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

html = """
<html>
    <body>
        <div></div>
        <h1 class='big'>Example</h1>
        <p class='small'>Some article text.</p>
    </body>
</html>
"""

print(processor.preprocess_html(html))
# <html><body><h1>Example</h1><p>Some article text.</p></body></html>
```

### Faster Preprocessing

For faster preprocessing, you can use the code snippet below, which leverages multiprocessing to accelerate the preprocessor.

```python
import torch
from transformers import AutoProcessor, BatchEncoding

# must be an instance of `torch.utils.data.Dataset` yielding raw HTML strings
dataset = YOUR_DATASET_HERE

model_id = "Seznam/html-lm"

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
)

def collate_fn(html_inputs: list[str]) -> BatchEncoding:
    return processor(
        html_inputs,
        padding=True,
        truncation=True,
        max_length=4096,
        return_tensors="pt"
    )

loader = torch.utils.data.DataLoader(
    dataset, 
    batch_size=16, 
    collate_fn=collate_fn,
    num_workers=32
)
```

### Testing

To verify your own integration, the repository ships a small end-to-end example in
`test/`. It contains one sample page together with the expected output of **every**
stage of the pipeline, so you can check each step in isolation:

| File                  | Content                                                                       |
| --------------------- | ----------------------------------------------------------------------------- |
| `test/raw.html`       | Input: a raw HTML page with attributes, `<meta>` tags and a deeply nested DOM. |
| `test/processed.html` | Expected output of `processor.preprocess_html(raw_html)`.                      |
| `test/tokenized.npy`  | Expected `input_ids` — `int64`, shape `(133,)`, no padding, no truncation.     |
| `test/embedded.npy`   | Expected document embedding — `float32`, shape `(768,)`.                       |

The reference embedding was produced with the model loaded in `torch.float32` and taken
as the `[CLS]` vector (`outputs.last_hidden_state[:, 0, :]`).

```python
import numpy as np
import torch
from transformers import AutoModel, AutoProcessor

model_id = "Seznam/html-lm"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float32).to("cuda")
model.eval()

raw_html = open("test/raw.html").read()
expected_html = open("test/processed.html").read()
expected_ids = np.load("test/tokenized.npy")
expected_embedding = np.load("test/embedded.npy")

# 1) preprocessing
processed_html = processor.preprocess_html(raw_html)
assert processed_html == expected_html

# 2) tokenization
inputs = processor([raw_html], return_tensors="pt")
assert np.array_equal(inputs["input_ids"][0].numpy(), expected_ids)

# 3) embedding
with torch.no_grad():
    outputs = model(
        input_ids=inputs["input_ids"].to("cuda"),
        attention_mask=inputs["attention_mask"].to("cuda"),
    )
embedding = outputs.last_hidden_state[0, 0, :].float().cpu().numpy()

cos = np.dot(embedding, expected_embedding) / (
    np.linalg.norm(embedding) * np.linalg.norm(expected_embedding)
)
assert cos > 0.999, cos
```

## License

This model is released under the **CC BY-NC 4.0** license.

## Citation

If you use HTML-LM, please cite:

```bibtex
@inproceedings{dvorak2026html-lm,
  title     = {Size Matters: Foundation Model for Czech HTML documents},
  author    = {Dvořák, Martin and Tlustoš, Vít and Voronin, Artyom and Habrovec, Martin and Podlesná, Kateřina and Rišová, Barbora and Vonášek, Josef},
  year      = {TBD},
  publisher = {TBD}
}
```

## Acknowledgements

HTML-LM was developed by the **Seznam.cz Research team** as part of the HTML-LM project.