| --- |
| library_name: transformers |
| license: cc-by-nc-4.0 |
| tags: |
| - html |
| - document-embedding |
| - feature-extraction |
| pipeline_tag: feature-extraction |
| --- |
| |
| # HTML-LM |
|
|
| HTML-LM is a compact encoder model designed to generate general-purpose embeddings for HTML web pages, capturing both textual content and HTML structure. The embeddings can be used as inputs to lightweight downstream models for various classification and regression tasks. |
|
|
| HTML-LM representations can be reused across a **wide range of downstream applications**, supporting tasks such as **classification, regression, clustering, and broader web-document understanding**. Specifically, we use them for: |
|
|
| * **Explicit content classification** — determining whether a webpage contains explicit or adult content. |
| * **Article-page detection** — identifying whether a webpage primarily contains article or editorial content. |
| * **Product-page detection** — determining whether a webpage represents a product or e-commerce listing. |
| * **Page clustering** — grouping webpages with similar content and structural characteristics. |
| * **Page-quality regression** — estimating the overall quality and usefulness of a webpage. |
| * **Web-spam detection** — estimating the degree of spam or low-quality content on a webpage. |
|
|
| ## Model details |
|
|
| | Property | Value | |
| | --------------- | -------------- | |
| | Architecture | [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) | |
| | Parameters | 154M | |
| | Hidden size | 768 | |
| | Layers | 22 | |
| | Attention heads | 12 | |
| | Context length | 8192 (trained with 4096) | |
| | Vocabulary | 57K | |
|
|
| The model was trained from scratch for one pass over the training corpus using a combination of: |
|
|
| * Masked Language Modeling (MLM) |
| * [Bag-of-Words prediction](https://arxiv.org/abs/2401.11248) from `[CLS]` |
| * Contrastive distillation from [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) and [Seznam SeLLMa 8B](https://blog.seznam.cz/2024/10/diana-hlavacova-sellma-aneb-jak-v-seznamu-krotime-drave-jazykove-modely/) |
|
|
| ## Training data |
|
|
| HTML-LM was trained on **100M HTML documents** sampled from the Seznam.cz crawl database. |
|
|
| The corpus contains approximately: |
|
|
| * 53% Czech domains |
| * 34% primarily English-language domains |
| * 9% other European domains |
| * 4% other domains |
|
|
| ## HTML preprocessing |
|
|
| > ⚠️ HTML-LM expects HTML documents to be preprocessed in a specific way, which is handled by the bundled AutoProcessor.from_pretrained("Seznam/html-lm"). |
| |
| The included `HTMLLMProcessor` performs the following preprocessing steps: |
| |
| 1. Parsing the document into a DOM. |
| 2. Removing scripts, styles, comments, and irrelevant subtrees. |
| 3. Preserving text and selected structurally meaningful HTML tags. |
| 4. Removing HTML attributes. |
| 5. Simplifying the DOM hierarchy. |
| 6. Normalizing whitespace. |
| |
| > **Performance note:** The bundled processor may be slow out of the box for high-throughput or large-scale workloads. For improved performance, we recommend using it with `torch.utils.data.DataLoader` and setting `num_workers > 1`. See the example below. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install transformers torch beautifulsoup4 lxml |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoProcessor |
| |
| model_id = "Seznam/html-lm" |
| |
| # <class 'transformers_modules.hf.processor.HTMLLMProcessor'> |
| processor = AutoProcessor.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| # <class 'transformers.models.modernbert.modeling_modernbert.ModernBertModel'> |
| model = AutoModel.from_pretrained(model_id).to("cuda") |
| model.eval() |
| |
| html_pages = [ |
| "<html><body><h1>Example</h1><p>Some text.</p></body></html>", |
| ] |
| |
| inputs = processor( |
| html_pages, |
| padding=True, |
| truncation=True, |
| max_length=4096, |
| return_tensors="pt", |
| ) |
| |
| with torch.no_grad(): |
| outputs = model( |
| input_ids=inputs["input_ids"].to("cuda"), |
| attention_mask=inputs["attention_mask"].to("cuda") |
| ) |
| |
| # [CLS] representation = document embedding |
| document_embeddings = outputs.last_hidden_state[:, 0, :] |
| |
| print(document_embeddings.shape) |
| # torch.Size([1, 768]) |
| |
| print(document_embeddings) |
| # tensor([[-2.0882e-01, 9.9287e-01, -1.0417e+00, 9.4297e-01, -8.4005e-01, |
| # 1.1509e+00, 8.4631e-01, -3.2688e-01, 7.1889e-01, 3.8875e-02, |
| # ... |
| # -2.9921e-01, -1.0583e+00, 1.4555e+00]], device='cuda:0') |
| ``` |
|
|
| ## Performance |
|
|
| The models were evaluated using the following methodology: we first froze each model and extracted embeddings from the task-specific datasets. These embeddings were then used to train and evaluate lightweight, task-specific MLPs across five downstream tasks. The performance was then aggregated using NMM (Normalized Metric Mean), which represents the mean normalized improvement over a random baseline. |
|
|
| | Model | Parameters | NMM ↑ | |
| | ----------------------- | ---------: | ---------: | |
| | ModernBERT base | 149M | 0.4835 | |
| | text-embedding-3-small | — | 0.6544 | |
| | jina-embeddings-v3 | 570M | 0.6466 | |
| | Qwen3-Embedding-8B | 8B | 0.6571 | |
| | SeLLMa 8B | 8B | 0.6103 | |
| | **HTML-LM Base** | **154M** | **0.7001** | |
|
|
| HTML-LM achieves the highest aggregated score in this evaluation despite being substantially smaller than the evaluated large embedding models. |
|
|
| ## Examples |
|
|
| ### Preprocessing Example |
|
|
| `HTMLLMProcessor` exposes `preprocess_html(html: str)` for obtaining the cleaned HTML before tokenization. |
|
|
| ```python |
| from transformers import AutoProcessor |
| |
| model_id = "Seznam/html-lm" |
| |
| # <class 'transformers_modules.hf.processor.HTMLLMProcessor'> |
| processor = AutoProcessor.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| html = """ |
| <html> |
| <body> |
| <div></div> |
| <h1 class='big'>Example</h1> |
| <p class='small'>Some article text.</p> |
| </body> |
| </html> |
| """ |
| |
| print(processor.preprocess_html(html)) |
| # <html><body><h1>Example</h1><p>Some article text.</p></body></html> |
| ``` |
|
|
| ### Faster Preprocessing |
|
|
| For faster preprocessing, you can use the code snippet below, which leverages multiprocessing to accelerate the preprocessor. |
|
|
| ```python |
| import torch |
| from transformers import AutoProcessor, BatchEncoding |
| |
| # must be an instance of `torch.utils.data.Dataset` yielding raw HTML strings |
| dataset = YOUR_DATASET_HERE |
| |
| model_id = "Seznam/html-lm" |
| |
| processor = AutoProcessor.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| ) |
| |
| def collate_fn(html_inputs: list[str]) -> BatchEncoding: |
| return processor( |
| html_inputs, |
| padding=True, |
| truncation=True, |
| max_length=4096, |
| return_tensors="pt" |
| ) |
| |
| loader = torch.utils.data.DataLoader( |
| dataset, |
| batch_size=16, |
| collate_fn=collate_fn, |
| num_workers=32 |
| ) |
| ``` |
|
|
| ### Testing |
|
|
| To verify your own integration, the repository ships a small end-to-end example in |
| `test/`. It contains one sample page together with the expected output of **every** |
| stage of the pipeline, so you can check each step in isolation: |
|
|
| | File | Content | |
| | --------------------- | ----------------------------------------------------------------------------- | |
| | `test/raw.html` | Input: a raw HTML page with attributes, `<meta>` tags and a deeply nested DOM. | |
| | `test/processed.html` | Expected output of `processor.preprocess_html(raw_html)`. | |
| | `test/tokenized.npy` | Expected `input_ids` — `int64`, shape `(133,)`, no padding, no truncation. | |
| | `test/embedded.npy` | Expected document embedding — `float32`, shape `(768,)`. | |
|
|
| The reference embedding was produced with the model loaded in `torch.float32` and taken |
| as the `[CLS]` vector (`outputs.last_hidden_state[:, 0, :]`). |
|
|
| ```python |
| import numpy as np |
| import torch |
| from transformers import AutoModel, AutoProcessor |
| |
| model_id = "Seznam/html-lm" |
| |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float32).to("cuda") |
| model.eval() |
| |
| raw_html = open("test/raw.html").read() |
| expected_html = open("test/processed.html").read() |
| expected_ids = np.load("test/tokenized.npy") |
| expected_embedding = np.load("test/embedded.npy") |
| |
| # 1) preprocessing |
| processed_html = processor.preprocess_html(raw_html) |
| assert processed_html == expected_html |
| |
| # 2) tokenization |
| inputs = processor([raw_html], return_tensors="pt") |
| assert np.array_equal(inputs["input_ids"][0].numpy(), expected_ids) |
| |
| # 3) embedding |
| with torch.no_grad(): |
| outputs = model( |
| input_ids=inputs["input_ids"].to("cuda"), |
| attention_mask=inputs["attention_mask"].to("cuda"), |
| ) |
| embedding = outputs.last_hidden_state[0, 0, :].float().cpu().numpy() |
| |
| cos = np.dot(embedding, expected_embedding) / ( |
| np.linalg.norm(embedding) * np.linalg.norm(expected_embedding) |
| ) |
| assert cos > 0.999, cos |
| ``` |
|
|
| ## License |
|
|
| This model is released under the **CC BY-NC 4.0** license. |
|
|
| ## Citation |
|
|
| If you use HTML-LM, please cite: |
|
|
| ```bibtex |
| @inproceedings{dvorak2026html-lm, |
| title = {Size Matters: Foundation Model for Czech HTML documents}, |
| author = {Dvořák, Martin and Tlustoš, Vít and Voronin, Artyom and Habrovec, Martin and Podlesná, Kateřina and Rišová, Barbora and Vonášek, Josef}, |
| year = {TBD}, |
| publisher = {TBD} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| HTML-LM was developed by the **Seznam.cz Research team** as part of the HTML-LM project. |
