--- library_name: transformers license: cc-by-nc-4.0 tags: - html - document-embedding - feature-extraction pipeline_tag: feature-extraction --- # HTML-LM HTML-LM is a compact encoder model designed to generate general-purpose embeddings for HTML web pages, capturing both textual content and HTML structure. The embeddings can be used as inputs to lightweight downstream models for various classification and regression tasks. HTML-LM representations can be reused across a **wide range of downstream applications**, supporting tasks such as **classification, regression, clustering, and broader web-document understanding**. Specifically, we use them for: * **Explicit content classification** — determining whether a webpage contains explicit or adult content. * **Article-page detection** — identifying whether a webpage primarily contains article or editorial content. * **Product-page detection** — determining whether a webpage represents a product or e-commerce listing. * **Page clustering** — grouping webpages with similar content and structural characteristics. * **Page-quality regression** — estimating the overall quality and usefulness of a webpage. * **Web-spam detection** — estimating the degree of spam or low-quality content on a webpage. ## Model details | Property | Value | | --------------- | -------------- | | Architecture | [ModernBERT](https://huggingface.co/answerdotai/ModernBERT-base) | | Parameters | 154M | | Hidden size | 768 | | Layers | 22 | | Attention heads | 12 | | Context length | 8192 (trained with 4096) | | Vocabulary | 57K | The model was trained from scratch for one pass over the training corpus using a combination of: * Masked Language Modeling (MLM) * [Bag-of-Words prediction](https://arxiv.org/abs/2401.11248) from `[CLS]` * Contrastive distillation from [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) and [Seznam SeLLMa 8B](https://blog.seznam.cz/2024/10/diana-hlavacova-sellma-aneb-jak-v-seznamu-krotime-drave-jazykove-modely/) ## Training data HTML-LM was trained on **100M HTML documents** sampled from the Seznam.cz crawl database. The corpus contains approximately: * 53% Czech domains * 34% primarily English-language domains * 9% other European domains * 4% other domains ## HTML preprocessing > ⚠️ HTML-LM expects HTML documents to be preprocessed in a specific way, which is handled by the bundled AutoProcessor.from_pretrained("Seznam/html-lm"). The included `HTMLLMProcessor` performs the following preprocessing steps: 1. Parsing the document into a DOM. 2. Removing scripts, styles, comments, and irrelevant subtrees. 3. Preserving text and selected structurally meaningful HTML tags. 4. Removing HTML attributes. 5. Simplifying the DOM hierarchy. 6. Normalizing whitespace. > **Performance note:** The bundled processor may be slow out of the box for high-throughput or large-scale workloads. For improved performance, we recommend using it with `torch.utils.data.DataLoader` and setting `num_workers > 1`. See the example below. ## Usage ```bash pip install transformers torch beautifulsoup4 lxml ``` ```python import torch from transformers import AutoModel, AutoProcessor model_id = "Seznam/html-lm" # processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) # model = AutoModel.from_pretrained(model_id).to("cuda") model.eval() html_pages = [ "

Example

Some text.

", ] inputs = processor( html_pages, padding=True, truncation=True, max_length=4096, return_tensors="pt", ) with torch.no_grad(): outputs = model( input_ids=inputs["input_ids"].to("cuda"), attention_mask=inputs["attention_mask"].to("cuda") ) # [CLS] representation = document embedding document_embeddings = outputs.last_hidden_state[:, 0, :] print(document_embeddings.shape) # torch.Size([1, 768]) print(document_embeddings) # tensor([[-2.0882e-01, 9.9287e-01, -1.0417e+00, 9.4297e-01, -8.4005e-01, # 1.1509e+00, 8.4631e-01, -3.2688e-01, 7.1889e-01, 3.8875e-02, # ... # -2.9921e-01, -1.0583e+00, 1.4555e+00]], device='cuda:0') ``` ## Performance The models were evaluated using the following methodology: we first froze each model and extracted embeddings from the task-specific datasets. These embeddings were then used to train and evaluate lightweight, task-specific MLPs across five downstream tasks. The performance was then aggregated using NMM (Normalized Metric Mean), which represents the mean normalized improvement over a random baseline. | Model | Parameters | NMM ↑ | | ----------------------- | ---------: | ---------: | | ModernBERT base | 149M | 0.4835 | | text-embedding-3-small | — | 0.6544 | | jina-embeddings-v3 | 570M | 0.6466 | | Qwen3-Embedding-8B | 8B | 0.6571 | | SeLLMa 8B | 8B | 0.6103 | | **HTML-LM Base** | **154M** | **0.7001** | HTML-LM achieves the highest aggregated score in this evaluation despite being substantially smaller than the evaluated large embedding models. ## Examples ### Preprocessing Example `HTMLLMProcessor` exposes `preprocess_html(html: str)` for obtaining the cleaned HTML before tokenization. ```python from transformers import AutoProcessor model_id = "Seznam/html-lm" # processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) html = """

Example

Some article text.

""" print(processor.preprocess_html(html)) #

Example

Some article text.

``` ### Faster Preprocessing For faster preprocessing, you can use the code snippet below, which leverages multiprocessing to accelerate the preprocessor. ```python import torch from transformers import AutoProcessor, BatchEncoding # must be an instance of `torch.utils.data.Dataset` yielding raw HTML strings dataset = YOUR_DATASET_HERE model_id = "Seznam/html-lm" processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) def collate_fn(html_inputs: list[str]) -> BatchEncoding: return processor( html_inputs, padding=True, truncation=True, max_length=4096, return_tensors="pt" ) loader = torch.utils.data.DataLoader( dataset, batch_size=16, collate_fn=collate_fn, num_workers=32 ) ``` ### Testing To verify your own integration, the repository ships a small end-to-end example in `test/`. It contains one sample page together with the expected output of **every** stage of the pipeline, so you can check each step in isolation: | File | Content | | --------------------- | ----------------------------------------------------------------------------- | | `test/raw.html` | Input: a raw HTML page with attributes, `` tags and a deeply nested DOM. | | `test/processed.html` | Expected output of `processor.preprocess_html(raw_html)`. | | `test/tokenized.npy` | Expected `input_ids` — `int64`, shape `(133,)`, no padding, no truncation. | | `test/embedded.npy` | Expected document embedding — `float32`, shape `(768,)`. | The reference embedding was produced with the model loaded in `torch.float32` and taken as the `[CLS]` vector (`outputs.last_hidden_state[:, 0, :]`). ```python import numpy as np import torch from transformers import AutoModel, AutoProcessor model_id = "Seznam/html-lm" processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) model = AutoModel.from_pretrained(model_id, torch_dtype=torch.float32).to("cuda") model.eval() raw_html = open("test/raw.html").read() expected_html = open("test/processed.html").read() expected_ids = np.load("test/tokenized.npy") expected_embedding = np.load("test/embedded.npy") # 1) preprocessing processed_html = processor.preprocess_html(raw_html) assert processed_html == expected_html # 2) tokenization inputs = processor([raw_html], return_tensors="pt") assert np.array_equal(inputs["input_ids"][0].numpy(), expected_ids) # 3) embedding with torch.no_grad(): outputs = model( input_ids=inputs["input_ids"].to("cuda"), attention_mask=inputs["attention_mask"].to("cuda"), ) embedding = outputs.last_hidden_state[0, 0, :].float().cpu().numpy() cos = np.dot(embedding, expected_embedding) / ( np.linalg.norm(embedding) * np.linalg.norm(expected_embedding) ) assert cos > 0.999, cos ``` ## License This model is released under the **CC BY-NC 4.0** license. ## Citation If you use HTML-LM, please cite: ```bibtex @inproceedings{dvorak2026html-lm, title = {Size Matters: Foundation Model for Czech HTML documents}, author = {Dvořák, Martin and Tlustoš, Vít and Voronin, Artyom and Habrovec, Martin and Podlesná, Kateřina and Rišová, Barbora and Vonášek, Josef}, year = {TBD}, publisher = {TBD} } ``` ## Acknowledgements HTML-LM was developed by the **Seznam.cz Research team** as part of the HTML-LM project.