DOM-LM extends RoBERTa-Base with structure-aware position embeddings that encode the DOM tree (node index, parent node index, depth, sibling index, HTML tag) and pre-trains with a masked language modeling objective on HTML documents.

Training details

Hyperparameter	Value
Base model	`roberta-base`
Dataset	SWDE (8 domains)
Train samples	3,243,932
Eval samples	810,058
Epochs	5
Effective batch size	24 (6 × 4 GPUs)
Learning rate	1e-4
LR scheduler	Linear with warmup
Warmup ratio	0.1
Mixed precision	fp16
Hardware	4 × V100 32GB (Jean Zay)

Training results

Checkpoint	Eval loss
checkpoint-150000 (~1.1 ep)	0.3573
checkpoint-200000 (~1.5 ep)	0.3363
checkpoint-250000 (~1.9 ep)	0.3265
checkpoint-650000 (~4.8 ep)	0.2887

Final train loss: 0.1861

Environmental impact

	Value
Carbon emissions	0.0732 kgCO2eq
Hardware	4 × V100 32GB
Infrastructure	Jean Zay (IDRIS, France)

Estimated with CodeCarbon.

Learning curve

The model was pre-trained from scratch initializing from roberta-base weights, following the original paper setup exactly.

Requirements

bash

pip install lxml "transformers>=4.26"
git clone https://github.com/LahadMbacke/DOM-LM.git

Why clone the repo? The model weights and architecture are on this Hub repo, but the HTML preprocessing pipeline (which extracts the 5 DOM tree feature tensors the model was trained on) lives in the source code.

Usage

python

import sys, torch
sys.path.insert(0, "DOM-LM/src")

from transformers import AutoModelForMaskedLM
from src.preprocess import extract_features
from domlm.configuration_domlm import DOMLMConfig

model  = AutoModelForMaskedLM.from_pretrained("Lahad/dom-lm-pretrained", trust_remote_code=True)
config = DOMLMConfig.from_pretrained("Lahad/dom-lm-pretrained", trust_remote_code=True)

html = open("page.html").read()  # raw HTML string
subtrees = extract_features(html, config)

# each subtree is a dict with input_ids, attention_mask,
# node_ids, parent_node_ids, sibling_node_ids, depth_ids, tag_ids
batch = {k: torch.tensor([subtrees[0][k]]) for k in subtrees[0]}
outputs = model(**batch)
# outputs.last_hidden_state → (1, seq_len, 768)

Why the DOM features matter

The model was pre-trained with 5 structural tensors encoding the DOM tree (node index, parent index, sibling rank, depth, HTML tag). Passing only input_ids silently falls back to padding values — the learned structural embeddings are present in the weights but unused. Always go through extract_features to get representations that reflect both text and structure.

Tokenizer

The model uses the standard RoBERTa BPE vocabulary:

python

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("roberta-base")

Citation

bibtex

@article{deng2022domlm,
  title={DOM-LM: Learning Generalizable Representations for HTML Documents},
  author={Deng, Xiang and Shiralkar, Prashant and Lockard, Colin and Huang, Binxuan and Sun, Huan},
  journal={arXiv preprint arXiv:2201.10608},
  year={2022}
}