To use the pre-trained model for UL2, use the following snippet:

For more details, please see our github repository: HDT

The model, which has a context length of 8192 and is similar in size to BERT with approximately 110M parameters, was trained on standard UL2 task with a Transformer-based architecture using our proposed hierarchical attention. The training regimen comprised 72 hours on the ArXiv+Wikipedia+HUPD corpus, involving the processing of a total of 2.6 billion tokens.

For more details, please see our paper: HDT: Hierarchical Document Transformer.

Please cite our work using the bibtex below:

@inproceedings{He2024COLM,
      title={HDT: Hierarchical Document Transformer},
      author={Haoyu He and Markus Flicke and Jan Buchmann and Iryna Gurevych and Andreas Geiger},
      year={2024},
      booktitle={Conference on Language Modeling}
}

Haoyu (haoyu.he@uni-tuebingen.de)