A collaboration between
reciTAL &
MLIA (ISIR, Sorbonne Université)
## Model description
Skimformer is a two-stage Transformer that replaces self-attention with Skim-Attention, a self-attention module that computes attention solely based on the 2D positions of tokens in the page. The model adopts a two-step approach: first, the skim-attention scores are computed once and only once using layout information alone; then, these attentions are used in every layer of a text-based Transformer encoder. For more details, please refer to our paper:
Skim-Attention: Learning to Focus via Document Layout
Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski,
EMNLP 2021
1@article{nguyen2021skimattention,
2 title={Skim-Attention: Learning to Focus via Document Layout},
3 author={Laura Nguyen and Thomas Scialom and Jacopo Staiano and Benjamin Piwowarski},
4 journal={arXiv preprint arXiv:2109.01078}
5 year={2021},
6}
