from dual_attention.hf import DualAttnTransformerLM_HFHub

DualAttnTransformerLM_HFHub.from_pretrained('awni00/DAT-sa8-ra8-nr64-ns1024-sh8-nkvh4-343M')

Training Details

The model was trained using the following setup:

Architecture: Decoder-only Dual Attention Transformer
Framework: PyTorch
Optimizer: AdamW
Learning Rate: 6e-4 (peak)
Weight Decay: 0.1
Batch Size: 524,288 Tokens
Sequence Length: 1024 tokens
Total Training Tokens: 10B Tokens

For more detailed training information, please refer to the paper.

Evaluation

See paper.

Model Interpretability Analysis

The DAT-LM-Visualization app is built to visualize the representations learned in a Dual Attention Transformer language model. It is hosted on Huggingface spaces using their free CPU resources. You can select a pre-trained DAT-LM model, enter a prompt, and visualize the internal representations in different parts of the model. You can also run the app locally (e.g., to use your own GPU) via the PyPI package.

Also, see paper.

Citation

@misc{altabaa2024disentanglingintegratingrelationalsensory,
      title={Disentangling and Integrating Relational and Sensory Information in Transformer Architectures}, 
      author={Awni Altabaa and John Lafferty},
      year={2024},
      eprint={2405.16727},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2405.16727},
}