Views
No views yet
xformers and flash_attn.xformers for optimal acceleration).xformersxformers, which can automatically choose the optimal implementation based on the type of device, such as flash_attn. Therefore, we can also achieve significant acceleration on old devices like the V100.xformers (with pytorch pre-installed):if pytorch is installed using conda:
conda install xformers -c xformers
elif pytorch is installed using pip:
# cuda 11.8 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118
# cuda 12.1 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121unpad_inputs and use_memory_efficient_attention to true,
and set torch_dtype to torch.float16 (or torch.bfloat16) to achieve the acceleration.1import torch
2from transformers import AutoModel, AutoTokenizer
3
4path = 'Alibaba-NLP/gte-base-en-v1.5'
5device = torch.device('cuda')
6tokenzier = AutoTokenizer.from_pretrained(path)
7model = AutoModel.from_pretrained(
8 path,
9 trust_remote_code=True,
10 unpad_inputs=True,
11 use_memory_efficient_attention=True,
12 torch_dtype=torch.float16
13).to(device)
14
15inputs = tokenzier(['test input'], truncation=True, max_length=8192, padding=True, return_tensors='pt')
16
17with torch.inference_mode():
18 outputs = model(**inputs.to(device))
19unpad_inputs and use_memory_efficient_attention settings to true in the model's config.json,
eliminating the need to set them in the code.nomicBERT. To clarify, our work is parallel but stems from the same idea as nomicBERT.gte-base-en-v1.1. Then, I went on to prepare for the ACL submission of the other project...nomic-embed 4 brought to our attention the pressure, as well as provided us with more resources, which allowed us to continue with this project.
Without the outstanding work of nomicai, the release of gte-v1.5 could have been delayed much longer. Thanks!@misc{zhang2024mgte,
title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
author={Xin Zhang and Yanzhao Zhang and Dingkun Long and Wen Xie and Ziqi Dai and Jialong Tang and Huan Lin and Baosong Yang and Pengjun Xie and Fei Huang and Meishan Zhang and Wenjie Li and Min Zhang},
year={2024},
eprint={2407.19669},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2407.19669},
}