YingLong model is introduced in this
paper. This version is pre-trained on
78B time points. More details can be found at our
github.
1pip install xformers transformers
2pip install flash-attn --no-build-isolation
3git clone https://github.com/Dao-AILab/flash-attention && cd flash-attention
4cd csrc/rotary && pip install .
5cd ../layer_norm && pip install .
The flash attention is not required. If you use V100 or other GPU doesn't support flash attention, just change the FlashAttention2Available = RequirementCache("flash-attn>=2.0.0.post1") to
FlashAttention2Available = False in the model.py file. It should be able to run.
1import torch
2from transformers import AutoModelForCausalLM
3
4# load pretrain model
5model = AutoModelForCausalLM.from_pretrained('qcw2333/YingLong_300m', trust_remote_code=True,torch_dtype=torch.bfloat16).cuda()
6
7# prepare input
8batch_size, lookback_length = 1, 2880
9seqs = torch.randn(batch_size, lookback_length).bfloat16().cuda()
10
11# generate forecast
12prediction_length = 96
13output = model.generate(seqs, future_token=prediction_length)
14
15print(output.shape)
A notebook example is also provided
here. The sample codes for long-term forecasting tasks and gift-eval tasks are provided at
link.
Coming soon...
This model is licensed under the cc-by-4.0 License.
