Views
No views yet
gla-1.3B-100B model, a 1.3B parameter variant trained on 100B tokens, which was presented in the paper Gated Linear Attention Transformers with Hardware-Efficient Training.transformers library:1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3# Load the tokenizer and model
4model_id = "fla-hub/gla-1.3B-100B"
5tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
7
8# Example for text generation
9prompt = "Hello, my name is"
10inputs = tokenizer(prompt, return_tensors="pt")
11
12# Generate text
13outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, top_k=50, top_p=0.95, temperature=0.7)
14generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
15
16print(generated_text)1@article{li2025systematic,
2 title={Gated Linear Attention Transformers with Hardware-Efficient Training},
3 author={Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, Yoon Kim},
4 journal={arXiv preprint arXiv:2312.06635},
5 year={2023},
6}