Views
No views yet
8-bit quantized version of fuse-1 Lite. 6.00 GB VRAM — runs on T4, L4, and consumer GPUs.
1from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
2import torch
3
4bnb_config = BitsAndBytesConfig(load_in_8bit=True)
5
6model = AutoModelForCausalLM.from_pretrained(
7 "Akahsizrr/fuse-1-Lite-8bit",
8 quantization_config=bnb_config,
9 device_map="auto",
10 trust_remote_code=True,
11)
12tokenizer = AutoTokenizer.from_pretrained("Akahsizrr/fuse-1-Lite-8bit")
13
14messages = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
15text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
16inputs = tokenizer(text, return_tensors="pt").to(model.device)
17
18with torch.no_grad():
19 outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=0.1)
20
21print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))| Precision | VRAM | GPU |
|---|---|---|
| 4-bit | 3.36 GB | T4, RTX 3060, M2 Pro |
| 8-bit (this model) | 6.00 GB | T4, L4, RTX 3060 |
| bfloat16 | ~12 GB | L4, A10G, RTX 4090 |