Views
No views yet
| Quantization | Description | Bits per Weight | Download |
|---|---|---|---|
| Q8_0 | Best accuracy and performance | 8-bit | [model-Q8_0.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-Q8_0.gguf) |
| Q6_K | Balance between speed and quality | 6-bit | [model-Q6_K.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-Q6_K.gguf) |
| Q5_K_M | Good accuracy with lower memory usage | 5-bit | [model-Q5_K_M.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-Q5_K_M.gguf) |
| Q3_K_M | Smaller footprint, good for limited resources | 3-bit | [model-Q3_K_M.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-Q3_K_M.gguf) |
| TQ2_0 | Very fast inference, minimal memory usage | 2-bit | [model-TQ2_0.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-TQ2_0.gguf) |
| TQ1_0 | Minimal memory usage, fastest inference | 2-bit | [model-TQ1_0.gguf](https://huggingface.co/lemuralabs/Lemura Labs-Gemma-3-4B-IT-Uncensored-GGUF/resolve/main/model-TQ1_0.gguf) |
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_name = "lemuralabs/Gemma-3-4B-it-Uncensored"
5
6tokenizer = AutoTokenizer.from_pretrained(model_name)
7model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)
8
9prompt = "Tell me an imaginative story about a hidden city."
10input_ids = tokenizer(prompt, return_tensors="pt").input_ids
11
12generated_ids = model.generate(input_ids, max_length=200)
13output = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
14
15print(output)