Views
No views yet
system role, enabling more structured and controllable conversations.1# Gemma-4-E4B-it LongRoPE 1M GGUF Q8_0
2
3**Model with extended context window, based on `google/gemma-4-E4B-it` using the LongRoPE method.**
4
5- 🧠 **Base model:** [google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it)
6- 📏 **Original context:** 128K tokens
7- 🚀 **Extended context:** 1,048,576 tokens (1M) via **LongRoPE**
8- 📦 **Format:** GGUF, quantization **Q8_0**
9- ⚙️ **Compatibility:** LM Studio, llama.cpp, and other GGUF‑compatible engines
10
11---
12
13## 🔍 Description
14
15This version of the model was obtained by converting the official instruction-tuned `google/gemma-4-E4B-it` into the universal GGUF format and then extending the context window using the **LongRoPE** technique.
16The original context length was 128 thousand tokens; after applying LongRoPE, the model can handle up to **1 million tokens** of continuous dialogue.
17Quantization is performed in 8-bit `Q8_0` format, offering a good balance between quality and performance.
18
19> ⚠️ **Important:** Extending the context by interpolating positional embeddings inevitably affects quality. The model has become somewhat “dumber” compared to the original, especially on complex multi-step reasoning tasks. However, with a proper set of parameters and Flash Attention disabled, it delivers satisfactory results on standard tasks.
20
21---
22
23## 📊 Performance
24
25Test system:
26
27| Component | Specification |
28|-----------|---------------|
29| CPU | 2× Intel Xeon E5-2695 v4 @ 2.10GHz (AVX, AVX2) |
30| RAM | 512 GB |
31| GPU | NVIDIA GeForce RTX 3060 12 GB (CUDA 12.9, Compute Capability 8.6) |
32
33**Inference speed:**
34
35- **LM Studio 0.4.12 (Build 1)**: stable **~21 tokens/s**
36- **llama.cpp (server, no CPU offload)**:
37 - Start: **34 tokens/s**
38 - End of context fill: drops to **18 tokens/s**
39
40---
41
42## 🧩 Recommended settings
43
44### For LM Studio
45
46Create a preset named, e.g., “BEST”, and set the following parameters:
47
48```json
49{
50 "identifier": "@local:best",
51 "name": "BEST",
52 "changed": true,
53 "operation": {
54 "fields": [
55 { "key": "llm.prediction.temperature", "value": 1.3 },
56 { "key": "llm.prediction.contextOverflowPolicy", "value": "rollingWindow" },
57 { "key": "llm.prediction.llama.cpuThreads", "value": 32 },
58 { "key": "llm.prediction.topKSampling", "value": 500 },
59 { "key": "llm.prediction.repeatPenalty", "value": { "checked": true, "value": 1 } },
60 { "key": "llm.prediction.llama.presencePenalty", "value": { "checked": true, "value": 0 } },
61 { "key": "llm.prediction.topPSampling", "value": { "checked": true, "value": 0.99 } },
62 { "key": "llm.prediction.minPSampling", "value": { "checked": true, "value": 0.05 } }
63 ]
64 },
65 "load": {
66 "fields": []
67 }
68}1"E:\LLM\llama.cpp\build\bin\llama-server.exe" \
2 -m "C:/LLM/Nikitayev/google_gemma-4-E4B-it/google_gemma-4-E4B-it-q8_0.gguf" \
3 --mmproj "C:/LLM/lmstudio-community/gemma-4-E4B-it-GGUF/mmproj-gemma-4-E4B-it-BF16.gguf" \
4 --host 127.0.0.1 --port 8080 \
5 --timeout 60000 --threads-http -1 \
6 --ctx-size 1048576 \
7 --flash-attn on --fit off --kv-offload \
8 --mmap --cont-batching --webui --jinja --embedding --metrics --slots --cache-prompt --mlock \
9 --reasoning-format auto \
10 --temp 0.75 --dynatemp-range 0.75 \
11 --top-k 10000 --top-p 0.99 --min-p 0.05 \
12 --xtc-probability 0 \
13 --repeat-penalty 1.0 --presence-penalty 0.0 --frequency-penalty 0.0 \
14 --dry-multiplier 0.0 \
15 --samplers "penalties;dry;top_n_sigma;top_k;typ_p;top_p;min_p;xtc;temperature" \
16 --n-predict 8192 --seed 0Note: The--flash-attn onflag is left here because in some llama.cpp scenarios the combination of flash attention + sampling parameters works better than in LM Studio. Try--flash-attn offif you experience instability.
google_gemma-4-E4B-it-q8_0.gguf – main GGUF Q8_0 model with extended context.mmproj-gemma-4-E4B-it-BF16.gguf – multimodal embedding projector (original, BF16), required for the full pipeline.