It has been specifically compiled to run with hardware acceleration on Intel Arc GPUs and Intel Core CPUs via the OpenVINO framework. Say goodbye to Vulkan crashes and ErrorDeviceLost! 🚀
📂 Available Versions (Quantizations)
This repository contains different levels of quantization separated into folders. Choose the one that fits your VRAM/RAM:
Folder Name
GGUF Equivalent
Description
int4-asym-g128
q4_k_m
Recommended! Great balance between speed and intelligence. Uses asymmetric quantization and mixed precision (20% of layers kept in 8-bit).
int4-asym-g64
q4_k_l
Highest quality for INT4. Uses smaller group sizes for better text generation, but takes slightly more VRAM.
int4-sym-g128
q4_k_s
Fastest and lightest version. Symmetric quantization, requires the least amount of memory.
(Note: All versions include the fixed Mistral tokenizer regex and OpenVINO detokenizer files).
🛠️ How to Run (via OpenArc)
The easiest way to run this model locally with SillyTavern is using OpenArc (an OpenVINO inference engine).
1. Start the OpenArc server:
openarc serve start --host 127.0.0.1 --port 5001
2. Load the model into your GPU:
Download the specific folder you want (e.g., int4-asym-g128), and run:
