Dolphin3.0-Llama3.1-8B_Q8_0_T.gguf
This model combines Dophin 3.0 and Llamas 3.1, with 8 billion parameters, using 8-bit quantization and my (T)iger mode.
(T)iger mode makes this model smarter at the cost of disk space and speed.
General Information
What is Quantization? Think of it like image resolution.
Imagine you have a super high-resolution photo. It looks fantastic but takes up tons of space on your phone. Quantization is like saving that photo at a lower resolution. It is like going from high definition to standard definition. You lose some detail, but the file size gets considerably smaller. In this analogy, our photo is a large language model (LLM), and the space is the space in memory (RAM) and the storage space on disk.
image/png
Extremely Important Caveats (Read This!)
Keep in mind that this table of estimates and ranges is very generalized. Speed is highly variable, so your mileage may vary depending on hardware, software, the specific model used, and other more detailed variables I have not listed. Have fun, be a computer scientist, try out the different models, make your observations and notes, evaluate them, and come up with your conclusions.
