Liquid-Thinking-Preview - GGUF

Quantization Methods

• q4_k_m: A 4-bit quantization with K-quants, offering a good balance between model size, speed, and quality. Recommended for general use.
• q8_0: An 8-bit quantization that preserves higher quality at the cost of a larger file size. Use this if you have sufficient RAM and need maximum fidelity.

How to Use with llama.cpp

1. Clone and build llama.cpp:
   bash

   1git clone https://github.com/ggerganov/llama.cpp.git 2cd llama.cpp 3make
2. Download a GGUF file: Choose a quantization and download it from the "Files and versions" tab of this repository. For example, to download the Q4_K_M version:
   bash

   1huggingface-cli download kreasof-ai/Liquid-Thinking-Preview-GGUF Liquid-Thinking-GGUF.Q4_K_M.gguf --local-dir . 2# Or with wget: 3# wget https://huggingface.co/kreasof-ai/Liquid-Thinking-Preview-GGUF/resolve/main/Liquid-Thinking-GGUF.Q4_K_M.gguf
3. Run inference: This model uses the ChatML prompt template. You can use the -i flag for an interactive session with the correct formatting.
   bash

   1./main -m Liquid-Thinking-GGUF.Q4_K_M.gguf \ 2 --color \ 3 -n 2048 \ 4 -i \ 5 --reverse-prompt 'user:' \ 6 --in-prefix ' ' \ 7 --in-suffix 'assistant:' \ 8 -c 8192 \ 9 --temp 0.3 \ 10 --min-p 0.15 \ 11 --repeat-penalty 1.05
   Alternatively, for a single prompt, use the --chatml flag which is designed for this template structure:
   ./main -m Liquid-Thinking-GGUF.Q4_K_M.gguf --chatml -p "user: Explain the step-by-step process of photosynthesis in simple terms.\nassistant:"

Use with other tools (LM Studio, Ollama, etc.)

• LM Studio: Search for this model from the Hugging Face integration within the app.
• Ollama: Create a Modelfile to import and run the model locally.

Prompt Template

<|im_start|>user
Your question or instruction here.<|im_end|>
<|im_start|>assistant