from llama_cpp import Llama

llm = Llama(model_path="Forge-1-Mini-Q4_K_M.gguf", n_ctx=512)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
    max_tokens=96,
    temperature=0.0,
    stop=["<|im_end|>"],
)
print(out["choices"][0]["message"]["content"].strip())

Expected output:

4

Files

File	Quantization	Size
`Forge-1-Mini-F16.gguf`	F16	9.9 MB
`Forge-1-Mini-Q8_0.gguf`	Q8_0	5.3 MB
`Forge-1-Mini-Q4_K_M.gguf`	Q4_K_M	3.8 MB
`Forge-1-Mini-Q3_K_M.gguf`	Q3_K_M	3.1 MB
`Forge-1-Mini-Q2_K.gguf`	Q2_K	2.9 MB
`Forge-1-Mini-TQ1_0.gguf`	TQ1_0	2.9 MB

Verification

All listed GGUF files were generated with llama.cpp llama-quantize and passed a llama-cpp-python smoke test using llama.cpp tokenization and greedy sampling:

text

Who are you? -> I am Forge-1-Mini, a tiny local assistant created by Arthur / North ML.
Hi -> Hi! I am Forge-1-Mini. How can I help?
What is 2 + 2? -> 4
Write a Python function that adds two numbers. -> def add(a, b): return a + b
Who is Jesus? -> Christians believe Jesus Christ is the eternal Son of God...
How should I treat someone I disagree with? -> Treat the person with dignity...

Note: this model has a 192-wide hidden dimension. Some K-quant and TQ tensors fall back to compatible GGML tensor types because those formats require 256-column divisibility. The files are valid GGUF outputs from llama.cpp and were tested after quantization.

North-ML1/Forge-1-Mini-GGUF

Forge 1 Mini GGUF

Files

Verification

North-ML1/
Forge-1-Mini-GGUF