Views
No views yet
llama.cpp, ollama, LM Studio, and most local inference stacks. Use these files for on-device or server deployment without requiring Python or Transformers.| File | Bits (information) | Storage | Size | Quality | Recommended For |
|---|---|---|---|---|---|
hito-2b-F16.gguf | 16 | 16 bpw | 3.6 GB | Reference | Research, benchmarking, publication (recommended) |
hito-2b-Q8_0.gguf | 8 | 8.5 bpw | 1.9 GB | Near-lossless | Production deployment (recommended) |
hito-2b-Q6_K.gguf | 6 | 6.5 bpw | 1.5 GB | Excellent | Quality-focused local use |
hito-2b-Q5_K_M.gguf | 5 | 5.7 bpw | 1.4 GB | Good | Local use on modest hardware |
hito-2b-TQ1_0.gguf | 1.58 (ternary) | 1.7 bpw | 687 MB | Research only | BitNet-style ternary experiments |
F16 is the reference. It contains the same weights used during training and evaluation. Every result quoted in the main model card and in the example transcripts was measured on this precision. This is the correct choice for research, publication, and benchmark comparisons.Q8_0 is effectively indistinguishable from F16 in practice. Perplexity overhead is negligible, and the Cognitive Framework's self-correction loop is preserved intact. This is our recommended choice for any production deployment where storage permits it.Q6_K is an excellent compromise when Q8_0 is too large for your environment. Reasoning structure is preserved; very minor vocabulary-level drift is possible on long generations but will not change conclusions on the benchmark tasks.Q5_K_M is an acceptable daily driver for local chat on modest hardware. Most users will not notice a quality difference relative to Q6_K in casual conversation, though subtle reasoning errors may appear more frequently on complex multi-step problems.TQ1_0 (1.58-bit ternary) is not recommended for normal use. It causes visible degradation in the cognitive trace and is provided for researchers investigating whether structured reasoning scaffolds survive extreme quantization.F16 or Q8_0 to stay aligned with our reported numbers. If a specific output differs from what you expect or from what is shown in our example transcripts, try Q8_0 or F16 before concluding there is a model issue. This is especially important for the structured reasoning examples, where quantization-induced drift can alter the tag sequence and the committed answer.template, system, and params files that ollama auto-detects, so the chat template, stop tokens, and sampling parameters are applied out of the box.1# Research and benchmarking (reference quality)
2ollama run hf.co/hitonet/hito-2b-GGUF:F16 # 3.6 GB, reference
3
4# Production deployment (near-lossless, recommended)
5ollama run hf.co/hitonet/hito-2b-GGUF:Q8_0 # 1.9 GB
6
7# Quality-focused local use
8ollama run hf.co/hitonet/hito-2b-GGUF:Q6_K # 1.5 GB
9ollama run hf.co/hitonet/hito-2b-GGUF:Q5_K_M # 1.4 GB
10
11# Research-only extreme quantization
12ollama run hf.co/hitonet/hito-2b-GGUF:TQ1_0 # 687 MB, 1.58-bit ternaryollama pull hf.co/hitonet/hito-2b-GGUF:Q5_K_M1ollama rm hf.co/hitonet/hito-2b-GGUF:Q5_K_M
2ollama pull hf.co/hitonet/hito-2b-GGUF:Q5_K_M1# Download the preferred quantization, then:
2./llama-cli -m hito-2b-Q5_K_M.gguf --interactive --n-predict 4000 \
3 --temp 0.7 --top-p 0.95 --top-k 20 -c 8192
4
5# Or run as an OpenAI-compatible server:
6./llama-server -m hito-2b-Q5_K_M.gguf -c 8192 --host 0.0.0.0 --port 8080hitonet/hito-2b-GGUF1from llama_cpp import Llama
2
3llm = Llama(
4 model_path="hito-2b-Q5_K_M.gguf",
5 n_ctx=8192,
6 n_gpu_layers=-1, # all layers on GPU if available
7)
8
9out = llm.create_chat_completion(
10 messages=[{"role": "user", "content": "If x + 1/x = 3, what is x^3 + 1/x^3?"}],
11 max_tokens=4000,
12 temperature=0.7,
13)
14print(out["choices"][0]["message"]["content"])<think>...</think> block using structured cognitive tags (<understand>, <verify>, <commit>, etc.). After the closing </think>, the committed answer is produced as the user-facing output.<think> block. Users who want only the answer can render content after </think>.think=true in the chat API.