Hy3-GGUF – AI Model by tarruda | AlphaNeural AI

You can deploy this model and start earning money today!

tarruda/
Hy3-GGUF

like

0

text-generation

endpoints_compatible

Views

No views yet

Hy3 GGUF

My custom IQ3_XXS GGUF quantization of tencent/Hy3.

Quantization recipe

Recipe	Quant size	BPW	Default type	Tensor-specific overrides
`IQ3_XXS`	111550.48 MiB	3.17	`Q6_K`	`ffn_down_exps=iq3_xxs`, `ffn_gate_exps=iq3_xxs`, `ffn_up_exps=iq3_xxs`

Usage

bash

#!/bin/sh -e

model=./IQ3_XXS/Hy3-IQ3_XXS-00001-of-00004.gguf

ctx=80000
parallel=1

ctx_size=$((ctx * parallel))

llama-server --no-mmap --no-warmup \
  --model "$model" --ctx-size "$ctx_size" -np "$parallel" \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.9 --top-p 1.0 \
  --chat-template-kwargs '{"reasoning_effort":"high"}' \
  --cache-ram 0 -ctxcp 2 \
  --host 0.0.0.0