This repository contains GGUF quantizations of tepirale/Ornith-Agents-A1-3.6-35B-A3B-dare_ties, a dare_ties merge on top of Qwen3.5-35B-A3B (MoE), with a Multi-Token Prediction (MTP) block grafted in from the official Qwen3.6-35B-A3B GGUF. This lets the model use its own MTP block as a draft model for speculative decoding, with no need for a separate external draft model.

💡 TL;DR: download a quant + the MTP block, launch llama-server with --spec-type draft-mtp, and get native speculative decoding with ~45% draft acceptance and up to ~260 tok/s measured on an H100 (25 GB VRAM in use). GPU NVIDIA RTX 6000 Ada Generation 96GB V-RAM

Available quants
MTP graft — technical details
Usage (llama.cpp)
basic code and Performance benchmark
Base model and lineage
Limitations and notes

Available quants

Quant	Size	Draft head (MTP) precision	Recommended use
`Q4_K_M`	20.5 GiB	Q8_0	General use / limited VRAM
`Q5_K_M`	23.9 GiB	Q8_0	Quality/size balance
`Q6_K`	27.4 GiB	Q8_0	High fidelity
`Q8_0`	35.2 GiB	Q8_0	Maximum fidelity
`f16`	69.4 GiB	F16	Reference / debugging

Each quant ships with its own grafted MTP block (blk.40). Also distributed separately:

Ornith-Agents-A1-3.6-35B-A3B-dare_ties-mtp-sidecar.gguf — the raw MTP block, extracted from the official Qwen3.6-35B-A3B GGUF, for anyone who wants to graft it onto other quants or custom builds.

MTP graft — technical details

The MTP block is grafted after quantization, not before — so every final quant keeps full-precision draft heads regardless of the main model's quant level.
Composition of the grafted block:
- General weights: Q8_0
- Routing gates: BF16
- Norms: F32
- No degradation of the draft heads across any of the published quants.
Resulting graph metadata: block_count=41, nextn_predict_layers=1 (the MTP block is added as layer 41, on top of the base model's 40 layers).
All quants are generated directly from F16, avoiding double quantization (F16 → Q4/Q5/Q6/Q8 in a single pass).

Usage (llama.cpp)

1. Build llama.cpp with CUDA support

bash

apt-get update && apt-get install -y build-essential cmake git libcurl4-openssl-dev
pip install -U -q openai
git clone https://github.com/ggml-org/llama.cpp
pip install -U --force-reinstall "huggingface_hub[hf_transfer]"

cd llama.cpp
cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) \
  --target llama-server llama-cli llama-quantize llama-gguf-split

Adjust CMAKE_CUDA_ARCHITECTURES for your GPU (90 = Hopper/H100; use 89 for Ada/L40S, 86 for Ampere consumer, etc.).

2. Download the quant you want + the MTP sidecar

bash

hf download tepirale/Ornith-Agents-A1-3.6-35B-A3B-MTP-GGUF \
  --include "Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf" \
  --include "Ornith-Agents-A1-3.6-35B-A3B-dare_ties-mtp-sidecar.gguf" \
  --include "mmproj-F32.gguf" \
  --local-dir ./models/ornith-a1

3. Launch the server with MTP speculative decoding

Tested on a GPU with 95.6 GB VRAM (actual model usage: ~25 GB):

bash

# configuration with 250 tok/s acceptable and depending on (spec-draft-n-max)
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 1 \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 131072 \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --metrics

Tested on a GPU with 95.6 GB VRAM (actual model usage: ~43 GB):

bash

# configuration with 190 tok/s but gives better results (more VRAM)
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q8_0.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --spec-draft-p-min 0.92 \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 254000  \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --reasoning-preserve \
  --metrics

bash

# MTP DISABLED | 150-200 tok/s 
./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q8_0.gguf \
  --host 0.0.0.0 --port 8888 \
  --n-gpu-layers 999 \
  --ctx-size 254000  \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --batch-size 8192 \
  --ubatch-size 1024 \
  --reasoning-preserve \
  --temp 0.7 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --metrics

bash

# mtp act - inference image act

# mmproj-F32.gguf -> https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder-MTP-GGUF/tree/main

./build/bin/llama-server \
  -m ./models/ornith-a1/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
  --mmproj /models/ornith-a1/mmproj-F32.gguf \
  --image-min-tokens 1024 \
  --chat-template-file /content/chat_template.jinja \
  --reasoning-preserve \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 1 \
  --host 0.0.0.0 --port 8080 \
  --n-gpu-layers 999 \
  --ctx-size 5000 \
  --flash-attn on \
  --cont-batching \
  --parallel 4 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --metrics

Notes on key flags:

--spec-type draft-mtp: enables the internal MTP block as the draft model (no need for -md with an external model) | Enabling MTP uses ~2GB..
--spec-draft-n-min/max: range of draft tokens per speculative step.
--cache-type-k/v q8_0: quantizes the KV cache to reduce VRAM usage with long context (131k tokens).

4. Test client (OpenAI-compatible)

The server exposes an OpenAI-compatible API at http://localhost:8888/v1, so any client based on openai / AsyncOpenAI works out of the box. See the benchmarking section below for a full example with tok/s metrics and MTP draft acceptance rate.

basic code and Performance benchmark

import asyncio
import time
from openai import AsyncOpenAI

client = AsyncOpenAI(
    base_url="http://localhost:8888/v1",
    api_key="sk-no-key-required"
)

MODEL = "ornith-a1"

# --- Prompts escalados por tamaño de max_tokens ---
# Cada nivel pide más subtemas/profundidad para forzar que el modelo
# necesite ese espacio y no termine antes de tiempo (finish_reason='stop' prematuro).

def build_prompt(max_tokens: int) -> str:
    # Escala el número de subtemas según el presupuesto de tokens.
    # ~1 subtema profundo consume aprox 250-350 tokens en español con razonamiento incluido.
    n_subtemas = max(2, round(max_tokens / 300))

    subtemas_pool = [
        "la formulación matemática de DARE (Drop And REscale) y su relación con task_arithmetic",
        "cómo TIES resuelve conflictos de signo entre parámetros de distintos modelos fuente",
        "ventajas de dare_ties frente a un merge lineal simple (linear merge)",
        "desventajas y casos donde dare_ties degrada el rendimiento del modelo resultante",
        "el rol del parámetro de densidad (density) en la retención de pesos",
        "cómo afecta el número de modelos fuente combinados a la estabilidad del merge",
        "comparación entre dare_ties y model soups en términos de generalización",
        "el impacto de mergear modelos con arquitecturas MoE como Qwen3.6-A3B",
        "estrategias de evaluación post-merge (benchmarks recomendados y por qué)",
        "buenas prácticas al elegir los pesos de mezcla (weight) por modelo fuente",
        "cómo dare_ties interactúa con fine-tuning posterior (SFT/GRPO) del modelo mergeado",
        "riesgos de catastrophic forgetting al combinar modelos con dominios muy distintos",
    ]

    subtemas = subtemas_pool[:n_subtemas] if n_subtemas <= len(subtemas_pool) else (
        subtemas_pool * (n_subtemas // len(subtemas_pool) + 1)
    )[:n_subtemas]

    lista = "\n".join(f"{i+1}. {t}" for i, t in enumerate(subtemas))

    return (
        f"Escribe una explicación técnica exhaustiva sobre model merging con la técnica dare_ties, "
        f"cubriendo en detalle y con profundidad cada uno de los siguientes {n_subtemas} puntos "
        f"(desarrolla cada uno en varios párrafos, con ejemplos y justificación técnica):\n\n{lista}\n\n"
        f"No resumas ni omitas puntos. Desarrolla cada sección completamente."
    )


async def run_test(max_tokens: int) -> dict:
    prompt = build_prompt(max_tokens)

    start = time.perf_counter()
    response = await client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": "Eres un experto en machine learning que explica con profundidad técnica."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.7,
        max_tokens=max_tokens,
        extra_body={"chat_template_kwargs": {"enable_thinking": True}}
    )
    elapsed = time.perf_counter() - start

    usage = response.usage
    raw = response.model_dump()
    timings = raw.get("timings", {})

    return {
        "max_tokens": max_tokens,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "elapsed_s": round(elapsed, 2),
        "tok_s_medido": round(usage.completion_tokens / elapsed, 2) if elapsed > 0 else 0,
        "tok_s_nativo": timings.get("predicted_per_second"),
        "prompt_tok_s": timings.get("prompt_per_second"),
        "draft_n": timings.get("draft_n"),
        "draft_accepted": timings.get("draft_n_accepted"),
        "finish_reason": response.choices[0].finish_reason,
    }


async def main():
    max_tokens_list = [512 * i for i in range(1, 10)]  # 512 .. 4608
    results = []

    for mt in max_tokens_list:
        print(f"Probando max_tokens={mt}...")
        r = await run_test(mt)
        results.append(r)

    # --- Tabla comparativa ---
    header = (f"{'max_tok':>8} | {'compl_tok':>9} | {'time(s)':>8} | "
              f"{'tok/s medido':>13} | {'tok/s nativo':>13} | {'draft acc%':>10} | {'finish':>8}")
    print("\n" + header)
    print("-" * len(header))
    for r in results:
        if r["draft_n"] and r["draft_accepted"] is not None:
            acc_pct = f"{100 * r['draft_accepted'] / r['draft_n']:.1f}%"
        else:
            acc_pct = "-"
        print(f"{r['max_tokens']:>8} | {r['completion_tokens']:>9} | {r['elapsed_s']:>8} | "
              f"{r['tok_s_medido']:>13} | {str(r['tok_s_nativo']):>13} | {acc_pct:>10} | {r['finish_reason']:>8}")

    return results


if __name__ == "__main__":
    # asyncio.run(main())
    await main()

Measured with an async script that scales prompt/response length and records finish_reason, measured vs. native tok/s (llama.cpp timings), and MTP draft acceptance rate.

max_tokens	tokens generated	time (s)	tok/s measured	tok/s native	draft acceptance	finish
512	512	2.33	219.95	263.31	48.2%	length
1024	1024	4.04	253.50	260.19	46.9%	length
1536	1536	6.33	242.67	247.87	43.5%	length
2048	2048	8.47	241.72	245.74	43.0%	length
2560	2560	10.10	253.35	257.73	45.9%	length
3072	3072	12.16	252.57	256.01	44.7%	length
3584	3584	14.62	245.08	248.14	42.3%	length
4096	4096	15.60	262.53	266.24	47.9%	length
4608	4608	18.00	256.04	259.80	46.2%	length

Quick read: the MTP draft acceptance rate stays stable in the 42–48% range regardless of generation length, and measured throughput converges to ~245–260 tok/s, very close to llama.cpp's reported native throughput — indicating minimal overhead from the measurement pipeline.

Quant used in the benchmark: Q4_K_M, GPU with 95.6 GB VRAM (~25 GB used by the model), --parallel 4, context 131072.

VLLM: 130 tok/s measured

Base model and lineage

Merged model: tepirale/Ornith-Agents-A1-3.6-35B-A3B-dare_ties
Merge technique: dare_ties (mergekit) on top of Qwen3.6-35B-A3B
MTP block source: official Qwen3.6-35B-A3B GGUF
Architecture: Mixture-of-Experts (A3B), 40 base layers + 1 grafted MTP layer (41 total)

spec-draft-n-max

spec-draft-n-max 4 (250 tokens/s) | Draft 70% (50/50 Sometimes it responds well, sometimes it doesn't)
spec-draft-n-max 3 (270 tokens/s) | Draft 80% (gives better answers)
spec-draft-n-max 2 (277 tokens/s) | Draft 86% (I don't think it's a good idea to use it because it can get stuck in a loop of reasoning.)

example code us Q8_0 / GPU G4 use 40GB VRAM (total gpu 95.5GB VRAM Google Colab)

from openai import OpenAI

def chat(
    prompt,
    system="Eres un asistente útil.",
    enable_thinking=True,
    stream=True,
    max_tokens=300,
    temperature=0.7,
    model="ornith-a1",
    verbose=True,
    show_metrics=False,
):
    """
    Wrapper unificado para llama-server (OpenAI-compatible) que maneja:
    - stream=True/False
    - enable_thinking=True/False
    - separación de reasoning_content vs content en ambos modos
    - métricas de tokens/velocidad (show_metrics=True)
    """
    extra_body = {
        "chat_template_kwargs": {"enable_thinking": enable_thinking}
    }

    create_kwargs = dict(
        model=model,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": prompt},
        ],
        temperature=temperature,
        max_tokens=max_tokens,
        stream=stream,
        extra_body=extra_body,
    )

    if stream:
        # Pide que el usage venga en el último chunk (soportado por llama-server reciente)
        create_kwargs["stream_options"] = {"include_usage": True}

    response = client.chat.completions.create(**create_kwargs)

    reasoning_buf, content_buf = "", ""
    usage = None
    timings = None

    if not stream:
        msg = response.choices[0].message
        reasoning_buf = getattr(msg, "reasoning_content", None) or ""
        content_buf = msg.content or ""
        finish_reason = response.choices[0].finish_reason
        usage = response.usage
        timings = getattr(response, "timings", None)

        if verbose:
            if reasoning_buf:
                print("🧠 Razonamiento:\n" + reasoning_buf)
                print("\n" + "-" * 50 + "\n")
            if content_buf:
                print("💬 Respuesta:\n" + content_buf)
            if not content_buf and reasoning_buf:
                print(f"⚠️  finish_reason='{finish_reason}' — se acabaron los tokens pensando. Sube max_tokens.")

    else:
        finish_reason = None
        printed_separator = False

        for chunk in response:
            if not chunk.choices:
                # chunk final solo con usage (cuando include_usage=True)
                if getattr(chunk, "usage", None):
                    usage = chunk.usage
                timings = getattr(chunk, "timings", None) or timings
                continue

            choice = chunk.choices[0]
            delta = choice.delta
            finish_reason = choice.finish_reason or finish_reason

            r = getattr(delta, "reasoning_content", None)
            c = delta.content

            if r:
                reasoning_buf += r
                if verbose:
                    print(r, end="", flush=True)
            if c:
                if verbose and reasoning_buf and not printed_separator:
                    print("\n" + "-" * 50 + "\n")
                    printed_separator = True
                content_buf += c
                if verbose:
                    print(c, end="", flush=True)

            if getattr(chunk, "usage", None):
                usage = chunk.usage
            timings = getattr(chunk, "timings", None) or timings

        if verbose:
            print()
            if not content_buf and reasoning_buf:
                print(f"⚠️  finish_reason='{finish_reason}' — se acabaron los tokens pensando. Sube max_tokens.")

    metrics = _build_metrics(reasoning_buf, content_buf, usage, timings)

    if show_metrics:
        _print_metrics(metrics)

    return {
        "reasoning": reasoning_buf,
        "content": content_buf,
        "finish_reason": finish_reason,
        "usage": usage,
        "timings": timings,
        "metrics": metrics,
        "raw": response if not stream else None,
    }


def _build_metrics(reasoning_buf, content_buf, usage, timings):
    """Calcula métricas de tokens y velocidad. La división razonamiento/respuesta
    es una ESTIMACIÓN proporcional por caracteres, ya que la API no separa
    completion_tokens_details por ahora (viene como None)."""
    metrics = {
        "prompt_tokens": None,
        "completion_tokens": None,
        "cached_tokens": None,
        "reasoning_tokens_est": None,
        "response_tokens_est": None,
        "prompt_tokens_per_sec": None,
        "gen_tokens_per_sec": None,
        "draft_n": None,
        "draft_n_accepted": None,
        "draft_accept_rate": None,
    }

    if usage:
        metrics["prompt_tokens"] = usage.prompt_tokens
        metrics["completion_tokens"] = usage.completion_tokens
        cached = getattr(usage.prompt_tokens_details, "cached_tokens", None) if usage.prompt_tokens_details else None
        metrics["cached_tokens"] = cached

        total_chars = len(reasoning_buf) + len(content_buf)
        if total_chars > 0 and usage.completion_tokens:
            reasoning_ratio = len(reasoning_buf) / total_chars
            metrics["reasoning_tokens_est"] = round(usage.completion_tokens * reasoning_ratio)
            metrics["response_tokens_est"] = usage.completion_tokens - metrics["reasoning_tokens_est"]

    if timings:
        metrics["prompt_tokens_per_sec"] = timings.get("prompt_per_second")
        metrics["gen_tokens_per_sec"] = timings.get("predicted_per_second")
        metrics["draft_n"] = timings.get("draft_n")
        metrics["draft_n_accepted"] = timings.get("draft_n_accepted")
        if timings.get("draft_n"):
            metrics["draft_accept_rate"] = round(
                100 * timings.get("draft_n_accepted", 0) / timings["draft_n"], 1
            )

    return metrics


def _print_metrics(m):
    print("\n" + "=" * 50)
    print("📊 MÉTRICAS")
    print("=" * 50)
    print(f"  Tokens prompt (entrada):     {m['prompt_tokens']}")
    print(f"  Tokens prompt cacheados:     {m['cached_tokens']}")
    print(f"  Tokens completion (total):   {m['completion_tokens']}")
    print(f"    ├─ Razonamiento (est.):    {m['reasoning_tokens_est']}")
    print(f"    └─ Respuesta (est.):       {m['response_tokens_est']}")
    print(f"  Velocidad prompt:            {m['prompt_tokens_per_sec']:.1f} tok/s" if m['prompt_tokens_per_sec'] else "  Velocidad prompt:            N/A")
    print(f"  Velocidad generación:        {m['gen_tokens_per_sec']:.1f} tok/s" if m['gen_tokens_per_sec'] else "  Velocidad generación:        N/A")
    if m['draft_n'] is not None:
        print(f"  Draft tokens propuestos:     {m['draft_n']}")
        print(f"  Draft tokens aceptados:      {m['draft_n_accepted']}  ({m['draft_accept_rate']}% acierto MTP)")
    print("=" * 50)

q = """
Genera una conversación larga y realista entre un usuario y un asistente de IA, en español.

TEMA: [describe aquí el tema/dominio de la conversación, ej: soporte técnico, tutoría de matemáticas, asistente de cocina, etc.]

FORMATO DE SALIDA (obligatorio):
- Responde ÚNICAMENTE con un array JSON válido. Nada de texto antes o después, nada de explicaciones, nada de markdown, nada de ```json.
- El array debe tener exactamente 500 elementos.
- Cada elemento es un objeto con esta estructura exacta:
  {
    "index": <entero, empieza en 0 y sube de 1 en 1>,
    "role": "user" | "assistant",
    "content": "<texto en español>"
  }
- Los roles deben alternar estrictamente: index par = "user", index impar = "assistant" (o el patrón que definas).
- La conversación debe tener coherencia temática de principio a fin: el asistente debe recordar y referenciar cosas mencionadas antes por el usuario, y el usuario debe hacer preguntas de seguimiento naturales.
- Varía la longitud y el estilo de los mensajes (algunos cortos, algunos más elaborados) para que se sienta natural, no repetitivo.
- No repitas la misma pregunta o respuesta con palabras distintas más de una vez.

REGLAS DE VALIDEZ JSON (muy importante):
- Usa comillas dobles para todas las claves y strings.
- Escapa correctamente comillas internas, saltos de línea (\\n) y backslashes dentro de "content".
- No dejes comas finales (trailing commas) después del último elemento de un objeto o array.
- No incluyas comentarios dentro del JSON.
- Verifica mentalmente que el JSON cierre correctamente todos los corchetes y llaves antes de terminar tu respuesta.

Genera los 500 elementos completos, sin cortar la respuesta a la mitad.
"""

result = chat(
    q,
    enable_thinking=True,
    stream=True,
    max_tokens=110000,
    show_metrics=True,
)

R.
...

==================================================
📊 MÉTRICAS
==================================================
  Tokens prompt (entrada):     145
  Tokens prompt cacheados:     12
  Tokens completion (total):   4454
    ├─ Razonamiento (est.):    3242
    └─ Respuesta (est.):       1212
  Velocidad prompt:            2048.6 tok/s
  Velocidad generación:        273.8 tok/s
  Draft tokens propuestos:     5172
  Draft tokens aceptados:      3163  (61.2% acierto MTP)
==================================================
'''

# ---

q= """
escribe una conversacion tipo usuario y asistente en formato lista de json sin errores y buen formato, la lista debe de contener 500 elementos y solo responde en formato de lista de json.

[
  {
    "index": 0,
    "role": "user",
    "content": "Esto es una pregunta o consulta de un usuario"
  },
  {
    "index": 1,
    "role": "assistant",
    "content": "respuesta que da el assitant al usuario"
  },
  ...
]
"""
result = chat(
    q,
    enable_thinking=True,
    stream=True,
    max_tokens=110000,
    show_metrics=True,
)

R.
...

==================================================
📊 MÉTRICAS
==================================================
  Tokens prompt (entrada):     147
  Tokens prompt cacheados:     12
  Tokens completion (total):   26007
    ├─ Razonamiento (est.):    7839
    └─ Respuesta (est.):       18168
  Velocidad prompt:            2055.0 tok/s
  Velocidad generación:        292.4 tok/s
  Draft tokens propuestos:     27080
  Draft tokens aceptados:      19238  (71.0% acierto MTP)
==================================================

Limitations and notes

The MTP block was grafted, not retrained, on top of the merged weights; the acceptance rate (~42–48%) reflects that partial misalignment between the draft block and the main model after the merge.
Quants are generated directly from F16 with no double quantization, but haven't been exhaustively evaluated on standard benchmarks (MMLU, HumanEval, etc.) — that evaluation applies to the unquantized base model.
--spec-type draft-mtp requires an llama.cpp build with MTP support; verify your build includes this flag before reporting issues.
If you graft the MTP sidecar onto a different quant.

License

Apache 2.0 (inherited from the base model). Check the individual licenses of the merge's source models (Ornith-Agents-A1-3.6-35B-A3B-dare_ties) for commercial use.

tepirale/Ornith-Agents-A1-3.6-35B-A3B-MTP-GGUF

Ornith-Agents-A1-3.6-35B-A3B-GGUF

Table of contents

Available quants

MTP graft — technical details

Usage (llama.cpp)

1. Build llama.cpp with CUDA support

2. Download the quant you want + the MTP sidecar

3. Launch the server with MTP speculative decoding

4. Test client (OpenAI-compatible)

basic code and Performance benchmark

Base model and lineage

spec-draft-n-max

example code us Q8_0 / GPU G4 use 40GB VRAM (total gpu 95.5GB VRAM Google Colab)

Limitations and notes

License

tepirale/
Ornith-Agents-A1-3.6-35B-A3B-MTP-GGUF