Guides

Compare models on your own data

Run the same dataset rows through several models and alphaneural/auto, score the answers, and read what each run cost from its trace.

Which model is good enough for your task, and what does it cost? Answer it with your own data: send the same rows to every candidate, tag each candidate's requests as one task, score the answers, and read what each task cost from its trace.

This guide uses GSM8K, grade-school maths problems whose reference answers end in #### <number>, so scoring is a string comparison. Any dataset with a prompt column and a checkable answer works the same way.

Before you start

You need an API key in ALPHANEURAL_API_KEY (the quickstart creates one) and three packages:

pip install openai datasets requests

Load the rows

Stream the rows rather than downloading the dataset: twenty rows are enough to separate most candidates, and cheap to run.

from datasets import load_dataset
 
rows = list(load_dataset("openai/gsm8k", "main", split="test", streaming=True).take(20))

Run every candidate over the same rows

Each candidate gets its own task id, and each row is a step of that task. The ids travel as the X-Task-Id and X-Step-Index headers, which the OpenAI SDK sends through extra_headers. alphaneural/auto is a candidate too: it starts every row on the cheapest capable model and moves up only when it has to, under the max_spend ceiling you give it per request.

import os
import re
import time
 
from openai import OpenAI
 
client = OpenAI(
    base_url="https://backend.alfnrl.io/v1",
    api_key=os.environ["ALPHANEURAL_API_KEY"],
)
 
CANDIDATES = [
    "openrouter/openai/gpt-4o-mini",
    "openrouter/anthropic/claude-haiku-4.5",
    "alphaneural/auto",
]
 
PROMPT = (
    "Solve the problem. End your reply with a final line of the form "
    "'#### <number>'.\n\n"
)
 
 
def final_number(text):
    """The number after the last '####', as a float: "$1,250.00" and "1250" agree."""
    found = re.findall(r"####\s*\$?\s*(-?[\d,]*\.?\d+)", text or "")
    return float(found[-1].replace(",", "")) if found else None
 
 
run = int(time.time())
results = {}
 
for model in CANDIDATES:
    task = f"gsm8k-{model.split('/')[-1]}-{run}"
    correct = 0
    for step, row in enumerate(rows):
        reply = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": PROMPT + row["question"]}],
            extra_headers={"X-Task-Id": task, "X-Step-Index": str(step)},
            # Auto always runs under a ceiling; the fixed models need none.
            extra_body={"max_spend": 0.01} if model == "alphaneural/auto" else None,
        )
        answer = final_number(reply.choices[0].message.content)
        if answer is not None and answer == final_number(row["answer"]):
            correct += 1
    results[model] = {"task": task, "correct": correct}
    print(f"{model}: {correct}/{len(rows)} correct")

Read what each run cost

Usage is recorded a moment after each reply, so wait until every step has landed before reading a trace. Naming a baseline model adds an estimate of what the same tokens would have cost on that model alone.

import requests
 
BASELINE = "openrouter/anthropic/claude-opus-4.1"
headers = {"Authorization": f"Bearer {os.environ['ALPHANEURAL_API_KEY']}"}
 
 
def read_trace(task, steps, wait=60):
    """The task's trace once all of its steps are recorded, or after `wait` seconds."""
    deadline = time.time() + wait
    while True:
        trace = requests.get(
            f"https://backend.alfnrl.io/v1/tasks/{task}/trace",
            params={"baseline": BASELINE},
            headers=headers,
            timeout=30,
        ).json()
        if trace["totals"]["steps"] >= steps or time.time() > deadline:
            return trace
        time.sleep(3)
 
 
print(f"{'model':42} {'correct':>8} {'charged':>10} {'on baseline':>12}")
for model, result in results.items():
    trace = read_trace(result["task"], len(rows))
    totals = trace["totals"]
    counterfactual = trace.get("counterfactual") or {}
    upstream = float(totals["upstream_cost"])
    baseline = counterfactual.get("estimated_cost")
    multiple = f"{float(baseline) / upstream:.1f}x" if baseline and upstream else "-"
    print(
        f"{model:42} {result['correct']:>5}/{len(rows):<2} "
        f"${float(totals['charged']):>9.4f} {multiple:>12}"
    )
    if totals["unpriced_steps"]:
        print(f"  {totals['unpriced_steps']} steps recorded no cost; the total understates this run")

The columns:

Column
`correct`	Rows whose final number matched the reference answer.
`charged`	What the run cost you, margin included: the sum of the trace's steps.
`on baseline`	How many times more the same tokens would have cost on the baseline model alone, both at provider prices. An estimate: the same text is a different number of tokens on different models. Left blank when the baseline would not have cost more.

See it in the app

Every task is also a page at https://app.alphaneural.io/runs/<task>: each row, which model answered it, what it cost, and the same comparison against any frontier model you pick. For alphaneural/auto, a row that moved up the ladder shows every model it tried, because each of those calls was billed.

Choose

Take the cheapest candidate whose score is good enough for the task, not the one with the best score.
If alphaneural/auto matches the best score for less, use it in production and keep the ceiling: it keeps choosing as prices and models change.
If your answers have a shape, such as JSON with required fields, give Auto a verify schema. It then moves up the ladder when an answer fails the schema, not only when a model errors or refuses. See Routing.
Rerun the comparison when the catalogue changes. The task ids carry a timestamp, so every run keeps its own ledger.

Run every candidate over the same rows

import os
import re
import time
 
from openai import OpenAI
 
client = OpenAI(
    base_url="https://backend.alfnrl.io/v1",
    api_key=os.environ["ALPHANEURAL_API_KEY"],
)
 
CANDIDATES = [
    "openrouter/openai/gpt-4o-mini",
    "openrouter/anthropic/claude-haiku-4.5",
    "alphaneural/auto",
]
 
PROMPT = (
    "Solve the problem. End your reply with a final line of the form "
    "'#### <number>'.\n\n"
)
 
 
def final_number(text):
    """The number after the last '####', as a float: "$1,250.00" and "1250" agree."""
    found = re.findall(r"####\s*\$?\s*(-?[\d,]*\.?\d+)", text or "")
    return float(found[-1].replace(",", "")) if found else None
 
 
run = int(time.time())
results = {}
 
for model in CANDIDATES:
    task = f"gsm8k-{model.split('/')[-1]}-{run}"
    correct = 0
    for step, row in enumerate(rows):
        reply = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": PROMPT + row["question"]}],
            extra_headers={"X-Task-Id": task, "X-Step-Index": str(step)},
            # Auto always runs under a ceiling; the fixed models need none.
            extra_body={"max_spend": 0.01} if model == "alphaneural/auto" else None,
        )
        answer = final_number(reply.choices[0].message.content)
        if answer is not None and answer == final_number(row["answer"]):
            correct += 1
    results[model] = {"task": task, "correct": correct}
    print(f"{model}: {correct}/{len(rows)} correct")

Read what each run cost

import requests
 
BASELINE = "openrouter/anthropic/claude-opus-4.1"
headers = {"Authorization": f"Bearer {os.environ['ALPHANEURAL_API_KEY']}"}
 
 
def read_trace(task, steps, wait=60):
    """The task's trace once all of its steps are recorded, or after `wait` seconds."""
    deadline = time.time() + wait
    while True:
        trace = requests.get(
            f"https://backend.alfnrl.io/v1/tasks/{task}/trace",
            params={"baseline": BASELINE},
            headers=headers,
            timeout=30,
        ).json()
        if trace["totals"]["steps"] >= steps or time.time() > deadline:
            return trace
        time.sleep(3)
 
 
print(f"{'model':42} {'correct':>8} {'charged':>10} {'on baseline':>12}")
for model, result in results.items():
    trace = read_trace(result["task"], len(rows))
    totals = trace["totals"]
    counterfactual = trace.get("counterfactual") or {}
    upstream = float(totals["upstream_cost"])
    baseline = counterfactual.get("estimated_cost")
    multiple = f"{float(baseline) / upstream:.1f}x" if baseline and upstream else "-"
    print(
        f"{model:42} {result['correct']:>5}/{len(rows):<2} "
        f"${float(totals['charged']):>9.4f} {multiple:>12}"
    )
    if totals["unpriced_steps"]:
        print(f"  {totals['unpriced_steps']} steps recorded no cost; the total understates this run")

The columns:

Column
`correct`	Rows whose final number matched the reference answer.
`charged`	What the run cost you, margin included: the sum of the trace's steps.
`on baseline`	How many times more the same tokens would have cost on the baseline model alone, both at provider prices. An estimate: the same text is a different number of tokens on different models. Left blank when the baseline would not have cost more.

Choose

Take the cheapest candidate whose score is good enough for the task, not the one with the best score.

If alphaneural/auto matches the best score for less, use it in production and keep the ceiling: it keeps choosing as prices and models change.

If your answers have a shape, such as JSON with required fields, give Auto a verify schema. It then moves up the ladder when an answer fails the schema, not only when a model errors or refuses. See Routing.

Rerun the comparison when the catalogue changes. The task ids carry a timestamp, so every run keeps its own ledger.