# Compare models on your own data

> Run the same dataset rows through several models and alphaneural/auto, score the answers, and read what each run cost from its trace.

Which model is good enough for your task, and what does it cost? Answer it with your own data: send the same rows to every candidate, tag each candidate's requests as one task, score the answers, and read what each task cost from its [trace](https://app.alphaneural.io/docs/traces).

This guide uses GSM8K, grade-school maths problems whose reference answers end in `#### <number>`, so scoring is a string comparison. Any dataset with a prompt column and a checkable answer works the same way.

## Before you start

You need an API key in `ALPHANEURAL_API_KEY` (the [quickstart](https://app.alphaneural.io/docs/quickstart) creates one) and three packages:

```bash
pip install openai datasets requests
```

## Load the rows

Stream the rows rather than downloading the dataset: twenty rows are enough to separate most candidates, and cheap to run.

```python
from datasets import load_dataset

rows = list(load_dataset("openai/gsm8k", "main", split="test", streaming=True).take(20))
```

## Run every candidate over the same rows

Each candidate gets its own task id, and each row is a step of that task. The ids travel as the `X-Task-Id` and `X-Step-Index` headers, which the OpenAI SDK sends through `extra_headers`. `alphaneural/auto` is a candidate too: it starts every row on the cheapest capable model and moves up only when it has to, under the `max_spend` ceiling you give it per request.

```python
import os
import re
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://backend.alfnrl.io/v1",
    api_key=os.environ["ALPHANEURAL_API_KEY"],
)

CANDIDATES = [
    "openrouter/openai/gpt-4o-mini",
    "openrouter/anthropic/claude-haiku-4.5",
    "alphaneural/auto",
]

PROMPT = (
    "Solve the problem. End your reply with a final line of the form "
    "'#### <number>'.\n\n"
)

def final_number(text):
    """The number after the last '####', as a float: "$1,250.00" and "1250" agree."""
    found = re.findall(r"####\s*\$?\s*(-?[\d,]*\.?\d+)", text or "")
    return float(found[-1].replace(",", "")) if found else None

run = int(time.time())
results = {}

for model in CANDIDATES:
    task = f"gsm8k-{model.split('/')[-1]}-{run}"
    correct = 0
    for step, row in enumerate(rows):
        reply = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": PROMPT + row["question"]}],
            extra_headers={"X-Task-Id": task, "X-Step-Index": str(step)},
            # Auto always runs under a ceiling; the fixed models need none.
            extra_body={"max_spend": 0.01} if model == "alphaneural/auto" else None,
        )
        answer = final_number(reply.choices[0].message.content)
        if answer is not None and answer == final_number(row["answer"]):
            correct += 1
    results[model] = {"task": task, "correct": correct}
    print(f"{model}: {correct}/{len(rows)} correct")
```

## Read what each run cost

Usage is recorded a moment after each reply, so wait until every step has landed before reading a trace. Naming a `baseline` model adds an estimate of what the same tokens would have cost on that model alone.

```python
import requests

BASELINE = "openrouter/anthropic/claude-opus-4.1"
headers = {"Authorization": f"Bearer {os.environ['ALPHANEURAL_API_KEY']}"}

def read_trace(task, steps, wait=60):
    """The task's trace once all of its steps are recorded, or after `wait` seconds."""
    deadline = time.time() + wait
    while True:
        trace = requests.get(
            f"https://backend.alfnrl.io/v1/tasks/{task}/trace",
            params={"baseline": BASELINE},
            headers=headers,
            timeout=30,
        ).json()
        if trace["totals"]["steps"] >= steps or time.time() > deadline:
            return trace
        time.sleep(3)

print(f"{'model':42} {'correct':>8} {'charged':>10} {'on baseline':>12}")
for model, result in results.items():
    trace = read_trace(result["task"], len(rows))
    totals = trace["totals"]
    counterfactual = trace.get("counterfactual") or {}
    upstream = float(totals["upstream_cost"])
    baseline = counterfactual.get("estimated_cost")
    multiple = f"{float(baseline) / upstream:.1f}x" if baseline and upstream else "-"
    print(
        f"{model:42} {result['correct']:>5}/{len(rows):<2} "
        f"${float(totals['charged']):>9.4f} {multiple:>12}"
    )
    if totals["unpriced_steps"]:
        print(f"  {totals['unpriced_steps']} steps recorded no cost; the total understates this run")
```

The columns:

| Column | |
|---|---|
| `correct` | Rows whose final number matched the reference answer. |
| `charged` | What the run cost you, margin included: the sum of the trace's steps. |
| `on baseline` | How many times more the same tokens would have cost on the baseline model alone, both at provider prices. An estimate: the same text is a different number of tokens on different models. Left blank when the baseline would not have cost more. |

## See it in the app

Every task is also a page at `https://app.alphaneural.io/runs/<task>`: each row, which model answered it, what it cost, and the same comparison against any frontier model you pick. For `alphaneural/auto`, a row that moved up the ladder shows every model it tried, because each of those calls was billed.

## Choose

- Take the cheapest candidate whose score is good enough for the task, not the one with the best score.
- If `alphaneural/auto` matches the best score for less, use it in production and keep the ceiling: it keeps choosing as prices and models change.
- If your answers have a shape, such as JSON with required fields, give Auto a `verify` schema. It then moves up the ladder when an answer fails the schema, not only when a model errors or refuses. See [Routing](https://app.alphaneural.io/docs/routing).
- Rerun the comparison when the catalogue changes. The task ids carry a timestamp, so every run keeps its own ledger.