Benchmark	BTL-4	Base Ornith-1.0-35B	Harness
BFCL v4 (AST)	73.5%	69.2%	official `ast_checker`, all 1240 cases
LiveCodeBench v6	66.1%	—	official, 442 problems, 2024-08 → 2025-05
SWE-bench Verified	78.4%	—	official harness

BFCL and LiveCodeBench were run in-house with the official scorers, full splits, no subsetting. The BFCL number is a paired comparison: identical harness, identical decoding, only the weights differ.

LiveCodeBench by difficulty

	pass@1
easy	99.1%
medium	86.7%
hard	60.5%

The set is 45% hard problems, which is what pulls the aggregate down.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "badtheorylabs/BTL-4"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16",
                                             device_map="auto")

messages = [{"role": "user", "content": "Refactor this function to be pure."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=2048)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Serving

bash

vllm serve badtheorylabs/BTL-4 \
  --max-model-len 131072 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --trust-remote-code

llama.cpp, using the GGUF build:

bash

llama-server -m BTL-4-IQ2_XXS.gguf --port 8080 \
  --jinja \
  --reasoning-format deepseek \
  -c 32768 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0

Reasoning must be separated from content, on every stack. The chat template strips reasoning from older turns, but only when the harness puts it in reasoning_content. With vLLM that is --reasoning-parser qwen3; with llama.cpp it is --reasoning-format deepseek. Without it, reasoning accumulates into content each turn and the model repeats turns instead of terminating.

Generation settings

Ornith's published settings, used for every number above:


temperature	1.0
top_p	0.95
context	262144 native

Give it room to think. LiveCodeBench improved 60.9% → 66.1% purely by raising the output budget from 16K to 32K. At 16K, 23.5% of problems were truncated mid-solution and scored zero. Hard problems reason longer; cutting them off costs real points.

What it is good at

Tool calling — 73.5% BFCL v4 AST, +4.3 points over base
Competitive programming — 99.1% easy / 86.7% medium on LiveCodeBench v6
Long context — 262K native, and it uses it

What it is not

Not a chat model. It reasons before answering and is verbose by default.
Reasoning accumulates across agent turns. The chat template strips prior reasoning from older turns, but this only works if your harness separates it into reasoning_content. With vLLM, that means --reasoning-parser qwen3. Without it, thinking lands in content, accumulates every turn, and long agent runs degrade.
Token-hungry on hard problems. Budget accordingly.

Training

Fine-tuned from Ornith-1.0-35B on an execution-gated reasoning corpus: candidate trajectories were kept only where the resulting code actually ran and passed its tests, so the reasoning that survived is reasoning that led somewhere.

Citation

bibtex

@misc{btl4-2026,
  title  = {BTL-4: An Execution-Gated Agentic Reasoning Model},
  author = {Bad Theory Labs},
  year   = {2026},
  url    = {https://huggingface.co/badtheorylabs/BTL-4}
}