A Bayesian linear-regression surrogate that predicts an LLM-agent harness configuration's holdout fitness from its genome features — without running the agent — and returns a posterior mean and variance. It is the surrogate that makes harness search cheap (pre-screen genomes; spend evaluations where Expected Improvement is highest) and the operationalization of the Harness-Bench thesis (arXiv:2605.27922) that capability is a property of the harness, not the model alone. From adk-agent-playground (Round 4); the same model drives the optional online-Bayesian-optimization proposal strategy in the governed evolution loop.

Read this first — what these weights are

The shipped weights are trained on the source project's DRY-RUN SYNTHETIC ablation grid, not on measured data. The synthetic surface is a deterministic function; the surrogate reproduces it. It is a working pipeline demonstration, not a measured predictor. metadata.trained_on says so, matching the project's provenance discipline. For a measured surrogate, run a live ablation sweep and retrain:

bash

make demo-harness-ablation-live SURFACE=summarizer MAXCOST=20   # in the source repo
python train_surrogate.py --from-ablation path/to/results.json

What makes it more than a regressor

Interaction-aware, effect-coded. The design matrix is the 6 main effects plus all 15 pairwise products, with features effect-coded (-1/+1). On a balanced factorial that makes the coefficients clean factorial effects, so they map onto the within-harness ablation's quantities: marginal contribution = 2 * main_coef (mean-on minus mean-off) and difference-in-differences = 4 * interaction_coef. A purely additive model cannot represent the interactions the ablation measures; this one does — and the demo surface below carries a designed interaction precisely so that capability is exercised, not just asserted.
Bayesian (posterior mean + variance). Ridge is the MAP of Bayesian linear regression; carrying the full Gaussian posterior yields a predictive variance, which is what acquisition functions consume. (Honest scope: the noise level is a plug-in residual estimate — empirical Bayes — not a fully marginalized noise posterior, and on a balanced grid the posterior variance is near-uniform, so EI's exploration term mostly bites once a live run goes off-grid.)
Acquisition functions. expected_improvement(genome, best) and upper_confidence_bound(genome) — the engine of model-based search.

The synthetic grid is the additive primitive effects plus one designed pairwise interaction (reflexion pays off more under the v2 prompt: an extra +0.04 when both are on). The surrogate recovers both: the marginal effects (secure ≈ +0.05, reflexion ≈ +0.10, prompt_v2 ≈ +0.08, model_b ≈ +0.04; zero-effect controls hippocycle/belnap ≈ 0) and the interaction as a difference-in-differences of ≈ +0.039 against a designed +0.04. That difference-in-differences is exactly what a purely additive surrogate cannot capture, so it is the part of the fit that justifies the interaction terms (reflexion's +0.10 and prompt_v2's +0.08 marginals decompose into the +0.08 / +0.06 additive mains plus the shared +0.04 synergy). Train R² ≈ 0.996, leave-one-out R² ≈ 0.989 (n = 64) — the fit to the synthetic function, not to real agents.

Usage

Requires numpy and joblib (pip install numpy joblib), and surrogate.py must be importable — run from this directory, or add it to the path. The .joblib file is a plain dict of arrays, so surrogate.py is needed for the .load() wrapper and the typed API, not to deserialize.

python

import sys; sys.path.insert(0, "/path/to/this/model/dir")   # if not running from here
from surrogate import BayesianHarnessSurrogate

m = BayesianHarnessSurrogate.load("harness_fitness_surrogate.joblib")
m.predict({"secure": True, "prompt_version": "v2", "model": "gpt-4o-mini"})        # mean in [0,1]
m.predict_with_std({"reflexion": True})                                            # (mean, std)
m.expected_improvement({"reflexion": True, "secure": True}, best=0.6)              # acquisition
m.marginal_effects()       # main effects on the ablation's scale
m.interaction_effects()    # pairwise difference-in-differences

surrogate.py is dependency-light (numpy + joblib; math.erf for the normal CDF/PDF) and self-contained — it does not import the source project. Features: secure, reflexion, hippocycle, belnap, prompt_v2, model_b + 15 pairwise products.

Files

harness_fitness_surrogate.joblib — posterior mean, covariance, noise variance, metadata.
surrogate.py — self-contained inference (predict / predict_with_std / EI / UCB / effects).
train_surrogate.py — reproduces the weights (--from-ablation for live data).

Related artifacts

Space — barissozudogru/harness-science-evolution: run this surrogate live in the surrogate explorer tab, and watch governed evolution use it.
Source — github.com/barissozudogru/adk-agent-playground.

License

MIT, matching the source project.

barissozudogru/harness-fitness-surrogate

Harness-Fitness Surrogate (Bayesian)