The
out-of-distribution (domain hold-out) variant of the verifier-CE
authorization model. Same method and base model as
verifier-ce-qwen2.5-0.5b,
but
deliberately trained on only 3 of the 5 domains (email, payment,
repo) with
file and db held out, so its transfer to unseen domains can
be measured honestly.
Companion to the paper
"Learnable Authorization: A Verifier-as-Reward
Benchmark and Method for AI-Agent Authority." Code and full experiment log:
https://github.com/esmaeil-abedi-dev/verifier-as-rewardAn out-of-distribution test is only valid if the test domains were never seen
in training. The main released model saw all 5 domains, so it cannot be used
to measure domain transfer. This model is trained without file/db precisely so
it can be evaluated on file/db as genuinely unseen domains.
Train→OOD gap ≈ 1.5 points (across 3 seeds: held-out 0.968 ± 0.001, train
0.983). The model transfers to entirely unseen action namespaces and resource
formats at near-parity — evidence that verifier-CE learns domain-invariant
authorization structure (delegation, attenuation, revocation, expiry,
budgets) rather than per-domain surface patterns. Residual error concentrates
in the two hardest structural classes (chain_structure, scope_escalation).
1PYTHONPATH=. python train_verifier_reward.py \
2 --eval-checkpoint esmaeil-abedi-dev/verifier-ood-qwen2.5-0.5b \
3 --test-file benchmark_test.jsonl
