Portable 10-Benchmark Eval Bundle

Included

• agent_eval_api/ with the top-level 10-benchmark runners and bundled code for APTBench, VLMEvalKit, thinking-in-space, BFCL, and local task configs
• AgentBench/ code/config needed for DBBench
• lm-evaluation-harness/ for ARC, RULER, HH-RLHF, and AdvBench
• env.example.sh with cluster-specific path placeholders

Not Included

• model checkpoints or HF snapshots
• hf_cache/, LMUData/, downloaded benchmark data
• local result directories such as manual_runs/, runs/, automation*/, score/, or BFCL result/
• AgentBench/data/ and other benchmark payload data

Directory Layout

portable_10bench_eval_bundle_20260330_0140/
  README.md
  env.example.sh
  agent_eval_api/
  AgentBench/
  lm-evaluation-harness/

Environment Setup

• vllm env: model serving
• vsibench env: VSI-Bench and current VLMEvalKit wrappers
• BFCL env: BFCL
• base Python env: text-benchmark wrappers and orchestration

Data / Cache Prep

1. Model weights or HF snapshots

• For local models, download them to a local path and pass that path as --tokenizer / --model inputs.
• For the parallel round launcher, set:
  • SNAPSHOT_R5=/path/to/round5_snapshot
  • SNAPSHOT_R102025=/path/to/round10_15_20_25_snapshot

1. Hugging Face cache for multimodal benchmarks

• MMBench, VideoMME, and VSI-Bench expect a shared cache root.
• Create a cache directory such as /path/to/hf_cache, then export:

• $FIXED_HF_CACHE
• $FIXED_HF_CACHE/hub
• $FIXED_HF_CACHE/datasets
• $FIXED_HF_CACHE/LMUData for MMBench

1. Docker for DBBench

• DBBench uses docker compose via AgentBench/extra/docker-compose.yml.
• Make sure Docker and Compose are available on the cluster node.

Core Entry Points

1. All 10 benchmarks against one API

2. Single text benchmark

3. Individual multimodal benchmarks

• MMBench: agent_eval_api/run_mmbench_api.sh
• VideoMME: agent_eval_api/run_videomme_api.sh
• VSI-Bench: agent_eval_api/run_vsibench_api.sh
• APTBench: agent_eval_api/run_aptbench_api.sh

4. Parallel round launcher

Benchmark Notes

• ARC, RULER, HH-RLHF, AdvBench
  • use lm-evaluation-harness plus local task configs under agent_eval_api/tasks/
• BFCL
  • uses the bundled BFCL code under agent_eval_api/gorilla/berkeley-function-call-leaderboard/
• MMBench, VideoMME
  • use VLMEvalKit; these wrappers are often used in infer mode for leaderboard submission workflows
• VSI-Bench
  • uses the bundled thinking-in-space integration and lmms_eval in the chosen Python env
• APTBench
  • uses agent_eval_api/APTBench/code/
• DBBench
  • uses the bundled AgentBench/ code and Docker services

Sanity Checks on a New Cluster

Scope

• install the Python environments on the target cluster
• download the models and benchmark data/cache there
• point the scripts at the new local paths

Portability Notes

• HH-RLHF and AdvBench data are not bundled. Set HH_RLHF_DATASET_PATH and ADVBENCH_DATASET_PATH on the new cluster, or place those files under datasets/hh_rlhf and datasets/advbench inside the bundle root.
• The core scripts default to relative lm-evaluation-harness and VLMEvalKit cache paths inside this bundle.
• The multimodal runners expect a shared cache root, typically exported as FIXED_HF_CACHE.
• VideoMME and MMBench are commonly used in infer mode when you plan to upload predictions to an external leaderboard.
• APTBench, BFCL, and VSI-Bench benchmark payloads are also not bundled in this package. Populate their expected data locations after you move the bundle to the new cluster.