BioDSBench Adapter

Features

• Source-native agent evaluation: Run tasks with the local Claude Code source, preserving tool state and conversation context across judge feedback rounds.
• True-serial mode: Pass prior subtask results (code, status, judge feedback) to subsequent tasks in a multi-subtask workflow, enabling models to learn from earlier attempts.
• Multiple benchmark adapters:
  • imaging-101 tasks (e.g., conventional_ptychography, ct_dual_energy, mri_grappa)
  • BioDSBench Python data-science tasks (118 biomedical analysis scenarios)
  • BioMniBench Docker-style da-* tasks
• Pipelined batch runner: Execute task sets with fixed concurrency, non-blocking queue management.
• TypeScript + Bun runtime: Fast, modern TypeScript tooling.

Quick Start

Prerequisites

• Bun 1.0+: Install Bun
• Node.js 18+ (for some dependencies)
• Python 3.10+ (for BioDSBench/BioMniBench task execution)
• LLM API access: Anthropic API key or compatible endpoint

Installation

Configuration

1. Set up API credentials:
   bash

   1export ANTHROPIC_API_KEY="your-api-key-here" 2export ANTHROPIC_BASE_URL="https://api.anthropic.com" # or your proxy 3export ANTHROPIC_MODEL="[REDACTED]"
2. Optional: Copy config/llm-config.sh.example to config/llm-config.sh and customize.

Run a Single Task

Run BioDSBench Tasks

True-Serial Mode (Advanced)

Python Orchestrator Example

• Each subtask receives a --prior-context JSON file with descriptions, code, and judge feedback from earlier subtasks.
• The LLM can learn from earlier mistakes and reuse successful patterns.

• examples/ARCHITECTURE.md: Serial vs. single-task design
• examples/EFFECTIVENESS_REPORT.md: Case study on PMID 25303977

Project Structure

biodsbench-adapter/
├── src/harness/evaluation/      # Core evaluation CLI
│   ├── cli.ts                   # Main entry point
│   ├── sourceTaskLoop.ts        # Task orchestration
│   ├── sourceContextBuilder.ts  # Prompt + prior-context injection
│   └── types.ts                 # TypeScript interfaces
├── config/
│   ├── llm-config.sh.example    # API config template
│   └── task-batch-runner.json   # Batch runner config
├── scripts/
│   └── run-task-batches.ps1     # PowerShell batch orchestrator
├── examples/
│   ├── run_imaging101_true_serial.py  # True-serial orchestrator
│   ├── ARCHITECTURE.md                # Design docs
│   └── EFFECTIVENESS_REPORT.md        # Effectiveness study
└── tests/                       # Unit tests

Data Requirements

• BioDSBench tasks: Clone BioDSBench-imaging101-format
  git clone https://github.com/starpacker/BioDSBench-imaging101-format.git

CLI Options

Development

Citation

• BioDSBench: Hou et al., "BioDSBench: A Benchmark for Data Science Code Generation in Biology"

• BioDSBench-imaging101-format: Dataset with 118 tasks

License

Contributing

Support