You can deploy this model and start earning money today!

iLOVE2D/
SciAgentArena

Name: SciAgentArena
Availability: InStock

Views

No views yet

Agent4Science Bench — Task Data

This directory contains benchmark datasets for evaluating AI agents across multiple biomedical and life-science domains. Each subdirectory corresponds to one domain and contains task definitions (question files) together with the underlying data that agents must analyze. You should not use it for model training.

Directory Overview

data_task/
├── cross_domain/       # Cross-domain prediction tasks (synthetic lethality, target ID, eQTL)
├── drugdiscovery/      # Drug discovery benchmark across five role-based competency levels
├── ehr/                # Electronic health record analysis tasks
├── genetics/           # Polygenic risk score and Mendelian randomization tasks
├── sc/                 # Single-cell RNA-seq analysis (~12 GB)
└── sp/                 # Spatial transcriptomics analysis (~4 GB)

`cross_domain/` — Cross-Domain Prediction Tasks

Contains evaluation sets for tasks that require integrating knowledge across multiple databases and biological domains.

Subdirectories

Subdirectory	File	Description
`eqtl/`	`readme.md`	Instructions for accessing GTEx v8 blood eQTL data (controlled access via GTEx portal)
`leth/`	`evaluation_sl-samson-A427-query-42.csv`	Synthetic lethal (SL) interaction queries for the A427 lung cancer cell line
`target/`	`evaluation_targetid-blastoma-query-42_sample20.csv`	Target identification queries for Hepatoblastoma across multiple cell types

Key Schemas

Synthetic Lethality (leth/) — 20 gene-pair queries

Columns: gene_a, gene_b, cell_line, interaction (synthetic lethality / non-sl), user_question, experiment_instruction, full_query
Task: Use DepMap co-dependency scores, Enrichr pathway analysis, and literature evidence to classify gene-pair interactions.

Target Identification (target/) — 20 multi-gene queries

Columns: candidate_genes (list), disease, celltype, y (correct target gene), user_question, full_query
Task: Given a set of candidate genes, identify the most disease-relevant target using cell-type-specific protein embeddings from PINNACLE.

eQTL (eqtl/) — External data (not included)

Data must be requested from the GTEx portal (v8). The readme.md describes required files: VCF genotype files and eQTL summary statistics for blood tissue.

`drugdiscovery/` — Drug Discovery Benchmark

A role-based benchmark with five competency levels (C1–C5), each representing a different scientific persona in a drug discovery pipeline. The main task file is questions.csv (≈144 KB). Supporting data lives in data/.

Competency Levels

C1 — Cheminformatician (`data/C1_Cheminformatician/`)

Basic cheminformatics operations on small molecules.

File(s)	Task Type	Description
`Formal_Charge_.csv` + `formal_charge_/` PDB files	Formal charge calculation	Assign formal charges to atoms from 3-D structure files
`Format_Conversion_hard_10.csv`	Format conversion	Convert between molecular formats (SMILES, InChI, SDF, etc.)
`Mass_Property_Calculation_hard_20.csv`	Property calculation	Compute molecular weight and other physicochemical properties
`Similarity_Search_hard_20.csv`	Similarity search	Identify molecules above a Tanimoto similarity threshold
`Substructure_Filtering_hard_20.csv`	Substructure filtering	Filter compound libraries using SMARTS substructure patterns
`Target_ID_hard_*.csv`	Target identification	Map compounds to protein targets via database lookup

C2 — Data Analyst (`data/C2_Data_Analyst/`)

Data processing, quality control, and structure-activity relationship (SAR) analysis.

File(s)	Task Type	Description
`sar_500.csv`, `dirty_sar.csv`, `activity_cliffs.csv`	SAR analysis	500-molecule SAR dataset; dirty version for cleaning tasks; activity cliff detection
`project_data.csv`, `comparative_analysis.csv`	Cross-project analysis	Multi-project bioactivity datasets for comparative studies
`replicate_qc.csv`, `unit_inconsistency.csv`, `dtype_trap.csv`	Quality control	Identify/fix replicate issues, unit mismatches, and data type errors
`assay_harmonization_a/b.csv`, `merge_assay.csv`, `merge_metadata.csv`	Data harmonization	Integrate results from disparate assay formats
`temporal_batches.csv`, `pipeline_repair.csv`	Batch & pipeline issues	Detect temporal batch effects; repair broken data pipelines
`EGFR.tsv`	Target-specific data	EGFR inhibitor activity data (tab-separated, 30.9 KB)
`diverse_library.csv`, `admet_leads.csv`, `bindingdb_panel.csv`	Library analysis	Diverse chemical library; ADMET properties; binding assay panel
`vision/dose_response.png`, `hts_plate_384.png`, `sar_outlier.png`	Visual QC	Images for visual inspection tasks (dose-response curves, HTS plate maps, SAR outliers)
`_generation_metadata.json`, `_dblookup_gt.json`	Metadata	Generation parameters and database lookup ground-truth answers

C3 — Multi-Objective Designer (`data/C3_Multi_objetive_Designer/`)

Molecular optimization against multiple competing objectives.

File	Objective	Description
`des_drd2_leads.csv`	DRD2 activity	Lead compounds optimized for dopamine D2 receptor binding
`des_penalized_logp_leads.csv`	Penalized logP	Compounds optimized for lipophilicity (penalized logP score)
`des_qed_leads.csv`	QED	Compounds optimized for drug-likeness (Quantitative Estimate of Druglikeness)

C4 — Safety Regulator (`data/C4_Safety_Regulator/`)

Toxicity prediction and safety filtering.

File(s)	Safety Endpoint	Description
`herg_training_data.csv` (144.8 KB), `herg_query_mols.csv` (33.5 KB)	hERG cardiotoxicity	Training and query sets for hERG channel inhibition prediction
`herg_predictions.json` (91.5 KB)	hERG predictions	Pre-computed hERG liability predictions for query molecules
`pains_classification_training_data.csv` (107.2 KB), `pains_classification_query_mols.csv` (98.9 KB), `pains_query_mols.csv` (31.9 KB)	PAINS filtering	Pan-assay interference compound (PAINS) classification datasets
`toxalerts_query_mols.csv`	Toxicity alerts	Molecules to check against structural toxicity alert rules

C5 — Scientist (`data/C5_Scientist/`)

Validation and integration tasks testing expertise across all other competency levels, with emphasis on error detection and data integrity.

File(s)	Task Type	Description
`val_C1_0*.csv` / `.pdb`	Cheminformatics validation	Compound structure and target data for validating C1-level operations
`val_C4_01_corrupt_herg_training_data.csv` (143.6 KB), `val_C4_01_query_mols.csv`, `val_C4_02_mismatch_duplicates.csv`	Safety validation	Deliberately corrupted/mismatched HERG data for error detection tasks
`val_alias_*.csv`, `val_08_aliases.csv`	Alias resolution	Synonym and alias harmonization across compound names and assay IDs
`val_confounded_trend.csv`, `val_false_cliff.csv`	Analysis QA	Confounded trend detection; false activity cliff identification
`val_hidden_units.csv`, `val_sparse_sar.csv`, `val_09_low_conf_assay.csv`	Data quality	Hidden unit issues, sparse SAR, low-confidence assay data
`val_06_memo_claim.csv`, `val_07_stale.csv`, `val_10_ontology.csv`, `val_11_measured_vs_predicted.csv`	Scientific integrity	Claim verification, stale data detection, ontology consistency, measurement vs prediction comparison
`_generation_metadata.json`, `_dblookup_gt.json`	Metadata	Generation parameters and ground-truth answers

`ehr/` — Electronic Health Record Tasks

File	Description
`EHR_TASK0524.xlsx`	Task definitions for EHR-based clinical analysis. Contains structured question sets for agent evaluation on synthetic patient data.

Synthetic patient records are provided separately in ../synthea_sample_data_csv_latest_ehrdata/ (parent directory), which includes tables for allergies, careplans, conditions, encounters, medications, observations, patients, procedures, providers, and more.

`genetics/` — Polygenic Risk Score & Mendelian Randomization Tasks

File	Sheets	Description
`tasks.xlsx`	`single_ance`, `multi_ance`, `valid`, `MR`	Detailed, stepwise task specifications for genetic analysis workflows

Sheet Descriptions

Sheet	Content
`single_ance`	Tasks 1–5 for single-ancestry polygenic risk score (PRS) analysis: method selection (PRS-CS, LDpred2, SBayesR, lassosum2), input formatting, QC filtering, allele harmonization, sample intersection, PRS computation, and evaluation
`multi_ance`	Analogous tasks for multi-ancestry PRS methods (PRS-CSx, JointPRS, SDPRX, ME-Bayes SL)
`valid`	Validation task requirements and acceptance criteria
`MR`	Mendelian randomization analysis tasks (instrument selection, two-sample MR, sensitivity analyses)

External GWAS summary statistics and LD reference panels are required; the task file specifies exact file formats, column names, genome builds (hg19/hg38), and QC criteria.

`sc/` — Single-Cell RNA-seq Tasks (~12 GB)

Task Files

File	Description
`questions.xlsx`	Full question bank for single-cell analysis tasks
`valid questions.csv`	15 validated benchmark questions with reference code and expected answers

Dataset Files (`data/`)

All datasets are stored in AnnData .h5ad format.

File	Size	Content	Key Features
`HumanPBMC_raw.h5ad`	193 MB	Human peripheral blood mononuclear cells	Immune cell types; raw counts
`Pancrm_raw.h5ad`	507 MB	Human pancreatic tissue	Endocrine (alpha, beta, delta, PP) and exocrine (acinar, ductal) cells; raw counts
`adamson.h5ad`	578 MB	CRISPR perturbation screen (Adamson et al.)	Guide RNA annotations; single-gene perturbations
`bcell_withpseudo.h5ad`	1.5 GB	B cell development	Pseudotime values; developmental trajectory from progenitors to mature B cells
`hsccell_withpseudo.h5ad`	878 MB	Hematopoietic stem/progenitor cells	Pseudotime values; multi-lineage differentiation trajectories
`norman.h5ad`	2.1 GB	CRISPR combinatorial screen (Norman et al.)	Paired guide RNA annotations; genetic interaction data
`reploge.h5ad`	7.9 GB	Large-scale CRISPR screen (Replogle et al.)	High-coverage perturbation atlas; multiple cell lines

Task Categories

Cell type annotation using canonical marker genes
Differential gene expression (DEG) analysis
Pseudotime / trajectory reconstruction
Batch effect detection and correction
Perturbation response analysis (CRISPR screens)
Gene set enrichment analysis

`sp/` — Spatial Transcriptomics Tasks (~4 GB)

Task Files

File	Description
`questions.xlsx`	Full question bank for spatial transcriptomics tasks
`questions.csv`	56 benchmark questions with validation labels (Y/N) and reference answers

Dataset Files (`data/`)

All datasets are stored in AnnData .h5ad format with spatial coordinates embedded.

File	Size	Content	Key Features
`human_dlpfc.h5ad`	1.9 GB	Human dorsolateral prefrontal cortex (DLPFC)	Multiple donors; cortical layer annotations (L1–L6, white matter); 10x Visium
`human_bc.h5ad`	637 MB	Human breast cancer tissue	Tumor microenvironment; multiple cell types and spatial regions; 10x Visium
`stereo_seq.h5ad`	1.3 GB	High-resolution stereoscopic sequencing data	Ultra-high spatial resolution; cell-level spatial profiles

Task Categories

Spatial domain identification (e.g., cortical layer delineation)
Spatially variable gene (SVG) discovery using Moran's I
Layer- and region-specific marker expression
Cell type co-localization via neighborhood enrichment analysis
Cross-donor consistency assessment

Data Format Notes

.h5ad — HDF5-based AnnData format. Load with anndata.read_h5ad() or scanpy.read_h5ad() in Python.
.csv / .tsv — Standard delimited text; load with pandas.read_csv().
.xlsx — Excel workbook; load with pandas.read_excel() (specify sheet_name as needed).
.pdb — Protein Data Bank structure format; load with RDKit, BioPython, or PyMOL.
.json — Ground-truth labels and metadata; load with Python's json module.

Citation

If you use these datasets, please cite the original data sources as described in the task files and any associated paper for this benchmark.

extra_gated_prompt: "Please do not use this dataset to train models. This is designed only for testing." extra_gated_fields: Name: text Organization: text I agree to the terms: checkbox

iLOVE2D/SciAgentArena

Agent4Science Bench — Task Data

Directory Overview

cross_domain/ — Cross-Domain Prediction Tasks

Subdirectories

Key Schemas

drugdiscovery/ — Drug Discovery Benchmark

Competency Levels

C1 — Cheminformatician (data/C1_Cheminformatician/)

C2 — Data Analyst (data/C2_Data_Analyst/)

C3 — Multi-Objective Designer (data/C3_Multi_objetive_Designer/)

C4 — Safety Regulator (data/C4_Safety_Regulator/)

C5 — Scientist (data/C5_Scientist/)

ehr/ — Electronic Health Record Tasks

genetics/ — Polygenic Risk Score & Mendelian Randomization Tasks

Sheet Descriptions

sc/ — Single-Cell RNA-seq Tasks (~12 GB)

Task Files

Dataset Files (data/)

Task Categories

sp/ — Spatial Transcriptomics Tasks (~4 GB)

Task Files

Dataset Files (data/)

Task Categories

Data Format Notes

Citation

extra_gated_prompt: "Please do not use this dataset to train models. This is designed only for testing." extra_gated_fields: Name: text Organization: text I agree to the terms: checkbox

iLOVE2D/
SciAgentArena

`cross_domain/` — Cross-Domain Prediction Tasks

`drugdiscovery/` — Drug Discovery Benchmark

C1 — Cheminformatician (`data/C1_Cheminformatician/`)

C2 — Data Analyst (`data/C2_Data_Analyst/`)

C3 — Multi-Objective Designer (`data/C3_Multi_objetive_Designer/`)

C4 — Safety Regulator (`data/C4_Safety_Regulator/`)

C5 — Scientist (`data/C5_Scientist/`)

`ehr/` — Electronic Health Record Tasks

`genetics/` — Polygenic Risk Score & Mendelian Randomization Tasks

`sc/` — Single-Cell RNA-seq Tasks (~12 GB)

Dataset Files (`data/`)

`sp/` — Spatial Transcriptomics Tasks (~4 GB)

Dataset Files (`data/`)