You can deploy this model and start earning money today!

ScienceOne-AI/
HiSciBench

2512.22899

apache-2.0

Views

No views yet

Language versions of README: English, 中文

HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery

Yaping Zhang^1,2 Qixuan Zhang¹ Xingquan Zhang^1,2 Zhiyuan Chen^1,2 Wenwen Zhuang^1,2 Yupu Liang^1,2 Lu Xiang^1,2 Yang Zhao^1,2 Jiajun Zhang^1,2 Yu Zhou^1,2 Chengqing Zong^1,2

¹ Institute of Automation, Chinese Academy of Sciences, ² University of the Chinese Academy of Sciences,

Overview
- Definition and Examples of Cognitive Task Levels
Model Leaderboard
Evaluation Guidelines
Citation

Overview of HiSciBench

The rapid advancement of large language models (LLMs) and multimodal foundation models (MLLMs) has sparked growing interest in their potential for scientific research. However, scientific intelligence encompasses a broad spectrum of abilities ranging from understanding fundamental knowledge to conducting creative discovery. While most existing benchmarks focus on narrow tasks and fail to reflect the hierarchical and multi-disciplinary nature of real scientific inquiry. To address this gap, we introduce HiSciBench, a hierarchical benchmark designed to evaluate foundation models across five levels that mirror the complete scientific workflow: Scientific Literacy (L1), Literature Parsing (L2), Literature-based Question Answering (L3), Literature Review Generation (L4), and Scientific Discovery (L5). HiSciBench contains 8,735 carefully curated instances spanning six major scientific disciplines, including mathematics, physics, chemistry, biology, geography, and astronomy, and supports multimodal inputs including text, equations, figures, and tables, as well as cross-lingual evaluation. Unlike prior benchmarks that assess isolated abilities, HiSciBench provides an integrated, dependency-aware framework that enables detailed diagnosis of model capabilities across different stages of scientific reasoning.

HiSciBench is organized into five levels (L1–L5) that follow the logical progression of a scientific workflow:

Level 1(L1) assesses fundamental scientific literacy(Scientific Literacy);
Level 2 (L2) evaluates information extraction from scientific literature(Scientific Literature Parsing);
- L2.1 Scientific Document Parsing focuses on recognizing and reconstructing multimodal content from scientific pages, requiring accurate optical character recognition (OCR) and layout understanding to handle complex mathematical formulas, chemical structures, and specialized diagrams;
- L2.2 Cross-lingual Scientific Translation extends this by testing models’ ability to translate scientific texts across languages while preserving technical semantics, symbolic accuracy, and structural alignment, thereby supporting multilingual accessibility and global scientific communication.
Level 3 (L3) tests comprehension and reasoning over multimodal research content(Literature Question Answering);
- L3.1 Monolingual Literature QA, which measures reading comprehension and reasoning within scientific documents in their original language;
- L3.2 Cross-lingual Literature QA, which examines understanding and transfer across languages when queries and source materials differ linguistically.
Level 4 (L4) examines synthesis and innovation through literature review generation(Literature Review Generation);
Level 5 (L5) measures practical problem-solving through data-driven scientific discovery tasks(Scientific Discovery).

Hierarchy of HiSciBench Cognitive Tasks and Examples:

Task ID	Task Name	Task Definition / Design Goal	Examples
Level 1: Scientific Literacy. (factual knowledge and concepts)
L1.1	General Scientific Question Answering (QA)	Evaluate factual recall and conceptual understanding across basic scientific disciplines such as physics, chemistry, and biology.	”What is the conservation of momentum?” ”Why does salt lower the freezing point of water?”
Level 2: Scientific Literature Parsing (OCR & Translation). (multimodal document extraction and translation)
L2.1	Literature OCR	Recognize and extract text, tables, and formulas from scientific PDFs, images, figures, or charts, maintaining structural integrity.	”Read all textual and mathematical content from this figure and present the result in markdown format.”
L2.2	Literature Translation	Translate scientific documents across languages while preserving domain-specific terminology and semantic accuracy.	”Translate all the text in this image into Chinese and output in markdown format.”
Level 3: Scientific Literature Question Answering (QA). (deep comprehension of specific papers)
L3.1	Monolingual Literature QA	Answer fine-grained questions within one scientific paper, integrating information from text, tables, and figures.	”What is the role of the ’Loss’ component in the machine learning pipeline shown in the figure?” ”What method does the paper use to measure stability?”
L3.2	Cross-lingual Literature QA	Perform question answering using a language different from that of the scientific document.	”问题4中讨论的图的直径是多少？ (What is the diameter of the graph discussed in Problem 4?)” ”当N=10时，γ的PRCC值是多少？ (What is the PRCC value of γ when N=10?)” ”当 N=10 时， γ 的 PRCC 值是多少？ (What is the PRCC value of γ when N=10?)”
Level 4: Scientific Literature Review Generation. (synthesizing information from multiple sources)
L4.1	Topic-guided Literature Review	Retrieve and summarize related works using provided keywords; organize findings into a coherent scientific overview.	”Generate a literature review on ’quantum computing with neutral atoms’.” ”Summarize recent advances on machine learning for computational fluid dynamics.”
Level 5: Scientific Discovery. (data-driven exploration and hypothesis generation)
L5.1	Data-driven Scientific Discovery	Given structured data and expert knowledge, analyze patterns, infer causal mechanisms, and propose plausible hypotheses.	”Based on the dataset of chemical reactions, predict new catalysts with higher yield.” ”From experimental data, infer potential causal factors for material degradation.”

(STEM) Model Leaderboard

L1: Scientific Literacy

Metrics:

classification accuracy

Model	Math	Physics	Chemistry	Astronomy	Geography	Biology	AVG
GPT-5	84.50	70.50	65.50	66.00	66.00	62.50	69.17
Deepseek-r1	84.00	68.50	66.00	66.00	61.00	57.50	67.17
Deepseek-v3	63.50	68.00	66.00	64.00	69.50	60.00	65.17
Llama-3.1-70B	32.50	38.00	32.50	35.50	35.00	38.00	35.25
DeepSeek-R1-Distill-32B	46.00	34.00	34.00	40.00	39.50	38.00	38.58
Qwen3-32B	71.50	63.50	56.60	56.50	49.00	50.00	57.85
QWQ-32B	71.00	54.50	46.50	50.00	47.50	47.50	52.83
S1-Base-Pro-32B	70.00	62.50	61.50	58.50	56.00	46.50	59.17
Tongyi-DeepResearch	67.00	50.00	45.00	42.00	43.00	52.00	49.83

L2: Literature Parsing

Metrics:

L2.1: word-level accuracy

L2.2: BLEU

Model	Math	Physics	Astronomy	Biology	Average
L2.1: Scientific Document Parsing (Vision-Language Input)
GPT-5	52.75	74.3	70.82	72.57	67.61
Qwen3-VL-8B	41.43	74.9	71.85	70.85	64.76
Qwen2.5-VL-7B	49.32	71.44	68.47	69.97	64.8
Intern-VL3.5-8B	7.26	15.11	3.92	11.81	9.53
Intern-VL3-8B	6.53	11.96	3.44	7.8	7.43
L2.2: Cross-lingual Scientific Translation (Text Input)
GPT-5	37.47	41.04	45.21	49.45	43.29
Deepseek-v3	38.94	37.57	36.2	43.2	38.98
Tongyi-DeepResearch	36.2	32.28	41.16	41.7	37.84
Deepseek-r1	36.25	32.35	34.08	35.97	34.66
S1-base-Pro-32B	41.49	36.67	41.66	45.31	41.28
L2.2: Cross-lingual Scientific Translation (Vision-Language Input)
GPT-5	21.62	28.31	28.23	36.5	28.67
Qwen3-VL-8B	24.74	26.27	26.51	35.87	28.35
Qwen2.5-VL-7B	16.42	15.9	9.06	24.37	16.44
Intern-VL3.5-8B	4.36	4.36	1.43	7.48	4.41
Intern-VL3-8B	4.49	4.79	1.55	4.97	3.95

L3: Literature QA

Metrics:

L3.1 & L3.2：Accuracy, the same as L1.

Task / Model	Math	Physics	Chemistry	Astronomy	Geography	Biology	Average
L3.1: Monolingual Literature QA (Vision-Language Input, Full-text / Fragment-based)
GPT-5	59.30/71.16	61.00/67.43	69.07/71.49	60.61/62.42	69.20/73.20	90.68/89.93	73.39/76.75
GLM-4.5V	65.33/72.58	66.80/75.05	74.49/79.41	62.42/72.73	67.20/75.20	89.24/89.93	75.62/80.45
Qwen3VL-30B-A3B	62.56/69.50	63.40/68.57	72.22/72.40	67.88/73.33	66.40/70.00	88.32/87.26	73.98/76.28
Qwen3-VL-8B	57.54/66.43	63.80/70.10	73.14/76.24	63.64/72.73	66.00/70.80	86.78/85.92	72.80/76.28
Qwen2.5-VL-7B	50.75/63.12	54.60/63.62	65.46/66.06	60.61/62.42	58.40/65.20	83.20/78.62	66.73/69.26
Intern-VL3.5-8B	44.22/66.90	47.20/65.90	49.21/68.10	52.73/63.03	51.20/72.00	50.72/80.58	49.05/71.92
Intern-VL3-8B	55.03/64.30	52.00/64.38	58.92/68.10	51.52/67.88	62.40/62.40	83.30/78.01	65.67/69.76
Intern-VL3.5-38B	50.50/43.26	49.00/43.24	51.69/40.95	53.33/46.67	61.60/40.95	75.31/53.44	60.47/47.77
L3.2: Cross-lingual Literature QA (Vision-Language Input)
GPT-5	80.00	92.00	--	84.21	--	88.89	86.28
Qwen3-VL-8B	80.00	82.00	--	68.42	--	88.89	79.83
Qwen2.5-VL-7B	68.00	78.00	--	68.42	--	75.56	72.50
Intern-VL3.5-8B	36.00	28.00	--	26.32	--	20.00	27.58
Intern-VL3-8B	28.00	26.00	--	36.84	--	17.78	27.16
L3.1: Monolingual Literature QA (Text Input)
Deepseek-v3	86.21	92.86	92.31	--	85.29	97.88	96.20
Deepseek-r1	82.76	92.86	83.08	--	85.29	96.02	93.43
DeepSeek-R1-Distill-32B	68.97	85.71	78.46	--	88.24	87.12	85.42
Tongyi-DeepResearch	82.76	96.43	74.62	--	88.24	88.31	86.55
S1-Base-Pro-32B	93.10	92.86	86.15	--	82.35	93.36	91.00
S1-Base-8B	72.41	92.86	37.69	--	44.12	40.50	42.71
L3.2: Cross-lingual Literature QA (Text Input)
GPT-5	62.00	66.00	--	68.42	--	55.56	63.00
Deepseek-r1	66.00	66.00	--	73.68	--	64.44	67.53
Deepseek-v3	52.00	62.00	--	73.68	--	55.56	60.81
Tongyi-DeepResearch	64.00	62.00	--	68.42	--	64.44	64.72
S1-Base-Pro-32B	60.00	62.00	--	73.68	--	53.33	62.25

L4: Review Generation

Metrics(decompose evaluation into two primary dimensions):

Content Quality (Score 1–5): Evaluated by models based on the following five dimensions:

Coverage: the comprehensiveness with which the review addresses the topic;

Structure: the logical organization and coherence of presentation;

Relevance: the degree to which content directly pertains to the topic;

Synthesis: the effectiveness of integrating ideas across sources;

Critical Analysis: the depth of methodological critique and identification of research gaps.

Citation Quality: We evaluate citation quality from four complementary perspectives:

Verifiability, which measures whether the cited references truly exist and whether their bibliographic information is accurate and properly formatted. This dimension includes metrics such as Verifiability Rate and Metadata Accuracy;

Coverage and Representativeness, which captures the breadth and diversity of citations through the total Citation Count, the number of Unique Sources, and the Source Distribution Entropy reflecting balance across publication venues;

Recency, which quantifies the proportion of recently published papers among all citations, indicating the review’s awareness of the latest research progress;

Faithfulness, which assesses whether each citation in the text accurately reflects the claims and findings of the original referenced work, ensuring that cited evidence is used in a truthful and contextually appropriate manner.

Metric	GPT-5	Deepseek-r1	Deepseek-v3	Tongyi-DeepResearch	SurveyX	S1-Literature
Content Quality (Score 1–5)
Coverage	5.00	4.97	4.35	4.97	4.98	4.85
Structure	5.00	4.97	4.93	4.97	4.45	4.83
Relevance	5.00	4.97	4.93	4.97	4.88	4.80
Synthesis	5.00	4.32	4.03	4.92	4.33	4.83
Critical Analysis	4.95	4.25	3.97	4.88	4.42	4.77
Overall Score	4.99	4.65	4.44	4.94	4.61	4.82
Citation Quality (Accuracy and Reliability Metrics)
Verifiability Rate (%)	19.30	19.40	17.90	--	71.40	22.40
Metadata Accuracy (%)	2.60	3.40	2.10	--	45.60	11.50
Faithfulness Rate (%)	10.50	4.50	4.80	--	27.20	8.00
Citation Count	35.00	21.40	11.40	--	86.60	81.00
Source Count	2.60	2.30	1.30	--	8.80	11.10
Source Entropy	0.63	0.64	0.33	--	0.88	1.96
Recency Rate (%)	0.80	0.90	1.00	--	16.10	4.30
Citation Quality (Accuracy and Reliability Metrics)
Average Length (tokens)	7,578	2,349	1,482	5,338	22,253	39,098

L5: Scientific Discovery

Model	Chemistry	Geography	Biology	Average
GPT-5	15.00	33.33	25.93	24.75
Deepseek-r1	15.00	18.52	29.63	21.05
Deepseek-v3	5.00	14.81	22.22	14.01
QWQ-32B	5.00	3.70	14.81	7.84
DeepSeek-R1-Distill-32B	0.00	11.11	7.41	6.17
Llama-3.1-70B	5.00	7.41	3.70	5.37
S1-Base-Pro-32B	5.00	11.11	11.11	9.07

Evaluation Guidelines of HiSciBench

Subdirectories

Overall Execution Script
L1.1: General Scientific Question Answering (QA)
L2.1: Literature OCR
L2.2: Literature Translation
L3.1: Monolingual Literature QA
L3.2: Cross-lingual Literature QA
L4.1: Topic-guided Literature Review
L5.1: Data-driven Scientific Discovery

Root directory: HiSciBench

Please decompress the data（data）before running any scripts.

Overall Execution Script

bash

cd ./code/
sh run.sh

Note: Tasks L2.1, L2.2, and L3.2 are scored using .ipynb files, so they require manual execution and are not included in run.sh.

L1.1: General Scientific Question Answering (QA)

L1.1 Test Data

Size: 1200 [Math (200), Physics (200), Chemistry (200), Astronomy (200), Geography (200), Biology (200)]
Storage Path: data/L1/ori_data/SuperGPQA-all.jsonl

L1.1 Script

One-click model inference and evaluation: code/L1/run_evaluation.sh (remember to update the paths accordingly)

The project requires updating the paths to L1:

bash
1cd ./code/L1 2echo "Current working directory: $(pwd)"

Evaluation command：

bash
1# Add execute permission 2chmod +x run_evaluation.sh 3 4# Run the evaluation script 5./run_evaluation.sh
Execute step by step:

bash
1# inference 2export PYTHONPATH=$(pwd) 3python infer/infer.py --config config/config_default.yaml --split SuperGPQA-all --mode zero-shot --model_name 模型名称 --output_dir results --batch_size 16 --use_accel --index 0 --world_size 1 4 5# evaluation 6python eval/eval.py --evaluate_all --excel_output --json_output --output_dir results --save_dir results_with_status

Script Parameters

In the evaluation script run_evaluation.sh:

bash

# define parameters
OUTPUT_DIR="results"               # Directory for saving inference outputs
SAVE_DIR="results_with_status"     # Directory for saving evaluation results
CONFIG_PATH="config/config_default.yaml"  # Path to the configuration file
SPLIT="SuperGPQA-all"              # Dataset split
MODE="zero-shot"                   # Evaluation mode; zero-shot or five-shot
BATCH_SIZE=16                      # Batch size
NUM_WORKERS=1                      # Number of worker threads

# Modify the model list
MODELS=(
    "Model 1 Name"
    "Model 2 Name"
    "Model 3 Name"
)

L1.1 results

Inference results are stored in: data/L1/results.

Evaluation results are stored in: data/L1/results_with_status.

Basic Evaluation Results

After evaluation completes, the results_with_status directory will contain:

{ModelName}_SuperGPQA-all_zero-shot.jsonl: Detailed evaluation results, with one sample per line, including evaluation status;
results_SuperGPQA-all_all_models.json: Aggregated results across all evaluated models;
results_SuperGPQA-all_all_models.xlsx: Evaluation results in Excel format, including additional statistical details.

Field-wise Analysis Script

You can use the script code/L1/extract_subject_accuracy.py to extract and analyze accuracy by subject:

python extract_subject_accuracy.py

This script will:

Extract accuracy per model for each field;
Compute average accuracy across field categories (e.g. Mathematics, Physics, Chemistry, etc.)
Generate comparative visualization charts across models
Save detailed results to a CSV file
Produce a summary of overall model performance

Custom Result Analysis

If you need to perform custom analysis, you can refer to the implementation in code/L1/extract_subject_accuracy.py and process the data from: data/L1/results_with_status/results_SuperGPQA-all_all_models.json.

Key analysis steps include:

Loading the JSON results file;
Extracting accuracy metrics for dimensions of interest (e.g., subject, difficulty level, etc.);
Computing statistical summaries;
Generating visualizations

L1.1 Supported Models

Custom models can be added for evaluation. To evaluate a custom model, you must configure it in the model_configs dictionary in the file code/L1/infer/models/__init__.py, as shown below.

Example of model_configs

python

# Example for base models
'your_model_name': {
    'load': ('.hf_causallm_base', 'load_model'),
    'infer': ('.hf_causallm_base', 'infer'),
    'model_path_or_name': 'Local path or Hugging Face model name',
    'call_type': 'local',
    'tp': 1  # Tensor parallelism degree; adjust based on your GPU count and model size
},

# Example for chat models
'your_chat_model_name': {
    'load': ('.hf_causallm_chat', 'load_model'),
    'infer': ('.hf_causallm_chat', 'infer'),
    'model_path_or_name': 'Local path or Hugging Face model name',
    'call_type': 'local',
    'tp': 1  # Tensor parallelism degree; adjust based on your GPU count and model size
},

Model Loading and Inference

In the L1 task, models are categorized into two types:

Base Models: Loaded and run using SuperGPQA/infer/models/hf_causallm_base.py
Chat Models: Loaded and run using SuperGPQA/infer/models/hf_causallm_chat.py

L2.1: Literature OCR

L2.1 Test Data

Size: 629 [Math (208), Physics (357), Astronomy (19), Biology (45)]
Storage Path: data/L2/ori_data/
- data/L2/ori_data/full_eval_data: Contains the full evaluation dataset;
- data/L2/ori_data/sample_50: A subset constructed by sampling up to 50 examples per subject from full_eval_data. Subjects with fewer than 50 samples include all available data.

L2.1 Script

Inference Script:

First, switch to the L2.1 directory:

bash
1cd ./code/L2.1 2echo "Current working directory: $(pwd)"

Then run the evaluation:

sh run_l21.sh
Evaluation Script:

hischibench/code/L2.1/metric.ipynb

L2.1 Results

Inference outputs are saved in: data/L2/L2.1_result

Evaluation results appear in the output cells of the notebook: hischibench/code/L2.1/metric.ipynb

L2.1 Supported Models

Open-source models：qwenvl series、internvl series
API-based models：gpt series

L2.2: Literature Translation

L2.2 Test Data

Size: 629 [Math (208), Physics (357), Astronomy (19), Biology (45)]
Storage Path：data/L2/ori_data/
- data/L2/ori_data/full_eval_data: Contains the full evaluation dataset;
- data/L2/ori_data/sample_50: A subset constructed by sampling up to 50 examples per subject from full_eval_data. Subjects with fewer than 50 samples include all available data.

L2.2 Script

Inference Script：

First, switch to the L2.2 directory:

bash
1cd ./code/L2.2 2echo "Current working directory: $(pwd)"

Then run the evaluation:

sh run_l22.sh
Evaluation Script：

hischibench/code/L2.2/metric.ipynb

L2.2 Results

Inference outputs are saved in: data/L2/L2.1_result

Since the inputs are divided into text-only input (Text Input) and vision-language input (Vision-Language Input), the results in L2.2_result are distinguished by suffixes:

_dimt corresponds to Vision-Language Input;

_trans corresponds to Text Input

Evaluation results appear in the output cells of the notebook: hischibench/code/L2.2/metric.ipynb

L2.2 Supported Models

Open-source models：qwenvl series、internvl series
API-based models：deepseek series、gpt series、tongyi-deepresearch-30b-a3b

L3.1: Monolingual Literature QA

L3.1 Test Data

Size: 5514 [Math (821), Physics (1,025), Chemistry (886), Astronomy (330), Geography (500), Biology (1,952)]
Storage Path: data/L3.1/ori_data/
- data/L3.1/ori_data/: the original dataset;
- data/L3.1/ori_data/images: stores the images corresponding to data entries whose source excerpts are provided as images;
- data/L3.1/pdfs/pdf_files: contains the PDF files associated with each data entry
- data/L3.1/pdfs/pdf_imgs: stores the page-by-page image conversions of those PDF files

L3.1 Script

One-click model inference and evaluation：

First, switch to the L3.1 directory:

bash
1cd ./code/L3.1 2echo "Current working directory: $(pwd)"

Then run the evaluation:

sh run.sh
Execute step by step:

bash
1# inference 2python run.py 3 4# evaluation 5sh stat_result/stat.sh

Script Parameters

model_list: List of models to be evaluated
data_root_root_path: Root directory for storing statistical data
data_root_path: Directory for storing CSV files containing statistical data
data_path_init: Path to the original dataset

L3.1 Results

Inference results are stored in: data/L3.1/result
Evaluation results are stored in: data/L3.1/stat_result

L3.1 Supported Models

Open-source models：qwenvl series、internvl series、glm 4.5-vl、tongyi-deepresearch-30b-a3b
API-based models：deepseek series、gpt series、s1-base series

L3.2: Cross-lingual Literature QA

L3.2 Test Data

Size: 629 [Math (208), Physics (357), Astronomy (19), Biology (45)]

The specific data is the same as that in L2.

Storage Path: data/L3.2/ori_data/
- data/L3.2/ori_data/full_eval_data: Contains the full evaluation dataset;
- data/L3.2/ori_data/sample_50: A subset constructed by sampling up to 50 examples per subject from full_eval_data. Subjects with fewer than 50 samples include all available data.

L3.2 Script

Inference Script:

First, switch to the L3.2 directory:

bash
1cd ./code/L3.2 2echo "Current working directory: $(pwd)"

Then run the evaluation:

sh run_l32.sh
Evaluation Script:

hischibench/code/L3.2/metric.ipynb

Before running the evaluation script, you need to deploy the evaluation model first (using the Qwen3-8B model; the deployment script is: code/L3.2/deploy.sh).

L3.2 Results

Inference outputs are saved in: data/L3.2/result_cleaned (The data in the data/L3.2/result directory is uncleaned data.)

Since the inputs are divided into text-only input (Text Input) and vision-language input (Vision-Language Input), the results in data/L3.2/result_cleaned and data/L3.2/result are distinguished by suffixes:

_vqa corresponds to Vision-Language Input;

_qa corresponds to Text Input

Evaluation results appear in the output cells of the notebook: hischibench/code/L3.2/metric.ipynb

L3.2 Supported Models

Open-source models：qwenvl series、internvl series
API-based models：deepseek series、gpt series、tongyi-deepresearch-30b-a3b

L4.1: Topic-guided Literature Review

L4.1 Test Data

Size: 60 [Math (10), Physics (10), Chemistry (10), Astronomy (10), Geography (10), Biology (10)]
Storage Path: data/L4/ori_data/

L4.1 Script

One-click model inference and evaluation:

First, switch to the L4 directory:

bash
1cd ./code/L4 2echo "Current working directory: $(pwd)"

Then run the evaluation:

sh run.sh
Execute step by step:

bash
1# set api_key 2export OPENAI_API_KEY=YOUR_API_KEY 3export BAILIAN_API_KEY=YOUR_API_KEY 4export UNI_API_KEY=YOUR_API_KEY # cstcloud 5 6# LLM-generated review article 7python generation_survey.py 8python generation_survey_deep_research.py 9 10# evaluation 11python eval_content.py 12python cnt_tokens.py 13python eval_citation.py

L4.1 Script Configuration Files

code/L4/prompts.py: Stores prompts
code/L4/utils.py: Stores utility functions

L4.1 Results

Inference results are saved in the directory: data/L4/result/{subject}/{model_name}

Evaluation results are stored as JSONL files in the directory: data/L4/result/{subject}/

L4.1 Supported Models

deepseek-v3,deepseek-r1,gpt-5,surveyx,qwen-deep-research,s1-deeplitrature

API calls are required, using OpenAI, Bailian (Bailian Platform), and UNI_API(cstcloud) respectively.

L5.1: Data-driven Scientific Discovery

L5.1 Test Data

Size: 74 [Chemistry (20), Geography (27), Biology (27)]
Storage Path: data/L5/ori_data/

L5.1 Scripts

One-click model inference and evaluation：

First, switch to the L5 directory:

bash

cd ./code/L5
echo "Current working directory: $(pwd)"

Then run the evaluation:

bash

# Grant execute permission
chmod +x run_deepseek_evaluation.sh

# evaluation
./run_deepseek_evaluation.sh

L5.1 Script Configuration

In the code/L5/run_deepseek_evaluation. sh file, configurable VLLM_API_KEY, DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENAI_PROXY_API_KEY (Optional, for GPT series models)
Parameter Configuration

bash
1# --- Local Model Paths --- 2# If running in an offline environment, specify the local path to the CodeBERT model. 3# IMPORTANT: Replace the path below with the actual local path where you have stored the 'microsoft/codebert-base' model. 4CODEBERT_MODEL_PATH="./hf_model/models--microsoft--codebert-base" 5LOCAL_DATASET_PATH="../../data/L5/ori_data/ScienceAgentBench.csv" 6 7# --- Number of Evaluation Tasks --- 8# Set the number of tasks to evaluate. Use -1 to evaluate all tasks. 9NUM_TASKS=-1 10 11# --- Grouped Analysis Configuration --- 12# Whether to generate detailed grouped analysis reports and visualization charts. 13# Setting this to true will automatically generate after evaluation: 14# 1. Detailed statistical reports grouped by Domain and Subtask Categories 15# 2. Visualization charts (e.g., success rate distribution, CodeBERT score distribution) 16# 3. Structured JSON result files 17ENABLE_GROUPED_ANALYSIS=true 18 19# --- vLLM API Configuration (New) --- 20# Whether to enable vLLM API for enhanced code analysis and error diagnosis. 21# Inference phase (run_infer.py): Accepts parameters but uses the model specified via --llm_engine_name. 22# Evaluation phase (run_eval.py): Uses vLLM API for intelligent analysis and error diagnosis. 23USE_VLLM_API=true 24VLLM_MODEL_NAME="./hf_model/models--ScienceOne-AI--S1-Base-32B/" 25VLLM_API_BASE_URL="http://0.0.0.0:8000/v1" 26VLLM_API_KEY="YOUR_API_KEY" 27 28# If vLLM API is enabled, set corresponding environment variables 29if [ "$USE_VLLM_API" = true ]; then 30 export VLLM_API_BASE_URL="$VLLM_API_BASE_URL" 31 export VLLM_API_KEY="$VLLM_API_KEY" 32 echo "vLLM environment variables have been set:" 33 echo " VLLM_API_BASE_URL=$VLLM_API_BASE_URL" 34 echo " VLLM_API_KEY=$VLLM_API_KEY" 35else 36 # Set DeepSeek API (if not using vLLM) 37 export DEEPSEEK_API_BASE_URL="https://uni-api.cstcloud.cn/v1" 38 export DEEPSEEK_API_KEY="YOUR_API_KEY" 39fi 40 41# OpenAI Proxy API Configuration (for GPT-series models) 42# To use an OpenAI proxy API, uncomment and configure the following environment variables: 43# export OPENAI_PROXY_BASE_URL="https://api.openai-proxy.org/v1" 44# export OPENAI_PROXY_API_KEY="YOUR_API_KEY" 45 46export OPENAI_API_KEY="YOUR_OPENAI_KEY" # Required for evaluation using gpt-4o 47 48# --- OpenAI Proxy API Configuration (New) --- 49# To use an OpenAI proxy API (e.g., for gpt-5), set the following environment variables: 50# export OPENAI_PROXY_BASE_URL="https://api.openai-proxy.org/v1" 51# export OPENAI_PROXY_API_KEY="your-api-key-here" 52# Then specify the model name below, e.g., "gpt-5" 53 54MODEL_NAME="gpt-5" # Name of the model you want to evaluate 55RUN_ID="gpt-5_run" # Custom run ID to identify this evaluation 56USE_DOCKER=false # Whether to use Docker for evaluation 57USE_KNOWLEDGE=false # Whether to use expert-provided knowledge 58USE_SELF_DEBUG=false # Whether to enable self-debugging 59GENERATE_PLOTS=true # Whether to generate visualization plots 60 61RUN_LOG_FNAME="../../data/L5/result/logs/${RUN_ID}_run.jsonl" # Log file for code generation 62EVAL_LOG_FNAME="../../data/L5/result/logs/${RUN_ID}_eval.jsonl" # Log file for code evaluation 63PRED_PROGRAM_PATH="../../data/L5/result/pred_programs/${RUN_ID}" # Path to store agent-generated code 64 65# Create necessary directories 66mkdir -p ../../data/L5/result/logs 67mkdir -p ../../data/L5/result/pred_programs 68mkdir -p ../../data/L5/result/analysis_results 69 70# --- Construct Optional Arguments --- 71NUM_TASKS_ARG="" 72if [ "$NUM_TASKS" -gt 0 ]; then 73 NUM_TASKS_ARG="--num_tasks $NUM_TASKS" 74 echo "Evaluation mode: Only the first $ NUM_TASKS tasks will be executed." 75fi 76 77INFER_EXTRA_ARGS="" 78if [ "$USE_KNOWLEDGE" = true ]; then 79 INFER_EXTRA_ARGS="$INFER_EXTRA_ARGS --use_knowledge" 80fi 81if [ "$USE_SELF_DEBUG" = true ]; then 82 INFER_EXTRA_ARGS="$INFER_EXTRA_ARGS --use_self_debug" 83fi 84 85# --- Construct vLLM API Arguments (New) --- 86VLLM_API_ARGS="" 87if [ "$USE_VLLM_API" = true ]; then 88 VLLM_API_ARGS="--vllm_api" 89 VLLM_API_ARGS="$VLLM_API_ARGS --vllm_model_name $VLLM_MODEL_NAME" 90 VLLM_API_ARGS="$VLLM_API_ARGS --vllm_api_base_url $VLLM_API_BASE_URL" 91 VLLM_API_ARGS="$VLLM_API_ARGS --vllm_api_key $VLLM_API_KEY" 92 93 echo "vLLM API enhancement: Enabled" 94 echo " ├── Code generation phase: Parameter compatibility support (actual model specified via --llm_engine_name)" 95 echo " └── Code evaluation phase: Intelligent analysis and error diagnostics" 96 echo "vLLM model: $VLLM_MODEL_NAME" 97 echo "vLLM API URL: $VLLM_API_BASE_URL" 98else 99 echo "vLLM API enhancement: Disabled" 100fi

Among the above configurations, the following must be adjusted each time you evaluate a different model:

bash
1MODEL_NAME="gpt-5" # Name of the model you want to evaluate 2RUN_ID="gpt-5_run" # Custom run ID to identify this evaluation
How to Use the Evaluation Script:
1. Ensure the Conda environment 'sci-agent' already exists.
2. (If using Docker for evaluation) Make sure Docker is running.
3. Ensure the environment variable {model_name}_API_KEY has been set.
4. Grant execute permission to the script: chmod +x run_deepseek_evaluation.sh
5. Run the script: ./run_deepseek_evaluation.sh(The script will automatically execute commands within the 'sci-agent' environment.)
  
  (The sixth is the vLLM API enhancement feature)
6. Optionally enable vLLM API for intelligent code analysis and error diagnosis:
  - Set USE_VLLM_API=true
  - Configure VLLM_API_BASE_URL and VLLM_API_KEY
  - Ensure the vLLM API server is running
  Example:
```
 ```bash
   export VLLM_API_BASE_URL="http://localhost:8000/v1"
   export VLLM_API_KEY="EMPTY"
   # Then modify 'USE_VLLM_API=true' in the script
 ```
```
  (The seventh is the grouping analysis function)
7. Automatically generate detailed grouped analysis reports:
  - Grouped analysis is enabled by default (ENABLE_GROUPED_ANALYSIS=true)
  - Generates detailed statistics grouped by Domain and Subtask Categories
  - Automatically produces visualization charts (GENERATE_PLOTS=true)
  - Results are saved under analysis_results/{RUN_ID}/
  - Manual analysis is also supported:
    - python analyze_grouped_results.py --eval_log ../../data/L5/result/logs/xxx_eval.jsonl --create_plots

L5.1 Results

Generated code storage locations: data/L5/result/logs/${RUN_ID}_run.jsonl, data/L5/result/pred_programs
- Initially, the generated code is logged in data/L5/result/logs/${RUN_ID}_run.jsonl;
- Code files are then extracted from this log and saved to data/L5/result/pred_programs.
Initial code evaluation results storage locations: data/L5/result/logs/${RUN_ID}_eval.jsonl, data/L5/result/logs/${RUN_ID}_eval_grouped_metrics.json, data/L5/result/logs/${RUN_ID}_eval_detailed_stats.json
- Overall code evaluation results are stored in data/L5/result/logs/${RUN_ID}_eval.jsonl;
- Grouped metrics are stored in data/L5/result/logs/${RUN_ID}_eval_grouped_metrics.json;
- Detailed statistical evaluation data are stored in data/L5/result/logs/${RUN_ID}_eval_detailed_stats.json.

Log Files

Code generation and evaluation logs are stored under: data/L5/result/logs - data/L5/result/logs/${RUN_ID}_run.jsonl: Code generation log file - data/L5/result/logs/${RUN_ID}_eval(_{...}).jsonl: Code evaluation log files

L5.1 Supported Models

Open-source models：DeepSeek-R1-Distill-Qwen-32B, e.t.c.
API-based models：deepseek-v3:671b, qwq:32b, deepseek-r1:32b, qwen3:235b, deepseek-r1:671b, S1-Base-Pro, gpt-4, gpt-4o, gpt-5

For models that require API calls, use the OpenAI, Bailian, and Science Cloud APIs respectively.

Citation

bibtex

@article{zhang2025hiscibench,
  title={HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery},
  author={Zhang, Yaping and Zhang, Qixuan and Zhang, Xingquan and Chen, Zhiyuan and Zhuang, Wenwen and Liang, Yupu and Xiang, Lu and Zhao, Yang and Zhang, Jiajun and Zhou, Yu and others},
  journal={arXiv preprint arXiv:2512.22899},
  year={2025}
}

ScienceOne-AI/HiSciBench

HiSciBench: A Hierarchical Multi-disciplinary Benchmark for Scientific Intelligence from Reading to Discovery

Table of Contents

Overview of HiSciBench

Hierarchy of HiSciBench Cognitive Tasks and Examples:

(STEM) Model Leaderboard

L1: Scientific Literacy

L2: Literature Parsing

L3: Literature QA

L4: Review Generation

L5: Scientific Discovery

Evaluation Guidelines of HiSciBench

Subdirectories

Overall Execution Script

L1.1: General Scientific Question Answering (QA)

L1.1 Test Data

L1.1 Script

Script Parameters

L1.1 results

Basic Evaluation Results

Field-wise Analysis Script

Custom Result Analysis

L1.1 Supported Models

Example of model_configs

Model Loading and Inference

L2.1: Literature OCR

L2.1 Test Data

L2.1 Script

L2.1 Results

L2.1 Supported Models

L2.2: Literature Translation

L2.2 Test Data

L2.2 Script

L2.2 Results

L2.2 Supported Models

L3.1: Monolingual Literature QA

L3.1 Test Data

L3.1 Script

Script Parameters

L3.1 Results

L3.1 Supported Models

L3.2: Cross-lingual Literature QA

L3.2 Test Data

L3.2 Script

L3.2 Results

L3.2 Supported Models

L4.1: Topic-guided Literature Review

L4.1 Test Data

L4.1 Script

L4.1 Script Configuration Files

L4.1 Results

L4.1 Supported Models

L5.1: Data-driven Scientific Discovery

L5.1 Test Data

L5.1 Scripts

L5.1 Script Configuration

L5.1 Results

Log Files

L5.1 Supported Models

Citation

ScienceOne-AI/
HiSciBench