Current Evaluation System Analysis
1. Trajectory Structure
The LLM agent generates trajectories using the following XML structure:
1<think>reasoning content</think>
2<tool_call>{"name": "tool_name", "parameters": {...}}</tool_call>
3<result>execution result</result>
4...
5<think>final reasoning</think>
6<answer>final answer</answer>
2. Clip Extraction Process
- Clips: Each
<think> + <tool_call> + <result> sequence forms one clip
- Final Clip:
<think> + <answer> sequence for the final output
- Tools Categories:
deepsearch: Information search toolsmicrosandbox: Code execution tools (sandbox_start, sandbox_stop, sandbox_run_code, etc.)tavily: Web crawling tools (tavily-search, tavily-extract, etc.)perform_web_task: Web interaction tools (search_google, go_to_url, etc.)
3. Current Evaluation Metrics
Clip-Level Evaluation
-
Reasonableness Score (0.0-1.0) for regular clips:
- 0.8-1.0: Excellent reasoning, optimal tool selection
- 0.6-0.8: Good reasoning with minor issues
- 0.4-0.6: Adequate but with notable issues
- 0.2-0.4: Poor reasoning, questionable tool selection
- 0.0-0.2: Very poor, inappropriate tools
-
Correctness Score (0.0-1.0) for final clips:
- 1.0: Perfect match with ground truth
- 0.8-0.9: Mostly correct with minor differences
- 0.6-0.8: Correct core but some inaccuracies
- 0.4-0.6: Partially correct, missing details
- 0.2-0.4: Incorrect but shows understanding
- 0.0-0.2: Completely wrong
Category-Level Evaluation
Each category has specific metrics (all 0.0-1.0):
DeepSearch: Information Relevance, Tool Use Quality, Source Quality, Information Synthesis
MicroSandbox: Code Correctness, Tool Use Quality, Computational Efficiency, Result Interpretation
Perform Web Task: Tool Use Quality, Content Extraction, Interaction Quality, Goal Achievement
Tavily: Tool Use Quality, Information Relevance, Content Extraction, Goal Achievement
Final Assessment: Task Completion, Tool Use Quality, Reasoning Coherence, Problem Resolution, Answer Correctness
Reward Model Integration Architecture
1. Reward Model Setup
Model Configuration
- Base Model: Fine-tuned Qwen-8B
- Input Format: Task instruction + previous clips summary + current clip content
- Output Format: JSON with scores and summaries
- Inference Mode: Local deployment for real-time evaluation
Reward Model Integration Architecture
Reward Model Setup
Model Configuration
- Base Model: Fine-tuned Qwen-8B
- Input Format: Task instruction + previous clips summary + current clip content
- Output Format: JSON with scores and summaries
- Inference Mode: Local deployment for real-time evaluation
Reward Design Schemes 以下可以做对比实验（或者说是消融实验）
Scheme 1: Output-Only Reward
This scheme focuses solely on the final output quality without considering the process.就是tool-star 那一套
Scheme 2: Combined Process + Output Reward (Primary Scheme)
将PRM 和 Output-Reward 结合
This is the core scheme that combines process evaluation with output evaluation, providing a balanced signal that values both the reasoning journey and the final destination.
数学公式
The combined reward scheme is mathematically formulated as:
$$R_{final} = \begin{cases}
-1.0 & \text{if } \text{format_invalid}(T) \
\alpha \cdot R_{output} + \beta \cdot R_{process} & \text{otherwise}
\end{cases}$$
Where:
- $\alpha = 0.7$ (output weight) - emphasizes final correctness
- $\beta = 0.3$ (process weight) - encourages good reasoning
- $\alpha + \beta = 1.0$ (normalized weights)
- 具体的$\alpha， \beta$参数可以结合实验调整
Output Reward Component
$$R_{output} = \begin{cases}
1.0 & \text{if } \text{format_valid}(T) \land \text{answer_correct}(T, G) \
0.0 & \text{if } \text{format_valid}(T) \land \neg \text{answer_correct}(T, G) \
-1.0 & \text{if } \neg \text{format_valid}(T)
\end{cases}$$
Where:
- $T$ = trajectory
- $G$ = ground truth
- $\text{format_valid}(T)$ = boolean function checking XML format compliance
- $\text{answer_correct}(T, G)$ = semantic similarity function $> 0.8$
Process Reward Component
The process reward integrates both clip-level and category-level evaluations:
$$R_{process} = \gamma \cdot R_{clips} + (1-\gamma) \cdot R_{categories}$$
Where $\gamma = 0.6$ balances clip and category contributions.
clip score 主要注重推理的逻辑性是否合理，category score 主要注重工具调用的质量如何
Clip-Level Process Reward:
$$R_{clips} = \frac{\sum_{i=1}^{n} w_i \cdot s_i}{\sum_{i=1}^{n} w_i}$$
Where:
- $n$ = number of clips
- $w_i$ = progressive weight for clip $i$: $w_i = \frac{i}{n} \times \begin{cases} 1.5 & \text{if final clip} \ 1.0 & \text{otherwise} \end{cases}$
- $s_i$ = clip score: $\begin{cases} \text{correctness_score}_i & \text{if final clip} \ \text{reasonableness_score}_i & \text{otherwise} \end{cases}$
Category-Level Process Reward:
$$R_{categories} = \frac{\sum_{k=1}^{m} \lambda_k \cdot \bar{s}k}{\sum{k=1}^{m} \lambda_k}$$
Where:
- $m$ = number of categories
- $\lambda_k$ = category importance weight
- $\bar{s}k$ = average metric score for category $k$: $\bar{s}k = \frac{1}{|M_k|} \sum{j \in M_k} s{k,j}$
- $M_k$ = set of metrics for category $k$
Progressive Weighting Rationale
The progressive weighting scheme $w_i = \frac{i}{n}$ ensures that:
- Early clips (foundation setting) receive lower weights
- Later clips (building on context) receive higher weights
- Final clip receives a 1.5× boost for task completion importance
This reflects the natural importance hierarchy in multi-step reasoning tasks.
Scheme 3: Format Penalty Override 格式奖励
This ensures that format violations receive immediate negative rewards regardless of other factors.
Implementation
1def apply_format_penalty_override(base_reward: float, output_format_valid: bool) -> float:
2 """
3 Override any positive reward if format is invalid
4 """
5 if not output_format_valid:
6 return -1.0
7 return base_reward
