Can I train a model that beats a frontier model on a narrow, well-defined task?
Seeing a explosion of agent use case they are powered by LLM. Most agents use only a very narrow subset of their capabilities. The frontier models are expensive . Claude Sonnet lists at $3 / $15 per million input/output tokens, while self served model (Llama 3 8B Instruct) serves only at $0.10 per million tokens on Together AI -- 30x difference.
I have been using claude intensively for finance research - and I'm wondering if I can do the same for the same purpose - with a well defined scope. So here is the finance Q&A agent that can do tool call reatime:
Supported functionality including:
marketing data extraction
comparison across tickers
screening stocks in a specific sector
gathering news and insider trading signal
Model learns how to call these tools categories: Discovery, Per-ticker Data, News & Filings, Macro and Synthesis
Architecture
image
image
The core idea for RL training on tool use is to let the LLM learn the correct format, then during validation, the agent actually calls the tool and verify the reward based on the rubrics: trajectory, and final answer, etc.
For example: when I prompt:
What was NVDA's closing price on 2026-03-14?
Step 1 — Model emits tool call (token stream from the policy)
< thinking>
User wants a historical closing price for one ticker on one date.
Single call to get_price_history is sufficient.
< /thinking>
< tool_call>
{"name": "get_price_history",
"arguments": {"ticker": "NVDA", "start": "2026-03-14", "end": "2026-03-14", "interval": "1d"}}
< /tool_call>
Step 2 — Harness executes the tool (cache-backed yfinance in training, live in eval)
< tool_response>
{"ticker": "NVDA","rows": [{"date": "2026-03-14", "open": 142.10, "high": 144.85, "low": 141.62, "close": 144.27, "volume": 287340000}]}
< /tool_response>
Step 3 — Model emits final answer (after seeing the tool response)
< answer>
NVDA closed at $144.27 on 2026-03-14.
< /answer>
Environment:
We tended to avoid using real tool call during training - flaky and unpredictable. For verifiable tasks, we build a cache with SQLite/FastAPI; so when rollout harness call the tooling it's actually hitting the cache not real tool. We turn off the cache mode during serving so it can be actually querying live data; for open ended onces we used LLM as a judge.
Trajectory:
we need to generate data with correct trajectory: we first generate a few seed trajectory, under each use case; then ask claude to enrich the dataset. The dataset is curated to match the cache data.
Evaluation:
Compare the result with sonnet 4.6 on the same scoring; while use LLM as a judge (opus) on live data for validation
Reward :
This is interesting part as I spent a lot of time on different failing scenarios. Curret design after couple rounds of interations
Format gate — invalid format → cap at 0.10
Answer — deterministic scoring where possible
(exact/numeric/set), LLM judge only for open-ended
Tool efficiency — bonus only when answer is correct (multiplicative)
Refusal — ±0.30 signed override
Couple things I have learned along the way:
Use gates not sums, as policy could game the cheapest component ( for example if format is incorrect then it it should be penalized hard)
Turn exploit into negative signals. . For example, answering from memory, citing wrong field, matching gold via disclaimers. Each becomes a penalty terms
Reward hacking detection for each component for earlier detection.
5. What broke, and the fix
This is my second time doing RLHF and it took embarrisingly a couple iterations before the model is usable.
This is a list of interesting failure I have
Reward design
The wrong-field exploit: The model learned to call the right tool, get the right ticker, then report the wrong number. The answer score was 0.0 but tool_name_match and args_match were both 1.0, so the composite reward was still positive. Fixed by gating.
Premature-answer hack: The model discovered it could emit <tool_call> blocks and then immediately write < answer> using memorized knowledge. Fix: gating
The rubric length penalty catastrophe: Multi-turn never worked because of a careless reward function that aiming to reduce the response length
silence failing in the reward function When the model passed a malformed integer as a ticker arg (e.g., {"ticker": 12345}), the reward function called int.upper() and crashed. Instead of propagating the error, veRL caught the exception and assigned a default low score. This taught the model to avoid those tool calls entirely
Environment
Didn't enforce using cache mode during training, causing training instability.
vLLM OOM during GRPO weight update:reducing gpu_memory_utilization from 0.4→0.20, enabling param/optimizer offload to CPU, reducing microbatch sizes to 1, cutting sequence length from 16k→6144.
Data
Snapshot bias : All training rows had [today: 2026-05-02] because we started with data from their specific date. Model learned to treat every query as if it were May 2, 2026. On live eval, it couldn't handle historical dates,mfuture dates,mor holiday/weekend gaps. fix: full regen with date-diverse prompts and date validator, and update cache
Data corruption: for example a missing gold value for one family, self-contradirectory labels, this type error is found by family breakdown analysis the regen is required
Multi turn data error : The model emitted both tool calls in a single assistant turn all at once because of uncleared requiment definition ( multi-tool versus multi-turn). this was only caught after I started testing the tool in the UI and notice that it never generates calls in sequence .
other
SFT/inference prompt mismatch : SFT saw a 1.3K-char prompt; GRPO saw 14K because Qwen3's template silently injected tool schemas. Demo used a third variant. All three ran without errors — until tested in ui I started to investigate the rollout out. Fix: unified SYSTEM_PROMPT_WITH_TOOLS across all environments. SFT became effective immediately.
