LLMs, GenAI, RAG, agents, evaluation · insight-1513
Score tool calls on argument match and on executed outcome, since each alone misleads
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A flight-change agent is evaluated by exact string comparison of its tool-call JSON against a reference call.
Action
Compare calls structurally with documented normalisation (function name, required and no extra parameters, types, accepted value variants), execute them against a sandbox or recorded service to compare outcomes, report both, and include cases where no call is correct.
Reason
The Berkeley Function Calling Leaderboard pairs AST matching with executable evaluation because not every function is easily executable, and tests relevance detection, where no function fits and none should be called.
Boundary
Function-calling agents whose tools are deterministic or can be sandboxed.
Symptom or risk
Equivalent calls (reordered keys, AYT versus ayt, another ISO date form) are marked wrong, while calls that match the reference but hit a stale or unavailable service pass.
Exceptions
Tools with side effects that cannot be sandboxed, where structural matching plus review is what remains.
Trade-offs
Normalisation rules need maintenance and executable tests need fixtures.
Pitfalls
Leaving out the cases where the right behaviour is to make no call.
Lessons and concepts
Claims and their sources
The Berkeley Function Calling Leaderboard blog describes AST evaluation (function name, required parameters, no undocumented parameters, type and value checks) used because not all functions are easily executable, executable evaluation that runs calls and compares outputs, and relevance detection with functions that do not fit the request.