LLMs, GenAI, RAG, agents, evaluation · insight-1567
Search a large tool catalogue on demand instead of loading every definition
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- saas
Summary: situation, action, reason, boundary
Situation
An assistant connected to several internal systems exposes hundreds of tools, and tool choice is wrong or slow.
Action
Keep a few frequent tools always loaded, put the rest behind a tool-search step that returns only the handful relevant to the request, and test tool-selection accuracy on your own tasks as the catalogue grows.
Reason
Vendor documentation states that loading all definitions bloats context and that selection accuracy degrades beyond roughly 30 to 50 tools (vendor-reported guidance for its own models, not an independent benchmark), which on-demand loading is designed to avoid.
Boundary
Assistants over many MCP servers or internal APIs.
Symptom or risk
Definitions fill the context window and the model picks the wrong tool among similar ones, so cost and errors both rise.
Exceptions
Fewer than about ten tools, all used often, do better loaded directly.
Trade-offs
An extra search hop and a retrieval step that can miss the right tool.
Pitfalls
Poorly named or described tools that search cannot find.
Lessons and concepts
Claims and their sources
Anthropic's tool search documentation says loading every tool definition causes context bloat and that selection accuracy degrades once tools exceed roughly 30 to 50, and it recommends keeping the three to five most used tools loaded (vendor guidance for its own models).