Same Brain, Different Model: Testing LLMs in a Real Agent

Most LLM evaluations rely on generic benchmarks that often fail to predict performance in actual production agent systems. This experiment kept the entire agent infrastructure fixed (tools, prompts, retrieval, retry logic, and orchestration) and swapped only the underlying model across 15 LLMs.

The agent was tasked with turning natural language business questions into validated, executable queries against a large analytics store — a realistic, multi-step workflow involving planning, tool use, query generation, self-repair, and output validation.

Key findings:

•⁠ ⁠Some open-weight models matched or outperformed the proprietary baseline on execution accuracy.

•⁠ ⁠Robustness in reasoning, source selection, and self-correction mattered far more than model size or coding fluency.

•⁠ ⁠Standard leaderboards did not reliably predict real-world results in this agent loop.

•⁠ ⁠There were clear trade-offs in latency, token usage, and reliability that only became visible under production-like conditions.

The core message: When choosing models for agentic systems, test them inside your actual workflow rather than relying on generic benchmarks.

This Ledger Entry expands how readers think about evaluating and deploying LLMs in agentic systems by showing that real-world performance depends heavily on how well a model integrates into a specific agent loop — with robustness in reasoning, tool use, and self-repair often mattering more than size or generic benchmark scores.