We had good ingredients. The platform exposes its capabilities through a deep set of GraphQL APIs, so we wrapped those as tools and handed them to the agent. We ran it on Opus 4.8, one of the most capable models available today. We built it on AWS Bedrock with the Strands Agents SDK and connected the tools over MCP. Then we asked it some questions.
It couldn't answer them.
Not the hard ones. The simple ones. "How many people came from this UTM campaign?" is a query the platform can run in its sleep, and the agent had a perfectly good analytics tool that could do exactly that. It got lost looking for it. It spun in cycles. It reached for the wrong tools. With one of the best models in the world and full access to the system, it looked incapable.
There are two easy conclusions to draw from that, and both are wrong. The first is that the model isn't ready. The second is that the build was sloppy. What we had actually done was hand a very capable model every key to the building and no map. The problem was never the model. It was the environment we put it in.
When you expose every API as a tool and let an agent see all of them at once, you don't get a smart generalist. You get a model drowning in options, burning tokens reasoning about tools it will never need, and guessing wrong about which one fits the task. More access made it worse, not better. That is the part most people building their first agent don't see coming - capability does not scale with the number of tools you hand it. It scales with how well the agent understands the small set that matters right now.
Fixing it had nothing to do with swapping the model. It came down to three changes, and not one of them was about making the model smarter.
First, we stopped showing the agent everything. We started treating tool selection as a retrieval problem rather than as something to be solved by stuffing every option into the prompt. For any given request, the agent now sees only the handful of tools relevant to it. Smaller context, lower cost, far less room to get lost. But "which tools are relevant" is a judgment call, and that judgment had to come from somewhere.
Second, we got that judgment from the people who already had it. We sat marketing analytics experts down in front of the agent and watched them work it toward real answers. The real gap was language. A lot of marketing is lingo, and that lingo does not map cleanly onto API call - what a marketer means by an "audience" or a "campaign" is not what the platform literally does to satisfy it. We captured that translation, turned it into reusable skills, and fed the same definitions back into the tool descriptions and the retrieval itself. Once the search understood what an analyst actually means, it started surfacing the right tools instead of guessing. The agent didn't get smarter. The environment around it was distilled into something the system could use.
Third, we removed the architecture instead of adding it. We started as the current playbook tells you to, with a multi-agent setup that delegated skill execution to sub-agents. It lost the thread. Context from the conversation never fully survived the handoff to the sub-agent, no matter how hard we worked to pass it across. So we deleted that layer and put the skills and the tools they run directly in the main agent. That is when it stopped being dumb. We had been shielding the model from the complexity it could handle, and the shield was the problem.
No single one of these was the fix. They worked as a set, and what they have in common is the whole point - none of it came out of the model.
Looking back, that failure had less to do with the model and more to do with something every organization has - critical knowledge trapped inside experienced people.
It came out of people who knew the work. The capability we added wasn't intelligence; it was context, and the context lived in our client's analysts, not in the APIs or the docs. A model can read every endpoint and every page of documentation you have. It cannot read what your best people know and never wrote down, until someone sits with them and turns it into something the system can use. In practice, that's often the most valuable asset in the whole build - the operational knowledge trapped inside experienced people.
Once we did, the same agent that couldn't count campaign traffic started doing the work that mattered - generalizing customer personas out of pixel and cookie-level audiences, the kind of analysis that used to take an expert and an afternoon and now takes a sentence. That is the before and after. Not a better model. The same model, in an environment that finally knew what the work was.
So, if you've been told to put AI inside your product and the first version flailed in a demo, it's worth being precise about what failed. Most of the time, it isn't the model, and it isn't your team. It's that nobody has yet done the unglamorous work of teaching the system what your experts know. That work is the product. It's also the part that never shows up in a stack diagram or a model benchmark, and it is the difference between a demo and a product.
We've now done it inside a live product, in a domain with its own dense vocabulary, on real data. So if your own agent looks underpowered, it's worth checking whether it's actually under-informed. That's usually a short conversation, and it's the one we're glad to have.
