Building the project has been the most interesting engineering work I’ve done in years, and also the most humbling. A lot of what I believed about AI agents at the start turned out to be wrong, or at least incomplete. Here’s what I’d tell anyone starting on something similar.
1. The demo is the easy part
You can build an impressive agent demo in an afternoon. It handles the happy path beautifully and everyone in the room is excited. Then it meets real work: the email with three questions in it, the customer who’s angry about something unrelated, the record that’s missing a field everyone assumed was always there.
The gap between a demo and a dependable agent is almost entirely edge cases, and you only find them by running agents on real work, carefully, and watching what happens.
2. Narrow beats clever
Our first agents were generalists with long instructions and broad tools. They were clever and unpredictable. Splitting the work into narrow roles, each with a short brief and a handful of purpose-built tools, made agents less impressive in a demo and far more reliable in production.
If an agent’s instructions keep growing to cover new cases, that’s usually a sign it should be two agents.
3. Tools matter more than prompts
We spent weeks tuning prompts that would have been fixed in an hour by changing a tool. If the model keeps getting a parameter wrong, the tool is too complicated. If it keeps calling things in the wrong order, the tool should probably do both steps itself.
Good tool design is good API design: clear names, few parameters, helpful errors. Write tools for the model the way you’d write an API for a new developer.
4. Evaluation is the product
The single biggest change in how we worked came from taking evaluations seriously. Before that, every change was a judgment call: “this prompt feels better.” Afterward, it was a measurement.
Every real failure becomes an eval case. Every change runs against the whole set. Over time the evals became the most valuable thing we’d built, because they captured everything we’d learned about what “good” looks like for each role.
5. People want to approve, not supervise
We assumed people would want to watch agents work. They don’t. They want to be asked at the right moment, with enough information to decide quickly, and otherwise be left alone.
Designing the approval experience, short and clear with the exact action shown, did more for adoption than any improvement to the agents themselves. Trust grows from a long run of approvals that turned out to be right.
6. Start with the boring work
The roles that delivered value fastest weren’t the exciting ones. They were the repetitive, well-defined tasks that nobody wanted to do: syncing records, chasing missing information, filling in the weekly report, finding cover for a shift. Clear inputs, clear outputs and a clear definition of done make for good agent work.
The ambitious roles came later, built on the trust earned by the boring ones.
7. Watch the costs from day one
Agents that loop, retry and re-read the same context can quietly get expensive. Budgets per task, the right model for each step and caching what doesn’t change kept costs predictable. Cost per completed task is now one of the first numbers we look at for any role.
8. It’s still a people project
The technology is the smaller part of the challenge. The larger part is helping a team decide what they want agents to do, what they want to keep doing themselves, and how they’ll check the work. The businesses that got the most out of the project treated it like bringing on new team members, not like installing software.
What’s next
The direction from here is more of the same discipline: more roles, better tools, deeper evaluations and agents that earn more autonomy one approved action at a time. I’ll keep writing about it here as the project grows.