Most people’s experience of AI is renting it. You type into someone else’s app, your words go to someone else’s servers, and you pay by the token. That works for a quick question. It’s a lot less comfortable when the AI is working on a project all day, reading its tickets, its documents and its code.
So for a project I’m working on, I run my AI bots on local models: AI models running on my own machine. The tool that makes that practical is Rakazo, an open source bot platform that works with whatever model you point it at, including local ones.
Why local models
Three reasons, in order of how much they matter to me.
The work stays with me. Bots on a real project see real material: unreleased plans, internal documents, code, half-finished decisions. With a local model, that material is processed on my machine. Nothing about the project is sent to a model provider to be stored, logged or used under someone else’s terms.
Always-on doesn’t mean always paying. Bots that run routines, check things on a schedule and work through long tasks use a lot of tokens. On a hosted API that’s a meter running all day. On a local model, the cost is the hardware I already own and the power it draws, so I don’t hesitate to let a bot re-read a document or try a task again.
Nothing changes under me. Hosted models get updated, rate-limited or retired on the provider’s schedule. A local model behaves the same way tomorrow as it did today, which matters once you’ve tuned a bot’s role around how its model responds.
How I actually use it
The bots are organized the way I’d organize people: one job each. There’s a bot for engineering work, one that keeps Jira moving, one that writes up decisions in Confluence, and an assistant for the admin around the project in my Google tools.
A normal day looks something like this:
- I talk instead of type. With dictation I brain-dump a task out loud and the bot turns it into a proper brief. When a bot needs a decision, it can call me in the app and we talk it through.
- Bots keep the systems of record current. New Jira issues get triaged and summarized, finished work gets reflected in Confluence, and I get a short daily rundown of what moved and what needs me.
- Engineering work happens behind approvals. The engineering bot can work on code and prepare changes and deploys, but merging, deploying and anything irreversible wait for me to approve. Everything a bot does is logged.
The interesting part is that none of this feels different from using a hosted model day to day. The difference is underneath: where the thinking happens.
Picking the right model for each bot
Rakazo lets each bot use its own model, and that’s what makes local models work well. Not every job needs the biggest model you can run:
- Routine, high-volume work such as triaging tickets, summarizing changes or drafting status updates runs well on a smaller, faster local model.
- Work that needs more reasoning, like engineering tasks or untangling a messy problem, gets the most capable model my hardware can handle.
- The option to go hosted is always there. If a task genuinely needs a frontier model, I can point that one bot at a hosted provider without changing anything else.
Matching the model to the job keeps bots responsive and leaves the machine’s capacity for the work that needs it.
The trade-offs
Local models aren’t free of compromises, and it’s worth being honest about them.
- They’re smaller than the frontier models. That shows on long, open-ended tasks. The fix is the same one that helps any agent: narrow roles, clear instructions and good tools, so the model isn’t asked to figure everything out at once.
- Hardware matters. The size of the model you can run, and how fast it responds, depends on your machine’s memory and processor. A capable Mac goes a long way, but it’s a real constraint.
- You’re the ops team. Keeping models updated and the machine running is your job, not a provider’s.
For this project, those trade-offs are easily worth it.
What I’ve learned
- Privacy changes what you’re willing to delegate. I hand over far more real project work to bots running locally than I ever would through a hosted chat.
- Narrow roles make smaller models shine. A focused job description does more for a local model than any amount of prompt tweaking.
- Mix models by job. Small and fast for routine work, larger for reasoning, hosted only when it’s truly needed.
- Approvals still matter. Running locally keeps the data private; it doesn’t make a bot’s judgment perfect. Clear stopping points are still what make real autonomy safe.