· 3 min read
Why my AI is not allowed to send anything on its own
On Optiva, the AI reads the emails, prepares the replies, drafts the quotes. Then it stops. Every send goes through a human who clicks. That is not a technical limit, it is a decision, and here is where it comes from.
- Automation
- AI
- Product
When I started Optiva, the idea was simple: a friend had told me his boss received hundreds of emails a day, never managed to deal with all of them, and lost clients because of it. Not for lack of skill. For lack of time for tasks a machine can do today.
So we built agents that sort, reply, follow up, prepare quotes. And very quickly, one question came up: how far do we let them go?
I put myself in the client's shoes
If it were my business, I would not want an AI pushing numbers I have not seen into my accounting software. I would not want a quote reaching a customer without someone having read it. Not because the AI is bad. Because it is sometimes wrong, you don't know when, and there is nothing you can do once it has gone out.
There is a second problem, a more human one. When a tool does the work for you, you stop checking. At first you reread everything, after a month you skim, after three you trust it. The day the AI makes up an amount, nobody sees it go by. And the business pays, not the tool.
So the rule is built into the product, not into a checkbox: nothing goes out without a human's approval, and never full automation when money is involved.
What that looks like in practice
The AI prepares, the code checks, the human decides.
Take a quote. The agent suggests the lines and prices from the business's history. But plain code computes the totals and the GST, not the language model. A model can write a nice sentence; it must not do an addition an invoice depends on.
And when the agent does not have the information, it is allowed to say so. That sounds obvious. It is not what you get by default. An early version of the prompt said, in substance, "if you cannot find the price, make an estimate". An AI asked to estimate always estimates, confidently. We removed that sentence, added an "insufficient information" answer, and a dumb but effective safeguard: a quote where every line is zero is rejected before anyone sees it. A zero quote received by a customer is worse than no quote.
Automatic mode has to be earned
There is still a mode where the agent sends certain replies on its own when no money is at stake, such as a reply to an online review. But it is not switched on by ticking a box. It unlocks once the agent has built up enough drafts the human approved as they were, without edits.
There too, we got caught once. The confidence score was counting drafts the AI had not written, template replies generated by the system. The tool was grading itself. We fixed it by separating the two sources, and automatic mode now only counts what the agent actually produced.
What I take from it for your projects
A useful automation is not the one that does everything. It is the one that does the repetitive work and leaves you the only thing that matters: the decision. If someone sells you an agent that answers your customers and sends your invoices without you looking at anything, ask one question: what happens the day it gets it wrong, and who sees it?
That is the question I design every automation I deliver around.