What a custom AI assistant actually costs to run

The build cost of an AI feature is the part everyone quotes. The running cost is the part that surprises people in month three, because unlike a website, an assistant costs money every single time somebody uses it.

It is entirely possible to model this before you commit. Here is how.

You are billed per token, not per question

Model providers charge for tokens in and tokens out. A token is roughly three quarters of a word. The thing that catches people out is that the question your user typed is a tiny fraction of what you actually send.

A retrieval-augmented assistant — one that answers from your own documents — sends the system instructions, several retrieved passages and often the conversation so far, every single turn. A forty-word question can easily become a three-thousand-token request. Your bill tracks that number, not the question.

The three decisions that move the bill most

• How much context you retrieve. Sending ten passages instead of four roughly doubles the input cost and often does not improve the answer. This is the first thing to tune and the most commonly left at its default.
• Which model handles which request. Most queries do not need the most capable model. Routing the straightforward ones to a smaller model and escalating only when needed is usually the single largest saving available.
• Whether you cache. A surprising share of questions in any business repeat. Caching common answers, and caching the unchanged parts of your prompt where the provider supports it, takes real money off the bill for very little work.

How to model it before building

The arithmetic is simple enough to do on a call. Estimate expected queries per day. Estimate the tokens per query — a realistic starting assumption for a document assistant is two to four thousand in and three to six hundred out. Multiply by the provider rate, then multiply by thirty.

Then do it again at ten times the volume. The second number is the one that matters, because if the feature works, usage will not stay where it started. A design that is comfortable at a hundred queries a day and ruinous at a thousand is a design problem, not a billing surprise.

The costs that are not inference

• Vector storage and queries for the retrieval index, which grows with your document set rather than your traffic.
• Re-embedding when documents change. If your knowledge base is updated daily, this is an ongoing cost, not a one-off.
• Evaluation. If you cannot measure whether answers got worse after a change, you will not notice when they do. Building a test set is work, and it is not optional.
• Human review for anything below your confidence threshold. This is a staffing cost, and it is usually the right trade.

What we do about it

We agree a cost-per-query model during discovery, before any code is written, so the monthly running cost is a number you approved rather than one you discover. Guardrails, caching and model routing are part of the build, not an optimisation project afterwards.

And if the honest answer is that the volume does not justify a custom build — that an off-the-shelf tool does the job for a fraction of it — you will hear that first. It costs us the work and it saves you the budget.