---
title: "AI costs and latency: budget for a completed task"
description: "Published Arne Kellmann website information and agent integration guidance."
canonical: https://arnekellmann.de/ai-cost-latency
last-updated: 2026-09-26
---

# AI costs and latency: budget for a completed task

Measure the full request path, include retries and fallback, and choose where caching or deterministic code helps. A guide to practical AI cost and response-time budgets.

## Count completed tasks, not just model calls

The price of one successful inference is an incomplete operating budget. A user task may also need retrieval, several model calls, validation, retries and a fallback. Measure that entire path and separate completed, failed and abandoned tasks.

Use expected volume to build a scenario, then compare it with observed usage. State assumptions explicitly: input size, output size, retry rate and provider prices can all change. A scenario is not a spending cap.

## Find the work that does not need to repeat

Static information and deterministic rules often offer a simpler saving than a different model. The appropriate boundary depends on data freshness, permissions and the consequence of a stale answer.

| Repeated work | A useful design decision |
| --- | --- |
| Serving rarely changing public content | Generate it ahead of time and deliver static assets; do not query a database per visit. |
| Retrieving the same authorized context | Cache only within a defined freshness window and access boundary. |
| Parsing, arithmetic or checking a schema | Use deterministic code; reserve model calls for the semantic decision. |
| Independent tasks without an immediate deadline | Consider batching only when the provider supports it and the delay is acceptable. |

## Put a time budget around the entire path

Measure end-to-end response time alongside each stage. Look at the median and the slower requests, including failures. Bound retries by a deadline and decide what the user sees when that deadline is reached.

My published BTC/USD replay compared decisions and measured round-trip latency from Frankfurt. It shows a specific case where waiting for a decision matters. Those measurements apply to that study’s models, date and workload; they do not predict your application’s response time.

## Optimize against quality as well as the bill

Compare candidates on the same evaluation set before changing a provider or shortening context. A cheaper response that causes more retries or worse decisions can increase the cost of a completed task. Keep the acceptable error rate and response-time target visible next to the cost model.

An architecture assessment can document this request path and its per-request costs. It helps the team choose what to cache, what to measure next and where a model is actually necessary.

- [Measured decisions and round-trip latency in the BTC/USD replay](/blog/the-slow-model-was-right-more-often)
- [Architecture review: inputs and deliverables](/ai-architecture-review)