AI features tend to ship with a demo, a prompt and a hope. That is enough to get something in front of users, but not enough to operate it. Before launch, I want six numbers on one page. They are not perfect metrics, but they force the conversations that otherwise happen after the first bill or the first outage.
1. Cost per successful task
Cost per request is misleading when half the requests fail and get retried. Divide total model and tool spend by the number of tasks that actually completed. Track it per feature and per customer tier if you can.
cost_per_success = (
input_tokens * input_price
+ output_tokens * output_price
+ tool_costs
+ retry_costs
) / successful_tasksPut this in a dashboard before launch. A feature that costs ten cents per success at 100 users can be fine and catastrophic at 100,000.
2. p95 latency, not average
Averages hide the requests users remember. Measure end-to-end latency at p50, p95 and p99, and break it into queue time, retrieval, time to first token, generation and rendering. If p95 is four times p50, you have a queueing, cold-start or long-tail prompt problem.
Streaming improves perceived latency but not total completion time. Measure both: time to first token and time to done.
3. Token budget per request
A hard ceiling on input and output tokens protects you from a pathological prompt or a runaway agent loop. Set the ceiling from the product requirement, not from the model's maximum. Then enforce it server-side; a client-side limit is a suggestion.
Log the distribution. If p95 input is close to the ceiling, your retrieval or prompt assembly is bloated.
4. Cache hit rate
Prompt caching, exact-match response caching and retrieval caching all save money and latency. But a high hit rate can also mean stale answers. Track hit rate alongside the age of the cached entry and the rate of explicit invalidations. A cache with no invalidation strategy is a bug waiting for a support ticket.
5. Eval pass rate
Pick a small set of representative tasks and a pass condition. Run it on every prompt, model or retrieval change. Report the pass rate and the delta from the previous version. A 92% pass rate that drops to 84% after a 'small' prompt edit is the signal you need.
Keep a few adversarial cases: an empty input, a very long input, a prompt injection attempt, a request outside the feature's scope. The goal is not a perfect score; it is catching regressions before users do.
6. Tool failure rate
If the feature calls tools, measure failures by tool and by error class: timeout, 4xx, 5xx, rate limit, validation. A 2% tool failure rate can be invisible in a demo and dominant in production. Pair it with retry rate and duplicate-side-effect count.
| Number | What it answers | Where to look |
|---|---|---|
| Cost per success | Can we afford this at scale? | Billing + task completion |
| p95 latency | Does it feel slow? | Tracing spans |
| Token budget | Are prompts under control? | Model request logs |
| Cache hit rate | Are we paying twice? | Cache metrics + invalidations |
| Eval pass rate | Did the last change break it? | CI eval run |
| Tool failure rate | Is the outside world reliable? | Tool logs by error class |
What to remember
- Measure cost per successful task, not per request.
- Track p95 and p99 latency, and separate time to first token from time to done.
- Enforce token and step budgets on the server.
- Pair cache hit rate with staleness and invalidation metrics.
- Run a small eval set on every change and watch tool failures by error class.
