TokelyTHE SHORT ANSWER
LLM observability should connect each user action to its model calls, token usage, cost, latency, errors and accepted outcome. Token totals explain consumption. Traces explain why a feature became slow or expensive. Quality measurements tell you whether the spending produced something useful.
What belongs in a request record?
Assign an application request ID before calling a model. Record a trace ID, tenant identifier, feature, prompt version, requested model, actual route where disclosed, attempt number, start time, duration and result status. Keep upstream request IDs when available so support teams can investigate specific failures.
Record usage by billing category. Regular input, cache reads, cache writes and output are not interchangeable. Keep the tariff version or effective date used for your estimate. Where a provider exposes reasoning tokens as a subset of output, do not add them twice.
Which latency metrics matter?
Measure queue time, time to first token and total completion time separately. A fast provider behind an overloaded application queue still gives users a slow experience. Streaming responsiveness and completed-job throughput answer different product questions.
Use distributions rather than a single average. For example, a hypothetical feature with a 2-second median and a 20-second 95th percentile may feel inconsistent even when the average looks acceptable. Compare like workloads: prompt length, output length, region and concurrency all affect interpretation.
How do you measure failures accurately?
Separate transport failures, authentication errors, rate limits, provider errors, invalid output and user cancellation. A valid HTTP response containing malformed task output is an application failure even when the API reports success.
Track first-attempt success separately from success after recovery. Retries can hide a deteriorating route while increasing cost. Report partial streams and unknown completion states explicitly instead of labeling every timeout as unbilled.
How do you connect cost to quality?
Use cost per accepted task: total workflow spend divided by outputs that pass your acceptance criteria. For an extractor, acceptance may require a valid schema and correct fields. For code generation, it may require tests and reviewer approval.
An illustrative $40 run yielding 800 accepted results costs $0.05 per accepted result. A $30 run yielding 400 costs $0.075. The cheaper API run produced worse economics.
Should observability store every prompt?
Prefer operational metadata by default. If content logging is needed for evaluation, define sampling, redaction, access control and retention. Prompt text can contain customer secrets. Hashes and version IDs often support debugging without copying full documents into another system.
OpenTelemetry’s generative AI conventions provide a shared vocabulary; check the stability of the conventions before adopting individual fields.
Use Tokely’s usage and billing documentation to reconcile your application ledger with provider-reported consumption. Your product-level trace should remain the place where multiple API calls become one customer outcome.