OpenInference

The decision

The CFO asks what each AI feature costs per customer and which ones pay for themselves, so can your team answer from evidence, or from a spreadsheet of list prices? This one is about instrumenting every model call, deciding who owns the numbers, and agreeing what the traces must never contain.

The problem it removes

Most teams know their monthly model bill and not much else, so they can't say which feature spent it, which step failed without telling anyone, or how much went on reasoning nobody ever reads. Without that, every cost conversation is a guess and every quality problem gets found by a customer.

What I ran

I traced my job-search pipeline, which scores a posting and tailors a CV through several model calls, on two cheap models, and switching the tracing on took one line with no other code changes.

Five of six applications failed, and the traces showed why within minutes. Four stopped at the same claim-checking step: the model used its whole 16,000-token limit on hidden reasoning, so the API returned finish_reason "length" with an empty or cut-off answer. The trace still marked those calls OK, and my pipeline accepted the empty answer as a result. Raising the limit to 32,000 fixed three of three.

Hidden reasoning was 69 percent of the spend, and 38 percent went on applications that delivered nothing.

Priced at list rates the traces came to $0.85, while the actual bill was $0.59, mostly because the gateway sent one model to a provider charging 55 percent of list.

Where it sits in the stack

Beside every layer that calls a model. It hooks into the client libraries your teams already use and emits standard OpenTelemetry, so the traces land in whatever observability tool you run, with each call recording the model, tokens in and out, reasoning tokens, cached tokens, latency and finish reason, and your own spans grouping those calls into features, customers or jobs.

The first 30 days

Week 1. Switch on the instrumentation for your busiest AI feature with prompt and response text masked, add one parent span per customer action, and alert on finish reasons that mean truncation as well as on errors.

Weeks 2 to 4. Record the provider's reported cost on each call, join spans to features and customers, and go through the ten most expensive actions with the product owner to decide where reasoning is worth paying for.

Skills. An engineer who knows OpenTelemetry, and someone in finance who actually wants the numbers.

Running cost. The instrumentation is free and storage depends on your backend, and masking cut span size about 36-fold in my run.

Working at day 30 means cost per feature and failure rate per step are on a dashboard, and the CFO's question has a number next to it.

Risk

Traces copy your data by default. Every prompt and response goes into the trace store unless you mask it, which in my run meant a CV and a profile at 121 KB per call, so mask by default and let the DPO decide the exceptions.

Cost is not in the trace. Spans carry tokens, and the price depends on which provider served the call, so record the gateway's reported cost or finance will get numbers a third too high or too low.

Status lies about quality. Truncated answers were marked OK, so monitor finish reasons and output validity alongside error rates.

Pace and ownership. There were a hundred releases in six months across a lot of small packages, and the company behind it sells an observability platform, but the conventions and instrumentors are Apache-2.0 and export to any OpenTelemetry backend, so the exit is cheap as long as you pin versions.

The organisational risk is traces nobody reads, so give cost to the AI lead and finance together, and quality to whoever owns the feature.

Build, adopt or buy

Build. Logging tokens per call is an afternoon, but a shared vocabulary across frameworks and teams, parent spans, masking and exporters every tool understands takes a lot longer, and drifts away from the standard as it grows.

Buy. Observability platforms add dashboards, evaluation and alerting on top of traces like these, and some read this format directly, which is worth it once several teams are shipping AI features.

Adopt the instrumentation as the standard now, and pick the platform later against your own traces.

Verdict

I'd adopt it. One line of setup found a failure that had cost me most of my runs while looking like success everywhere else, and it showed me where two thirds of the spend went, so switch it on with masking from day one, add the provider's actual cost to every call, and alert on finish_reason "length" as well as on errors, because a truncated call keeps an OK status.