The request behind the loading spinner
Follow a request across services with traces, and turn a vague report of slowness into a useful question.
“The page is slow” is a real user report, but it is not yet a diagnosis. The delay could come from the browser, the API, a database query, a queue, or an external dependency. A collection of isolated log lines may mention every service without explaining how they participated in the same request.
Follow one journey
Distributed tracing represents work as spans connected into a trace. A span describes an operation over time and can carry attributes, events, and a relationship to other spans. OpenTelemetry provides a shared vocabulary and instrumentation model for these signals.
For a hypothetical search page, the trace might connect the incoming HTTP request to permission checking, retrieval, and response construction. If retrieval accounts for most of the elapsed time, there is a concrete place to investigate. If several operations overlap, adding their durations will overstate the total delay; the shape of the timeline matters as much as individual numbers.
Name operations for investigation
A span called “function” does little for the person debugging at midnight. A stable name such as “search.retrieve” gives the work an identity. Attributes can describe relevant context, but they should be chosen deliberately. Avoid putting access tokens, raw prompts, personal documents, or unbounded user input into telemetry by default.
Think about the question each attribute helps answer. A deployment version helps compare regressions. A broad outcome category helps group failures. The entire request body may add considerable privacy risk without improving routine diagnosis. Retention and access controls belong in the observability design, not in an afterthought.
Keep the other signals
Traces are not replacements for metrics or logs. A metric can reveal that latency has increased across many requests. A trace can show where one request spent its time. A structured log can explain a specific decision or error. Correlation identifiers make it easier to move between those views.
Sampling also limits what is available. If only a subset of traces is retained, a missing trace does not prove a request never happened. Choose sampling and retention policies with the operational questions in mind, and explain those limits to anyone relying on the data.
Instrument a real question first
Rather than adding spans to every small function, start with a question: why do some searches take much longer than others? Instrument the major boundaries, compare representative traces, and add detail where the evidence remains ambiguous.
After changing the system, check whether the measured bottleneck actually improved and whether another stage now dominates. Observability should shorten the distance between a person's experience and an engineering decision. A beautiful dashboard that cannot help explain the loading spinner has missed the point.
Further reading: OpenTelemetry: Traces.