Senior · Architecture and systems
Useful telemetry under a cardinality budget
A service adds user_id, request_id and transaction_hash as Prometheus-style metric labels to debug latency. What goes wrong?
Try explaining this aloud before opening the answer. State your assumptions and trace a boundary case.
package main
import (
"context"
"log/slog"
"os"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
ctx := context.Background()
logger.InfoContext(ctx, "order rejected", "operation", "place_order", "reason", "insufficient_stock", "request_id", "req-example")
}Reveal a strong answer
A service can emit millions of log lines while providing no evidence about a failed business operation. Start with a question: is the checkout successful within its time budget, and where is the budget spent? Logs explain individual events, metrics describe distributions and rates, and traces connect sampled work across boundaries. They should share stable operation semantics without sharing every field.
Define latency and error SLIs at the user-visible boundary. Measure dependency waiting separately from execution, and include retries in the end-to-end budget. Use bounded labels such as operation and outcome. Put trace/request identifiers in logs or exemplars. With OpenTelemetry, propagate context, end spans, record errors appropriately and flush exporters during bounded shutdown. The event has a fixed message and named fields, so a reader can search by operation and reason rather than parse prose. request_id is a log field, not a metric dimension. slog does not automatically create a distributed trace; tracing requires explicit instrumentation and propagation configured in the native service.
User IDs, transaction hashes and raw URLs as metric labels create unbounded time series. Logging tokens, keys or entire request bodies leaks secrets. A healthy average hides tail latency and errors hidden by retries. Test redaction, cancellation and exporter failure; telemetry failure must not accidentally become an unbounded request queue. Pick alerts that correspond to an actionable service objective.
Common trap
More log fields or more metric dimensions are not automatically better evidence.
Try it
Redesign labels and log/trace fields for latency, retry amplification and dependency waiting without recording secrets.
Worked solution
Unique identifiers create unbounded time-series cardinality and expensive storage. Metrics should use a bounded vocabulary such as operation and outcome, while protected logs or sampled traces carry correlation identifiers. Measure the user-visible operation including retry time and distinguish downstream wait from execution. Define an SLO and actionable alerts, test redaction/exporter failure, and do not let telemetry become an unbounded critical-path queue.
Change the constraint
How would exemplars and trace sampling help without hiding rare failures?
How did you do?
Self-assessment, not a compiler pass or interview score. Explain the reasoning unaided before marking understood.
Not marked. Progress stays on this browser, separate from course completion.
Primary references
Checked 2026-10-28. Use documentation for your pinned runtime, driver and tool version.