Skip to main content

Observability: OpenTelemetry, Prometheus & Grafana

Table of Contents​

  1. Three Pillars of Observability
  2. Metrics with Prometheus
  3. Distributed Tracing
  4. Logging
  5. OpenTelemetry
  6. Dashboards & Alerting
  7. Python Implementation
  8. Observability Foundations
  9. Structured Logging in Practice
  10. OpenTelemetry Deep Dive
  11. Metrics, Alerting and Incident Response
  12. Senior / Principal — Real World Observability Scenarios

Three Pillars of Observability​

┌─────────────────────────────────────────────────────────────┐
│ THREE PILLARS OF OBSERVABILITY │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ METRICS │ │ TRACES │ │ LOGS │ │
│ │ │ │ │ │ │ │
│ │ Numerical │ │ Request │ │ Discrete │ │
│ │ data about │ │ flows │ │ events │ │
│ │ the system │ │ across │ │ with │ │
│ │ │ │ services │ │ context │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ USE CASES │ │
│ ├─────────────────────────────────────────────────────┤ │
│ │ Metrics: Dashboards, alerting, capacity planning │ │
│ │ Traces: Debug latency issues, understand flow │ │
│ │ Logs: Debug specific errors, audit trails │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘

Metrics with Prometheus​

Prometheus Data Model​

┌─────────────────────────────────────────────────────────────┐
│ PROMETHEUS METRICS │
├─────────────────────────────────────────────────────────────┤
│ │
│ Metric: http_requests_total │
│ Labels: method="GET", status="200", handler="/api/users" │
│ Value: 1234567 │
│ Timestamp: 2026-03-20T12:00:00Z │
│ │
│ Metric Types: │
│ • Counter: Always increasing (e.g., request count) │
│ • Gauge: Can go up and down (e.g., memory usage) │
│ • Histogram: Distribution of values (e.g., latencies) │
│ • Summary: Aggregated percentiles (e.g., p99 latency) │
│ │
└─────────────────────────────────────────────────────────────┘

Prometheus Metrics​

# ============================================
# PROMETHEUS METRICS
# ============================================
from prometheus_client import Counter, Histogram, Gauge, Summary
import random

# Counter - Always incrementing
http_requests_total = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)

# Gauge - Can go up and down
active_connections = Gauge(
'active_connections',
'Number of active connections',
['service']
)

# Histogram - Bucket-based distribution
request_duration_seconds = Histogram(
'request_duration_seconds',
'HTTP request duration in seconds',
['method', 'endpoint'],
buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

# Summary - the Python client tracks only count and sum (no quantiles);
# for percentiles, use a Histogram and histogram_quantile() in PromQL
request_latency = Summary(
'request_latency_seconds',
'Request latency in seconds',
['endpoint']
)

# Using metrics
def handle_request(method, endpoint, status, duration):
http_requests_total.labels(
method=method,
endpoint=endpoint,
status=status
).inc()

request_duration_seconds.labels(
method=method,
endpoint=endpoint
).observe(duration)

request_latency.labels(endpoint=endpoint).observe(duration)

# Simulate usage
for _ in range(100):
handle_request('GET', '/api/users', '200', random.uniform(0.01, 0.5))

Custom Metrics Example​

# ============================================
# BUSINESS METRICS
# ============================================
from prometheus_client import Counter, Histogram, Gauge, Info

# User metrics
user_registrations = Counter(
'user_registrations_total',
'Total user registrations',
['source'] # web, mobile, api
)

user_logins = Counter(
'user_logins_total',
'Total user logins',
['status'] # success, failure
)

# Order metrics
orders_placed = Counter(
'orders_placed_total',
'Total orders placed',
['category', 'payment_method']
)

order_amount = Histogram(
'order_amount_usd',
'Order amount in USD',
['category'],
buckets=[10, 25, 50, 100, 250, 500, 1000, 5000]
)

# System metrics
queue_depth = Gauge(
'task_queue_depth',
'Number of pending tasks',
['queue_name']
)

cache_hit_ratio = Gauge(
'cache_hit_ratio',
'Cache hit ratio',
['cache_name']
)

# Application info
app_info = Info(
'app_version',
'Application version info'
)
app_info.info({'version': '1.2.3', 'environment': 'production'})

Distributed Tracing​

Trace Structure​

┌─────────────────────────────────────────────────────────────┐
│ DISTRIBUTED TRACE │
├─────────────────────────────────────────────────────────────┤
│ │
│ Trace ID: abc123-def456-ghi789 │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ SERVICE A │ │
│ │ Span: GET /api/users (2.5ms) │ │
│ │ └─────────────────────────────────────────────────┐ │ │
│ │ │ SERVICE B │ │ │
│ │ │ Span: Query users (1.2ms) │ │ │
│ │ │ └─────────────────────────────────────────┐ │ │ │
│ │ │ │ SERVICE C (Database) │ │ │ │
│ │ │ │ Span: SELECT users (0.8ms) │ │ │ │
│ │ │ └──────────────────────────────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ │ └─────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘

Span Structure:
┌─────────────────────────────────────────────────────────────┐
│ Span: /api/users │
├─────────────────────────────────────────────────────────────┤
│ Name: /api/users │
│ Trace ID: abc123-def456-ghi789 │
│ Span ID: xyz789-abc123 │
│ Parent ID: (root span) │
│ Start: 12:00:00.000 │
│ End: 12:00:00.0025 │
│ Duration: 2.5ms │
│ Status: OK (0) │
│ │
│ Attributes: │
│ http.method: GET │
│ http.url: /api/users │
│ http.status_code: 200 │
│ │
│ Events: │
│ "Database query started" │
│ "Database query completed" │
│ │
└─────────────────────────────────────────────────────────────┘

OpenTelemetry Tracing​

# ============================================
# OPENTELEMETRY TRACING
# ============================================
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.trace import Status, StatusCode

# Configure tracer
resource = Resource.create({
"service.name": "my-service",
"service.version": "1.0.0",
"deployment.environment": "production"
})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Get tracer
tracer = trace.get_tracer(__name__)

# Create spans
def process_order(order_id):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("order.status", "processing")

try:
# Validate order
with tracer.start_as_current_span("validate_order") as order_span:
order_span.set_attribute("order.id", order_id)
validate_order(order_id)

# Process payment
with tracer.start_as_current_span("process_payment") as payment_span:
payment_span.set_attribute("order.id", order_id)
payment_span.set_attribute("payment.method", "credit_card")
process_payment(order_id)

# Update inventory
with tracer.start_as_current_span("update_inventory") as inv_span:
inv_span.set_attribute("order.id", order_id)
update_inventory(order_id)

span.set_status(Status(StatusCode.OK))
span.set_attribute("order.status", "completed")

except Exception as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise

# Decorator-based tracing
@tracer.start_as_current_span("my_function")
def my_function(param1, param2):
# Function body
pass

Logging​

Structured Logging​

# ============================================
# STRUCTURED LOGGING
# ============================================
import logging
import json
from datetime import datetime

class StructuredFormatter(logging.Formatter):
"""Format logs as JSON for machine parsing"""

def format(self, record):
log_data = {
"timestamp": datetime.utcnow().isoformat(),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"module": record.module,
"function": record.funcName,
"line": record.lineno
}

# Add extra fields
if hasattr(record, 'extra_fields'):
log_data.update(record.extra_fields)

# Add exception info
if record.exc_info:
log_data["exception"] = self.formatException(record.exc_info)

return json.dumps(log_data)

# Configure logger
logger = logging.getLogger("myapp")
logger.setLevel(logging.INFO)

handler = logging.StreamHandler()
handler.setFormatter(StructuredFormatter())
logger.addHandler(handler)

# Custom adapter for structured logging
class StructuredLogger:
def __init__(self, name, extra_fields=None):
self.logger = logging.getLogger(name)
self.extra_fields = extra_fields or {}

def _log(self, level, message, **kwargs):
extra = {**self.extra_fields, **kwargs}
record = logging.LogRecord(
self.logger.name,
level,
"", 0, message, (), None
)
record.extra_fields = extra
self.logger.handle(record)

def info(self, message, **kwargs):
self._log(logging.INFO, message, **kwargs)

def error(self, message, **kwargs):
self._log(logging.ERROR, message, **kwargs)

def warning(self, message, **kwargs):
self._log(logging.WARNING, message, **kwargs)

# Usage
log = StructuredLogger("myapp", {"service": "user-service"})

log.info("User registered", user_id=123, email="john@example.com")
log.error("Payment failed", order_id=456, reason="insufficient_funds")

# Output JSON logs:
# {"timestamp": "...", "level": "INFO", "message": "User registered", "user_id": 123, ...}

Log Correlation​

# ============================================
# LOG CORRELATION WITH TRACES
# ============================================
import logging
from opentelemetry import trace

# Get current trace context
def get_trace_context():
span = trace.get_current_span()
span_context = span.get_span_context()
if span_context.is_valid:
return {
"trace_id": format(span_context.trace_id, '032x'),
"span_id": format(span_context.span_id, '016x')
}
return None

# Structured log with trace context
def correlated_log(logger, level, message, **kwargs):
trace_context = get_trace_context()
if trace_context:
kwargs["trace_id"] = trace_context["trace_id"]
kwargs["span_id"] = trace_context["span_id"]

log_record = {
"timestamp": datetime.utcnow().isoformat(),
"level": level,
"message": message,
**kwargs
}

logger.info(json.dumps(log_record))

# Usage
correlated_log(
logger, "INFO",
"User created successfully",
user_id=123,
email="john@example.com"
)

OpenTelemetry​

Complete OpenTelemetry Setup​

# ============================================
# OPENTELEMETRY COMPLETE SETUP
# ============================================
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import logging

# Setup tracing
def setup_tracing(service_name, otlp_endpoint):
resource = Resource.create({
SERVICE_NAME: service_name,
"deployment.environment": "production"
})

trace_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(trace_exporter)
provider.add_span_processor(processor)

trace.set_tracer_provider(provider)

return trace.get_tracer(service_name)

# Setup metrics
def setup_metrics(service_name, otlp_endpoint):
metric_exporter = OTLPMetricExporter(endpoint=otlp_endpoint)
reader = PeriodicExportingMetricReader(metric_exporter)

resource = Resource.create({SERVICE_NAME: service_name})

provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)

return metrics.get_meter(service_name)

# Instrument libraries
def instrument_app(app):
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

from sqlalchemy import create_engine
engine = create_engine("postgresql://...")
SQLAlchemyInstrumentor().instrument(engine=engine)

# Custom metrics
def create_metrics(meter):
# Counters
request_counter = meter.create_counter(
"http_requests_total",
description="Total HTTP requests",
unit="1"
)

# Histograms
request_duration = meter.create_histogram(
"http_request_duration_seconds",
description="HTTP request duration",
unit="s"
)

# Gauges
active_connections = meter.create_up_down_counter(
"active_connections",
description="Active connections",
unit="1"
)

return {
"requests": request_counter,
"duration": request_duration,
"connections": active_connections
}

# Complete initialization
tracer = setup_tracing("my-service", "http://otel-collector:4317")
meter = setup_metrics("my-service", "http://otel-collector:4317")
metrics_handlers = create_metrics(meter)

Dashboards & Alerting​

Grafana Dashboard JSON​

{
"dashboard": {
"title": "Application Overview",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(http_requests_total[5m])",
"legendFormat": "{{method}} {{endpoint}}"
}
]
},
{
"title": "Error Rate",
"type": "stat",
"targets": [
{
"expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100"
}
]
},
{
"title": "P99 Latency",
"type": "gauge",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[5m])) by (le))"
}
]
}
]
}
}

Prometheus Alert Rules​

# prometheus_alerts.yml
groups:
- name: application_alerts
rules:
# High error rate
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: 'Error rate is {{ $value | humanizePercentage }}'

# High latency
- alert: HighLatency
expr: |
histogram_quantile(0.99, sum(rate(request_duration_seconds_bucket[5m])) by (le)) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "High latency detected"
description: 'P99 latency is {{ $value | printf "%.2f" }}s'

# Service down
- alert: ServiceDown
expr: up{job="my-service"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Service is down"
description: "{{ $labels.instance }} is not responding"

# Memory usage high
- alert: HighMemoryUsage
expr: |
(container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
for: 5m
labels:
severity: warning
annotations:
summary: "Memory usage is high"
description: 'Memory usage is {{ $value | humanizePercentage }}'

Python Implementation​

Flask with Full Observability​

# ============================================
# COMPLETE FLASK OBSERVABILITY
# ============================================
from flask import Flask, request, jsonify
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
from opentelemetry import trace
import time

app = Flask(__name__)

# Initialize OpenTelemetry
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME

resource = Resource.create({SERVICE_NAME: "my-flask-app"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(
OTLPSpanExporter(endpoint="http://otel-collector:4317")
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

FlaskInstrumentor().instrument_app(app)

# Prometheus metrics
from prometheus_client import Counter, Histogram, Gauge

REQUEST_COUNT = Counter(
'http_requests_total',
'Total HTTP requests',
['method', 'endpoint', 'status']
)

REQUEST_LATENCY = Histogram(
'http_request_duration_seconds',
'HTTP request latency',
['method', 'endpoint']
)

ACTIVE_REQUESTS = Gauge(
'active_requests',
'Number of active requests'
)

@app.before_request
def before_request():
request.start_time = time.time()
ACTIVE_REQUESTS.inc()

@app.after_request
def after_request(response):
ACTIVE_REQUESTS.dec()

# Record metrics
endpoint = request.endpoint or 'unknown'
REQUEST_COUNT.labels(
method=request.method,
endpoint=endpoint,
status=response.status_code
).inc()

latency = time.time() - request.start_time
REQUEST_LATENCY.labels(
method=request.method,
endpoint=endpoint
).observe(latency)

return response

@app.route('/api/users/<int:user_id>')
def get_user(user_id):
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("get_user") as span:
span.set_attribute("user.id", user_id)
# ... fetch user logic
return jsonify({"id": user_id, "name": "John"})

@app.route('/metrics')
def metrics():
return generate_latest(), 200, {'Content-Type': CONTENT_TYPE_LATEST}

if __name__ == '__main__':
app.run()

Interview Questions​

Q1: What are the three pillars of observability?​

  • Metrics: Numerical measurements (counters, gauges, histograms)
  • Traces: Request flow across distributed systems
  • Logs: Discrete events with context

Q2: What's the difference between Prometheus Counter and Gauge?​

  • Counter: Always increments, never decreases (e.g., total requests)
  • Gauge: Can go up or down (e.g., current memory usage)

Q3: What is OpenTelemetry?​

A vendor-neutral instrumentation library providing standardized APIs for metrics, traces, and logs. Supports multiple backends (Jaeger, Prometheus, etc.)

Q4: How do you correlate logs with traces?​

By including the trace_id and span_id in log entries. The trace context can be extracted from the current span using OpenTelemetry.

Q5: What is the difference between Histogram and Summary?​

  • Histogram: Buckets values, calculates quantiles server-side (configurable buckets)
  • Summary: Samples observations, calculates quantiles client-side (pre-configured quantiles)

Q6: What is an SLO, SLI, and error budget?​

SLI (Service Level Indicator) is the measured metric, e.g., request success ratio. SLO (Service Level Objective) is the target, e.g., 99.9% success over 30 days. Error budget = 100% - SLO, the allowable failure time. When the budget is exhausted, teams stop risky releases and focus on reliability.

Q7: What is RED vs USE method?​

RED: Rate, Errors, Duration — used for request-driven services (typical web/API). USE: Utilization, Saturation, Errors — used for resources (CPU, memory, disk). Choose by system type: services → RED, resources → USE.

Q8: What is eBPF and why is it important for observability?​

eBPF lets you run sandboxed programs inside the kernel without changing kernel code. It enables high-fidelity, low-overhead observability: deep request tracing, network flows, and profiling across containers without instrumenting every app.

Q9: What is the "three pillars" vs "wide events" debate?​

Three pillars: metrics, logs, traces stored separately. Wide events (popularized by Honeycomb): a single rich event carries trace, log, and metric context together. Wide events reduce correlation joins but change storage cost and tooling. Many teams start with three pillars because ecosystem support is stronger.

Q10: What is cardinality and why does it break Prometheus?​

Cardinality = number of unique label combinations. Prometheus memory grows with every unique series. A label like user_id or request_id explodes cardinality (millions of series) and crashes Prometheus. Keep high-cardinality data in logs/traces, not metrics.

Q11: How do you propagate a trace across services?​

The caller injects a trace ID + span ID into the outgoing request headers (W3C traceparent). The receiving service extracts them and starts its span as a child. In Python, OpenTelemetry propagators handle this automatically via instrumentation.

Q12: What is sampling in tracing and why use it?​

Storing every trace at high traffic is expensive. Head sampling decides at the start of a trace (e.g. keep 10%), so it cannot know whether the request will fail; keeping 100% of error traces needs tail sampling (e.g. in the OpenTelemetry Collector), which decides after the trace completes. This preserves error visibility while bounding cost.

Q13: What is the difference between an alert and a dashboard?​

Dashboard: visualization for humans exploring state. Alert: automated signal that fires an incident when a condition is breached. Alerts must have clear runbooks; dashboards should not page anyone.

Q14: What is structured logging?​

Logs as JSON (or key-value) instead of free text, with fields like level, timestamp, service, trace_id, and user context. Structured logs can be indexed, filtered, and correlated by machines — free text cannot be queried reliably at scale.

Q15: What is the difference between black-box and white-box monitoring?​

Black-box: external probes (ping, TLS cert check, synthetic user flows) that see the system as users do. White-box: internal metrics exposed by the app (latency per endpoint, queue depth). Both are needed: black-box catches what users feel, white-box explains why.

Q16: What is anomaly detection in observability?​

Automated detection of deviations from expected baselines (spikes, dips, new patterns) using statistical or ML methods. It complements threshold alerts because static thresholds miss slow degradation and seasonal variance.

Q17: What is the OpenTelemetry Collector and why use it?​

A vendor-neutral agent that receives telemetry (OTLP), processes it (filtering, batching, sampling), and exports it to backends (Prometheus, Jaeger, Datadog). It decouples app instrumentation from backend choice and reduces vendor lock-in.

Q18: What is distributed tracing vs profiling?​

Tracing: end-to-end request flow across services with timing per span. Profiling: per-function CPU/memory sampling inside one process (pprof, Pyroscope). Tracing answers "where is the request slow"; profiling answers "why is this function slow."

Q19: How do you alert on SLO burn rate?​

Burn rate = how fast you consume the error budget. Use multi-window alerts: page when both the 1-hour and 5-minute burn rates are high (fast burn), and alert when both the 6-hour and 30-minute burn rates are elevated (slower burn). Burn-rate alerts detect problems before a 30-day SLO is violated.

Q20: What is trace context propagation failure and why does it matter?​

If one service drops the trace header, the trace splits into fragments that cannot be joined — you see orphan spans. Common causes: async tasks, message consumers, proxies stripping headers. Test propagation explicitly in integration tests.


Observability Foundations​

Q21: What is the difference between monitoring and observability?​

Monitoring answers questions you knew to ask in advance: "Is CPU above 80%?", "Is the error rate above 1%?". You decide the failure modes up front, build dashboards and alerts for them, and the system tells you when one of those known conditions is true.

Observability is a property of the system: how well you can explain any internal state, including ones you never predicted, from the telemetry it emits. An observable system lets you ask a new question during an incident ("Are only Android users on app version 5.2 in the EU seeing slow checkouts?") without shipping new code.

MonitoringObservability
Question typeKnown unknownsUnknown unknowns
Typical outputDashboards, threshold alertsAd-hoc queries, trace exploration, high-dimensional slicing
Data shapePre-aggregated metricsRich, high-cardinality events, traces and logs with context
Failure it catches"The thing we predicted broke""Something new and weird is happening"

They are not competitors. Monitoring is one practice that sits on top of observable telemetry. A good interview answer: "Monitoring tells me that something is wrong; observability lets me find out why, even when the failure mode is new."

Q22: What are the four golden signals, and how do they relate to RED and USE?​

The four golden signals come from Google's SRE book. They are the minimum set of signals to watch for any user-facing system:

Golden signalMeaningExample
LatencyTime to serve a request (track successful and failed requests separately, because a fast error can make latency look good)p50/p95/p99 request duration
TrafficDemand on the systemRequests per second, messages per second
ErrorsRate of failed requests, explicit (5xx) or implicit (200 with wrong content, or slower than an SLO)5xx ratio
SaturationHow "full" the most constrained resource isQueue depth, connection pool usage, CPU throttling

RED (Rate, Errors, Duration) is the golden signals minus saturation, applied per service. USE (Utilization, Saturation, Errors) is applied per resource (CPU, disk, pool). In practice you use RED for every service endpoint and USE for the resources underneath it. Together they cover all four golden signals.

# Traffic
sum by (service) (rate(http_requests_total[5m]))

# Errors (ratio)
sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (service) (rate(http_requests_total[5m]))

# Latency (p99 from a histogram)
histogram_quantile(0.99, sum by (service, le) (rate(http_request_duration_seconds_bucket[5m])))

# Saturation (CPU throttling of containers)
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ sum by (pod) (rate(container_cpu_cfs_periods_total[5m]))

Saturation is the signal teams most often forget. It is also the one that predicts trouble: latency and errors usually rise only after a resource is already saturated.

Q23: What is continuous profiling and why is it called the fourth pillar?​

A profile shows where in the code a process spends CPU time, allocates memory, or waits on locks, usually drawn as a flame graph. Continuous profiling runs a low-overhead sampling profiler in production all the time and stores the profiles as time series. You can then compare "this function's CPU share before and after the deploy" the same way you compare metrics.

Why it complements the other signals:

  • Metrics say CPU went from 40% to 85%.
  • Traces say the render_invoice span got slower.
  • Profiles say 60% of that CPU is now in json.dumps called from serialize_line_items, a line that changed in the last release.

How it works: a sampling profiler interrupts the process many times per second, records the current stack, and aggregates the stacks. Overhead is typically small because nothing is instrumented, only sampled. For Python, py-spy reads the interpreter's memory from outside the process, so it needs no code changes:

# Live "top"-style view of the hottest functions in a running process
py-spy top --pid 12345

# Record for 30 seconds and write a flame graph
py-spy record --pid 12345 --duration 30 --output profile.svg

# Dump the current stack of every thread (great for "the process is stuck")
py-spy dump --pid 12345

Continuous-profiling backends such as Grafana Pyroscope and Parca store these profiles over time. OpenTelemetry has also been adding profiles as a signal, so profiles can carry the same resource attributes, and eventually trace context, as the rest of your telemetry. Inside a container, py-spy usually needs the SYS_PTRACE capability.

Q24: What are OpenTelemetry semantic conventions and resource attributes, and why do they matter?​

Resource attributes describe the entity producing telemetry: which service, version, host, pod, cloud region. They are attached once to the provider and stamped on every span, metric and log that provider emits.

Semantic conventions are the agreed attribute names for common concepts, such as http.request.method, http.response.status_code, url.path, db.system and messaging.destination.name. They let one dashboard, one alert, or one query work across services written in different languages by different teams.

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider

resource = Resource.create(
{
"service.name": "checkout-api",
"service.version": "2.4.1",
"service.namespace": "payments",
# Newer semantic-convention versions use deployment.environment.name;
# older ones used deployment.environment. Pick one org-wide.
"deployment.environment.name": "production",
}
)
trace.set_tracer_provider(TracerProvider(resource=resource))

You can also set them without code changes through environment variables, which is ideal for platform teams:

export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES="service.version=2.4.1,service.namespace=payments,deployment.environment.name=production"

Why interviewers care:

  • Without a consistent service.name, every backend shows unknown_service and service maps are useless.
  • Semantic conventions have evolved (for example, http.method became http.request.method when the HTTP conventions were stabilised). Mixed versions across services split one concept into two fields, and queries silently miss half the data. A senior engineer pins the convention version in a shared instrumentation library and migrates deliberately.

Q25: What are exemplars and how do they connect metrics to traces?​

An exemplar is a sample data point attached to a metric observation. Usually it is the trace ID of one request that landed in a particular histogram bucket. On a latency graph, Grafana can draw exemplars as dots; clicking a dot in the slow p99 region opens that exact trace.

This closes the biggest gap in metrics: they tell you that p99 is 2 seconds but not which requests were slow. Exemplars give you a concrete example without putting a high-cardinality trace_id label on the metric (which would explode cardinality, see Q10).

from opentelemetry import trace
from prometheus_client import Histogram

REQUEST_LATENCY = Histogram(
"http_request_duration_seconds",
"HTTP request latency",
["route"],
buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)


def record_latency(route: str, seconds: float) -> None:
span_ctx = trace.get_current_span().get_span_context()
exemplar = None
if span_ctx.is_valid and span_ctx.trace_flags.sampled:
exemplar = {"trace_id": format(span_ctx.trace_id, "032x")}
REQUEST_LATENCY.labels(route=route).observe(seconds, exemplar=exemplar)

Points worth mentioning:

  • The Python client only exposes exemplars in the OpenMetrics exposition format, so the scraper must negotiate OpenMetrics.
  • Prometheus must have exemplar storage enabled (it sits behind a feature flag), and Grafana needs a data-source link from the exemplar's trace_id to your tracing backend (Tempo, Jaeger).
  • An exemplar is only useful if that trace was actually kept. With aggressive head sampling, many exemplar trace IDs point to traces that were dropped. Attach exemplars only when the span is sampled (span_ctx.trace_flags.sampled) or use tail sampling that keeps slow traces.

Structured Logging in Practice​

Q26: How do you attach a request ID to every log line in async Python without passing it through every function?​

Use a contextvars.ContextVar. A context variable holds a value per execution context. Each asyncio task gets its own copy of the context when it is created, so two concurrent requests never see each other's ID. That is something a global variable or threading.local cannot guarantee under asyncio, because many requests share one thread.

Then add a logging.Filter that copies the value onto every record, so no call site has to remember to pass it.

import asyncio
import contextvars
import json
import logging
import uuid

request_id_var: contextvars.ContextVar[str] = contextvars.ContextVar("request_id", default="-")


class RequestIdFilter(logging.Filter):
"""Copy the current request ID onto every record this handler emits."""

def filter(self, record: logging.LogRecord) -> bool:
record.request_id = request_id_var.get()
return True


class JsonFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
return json.dumps(
{
"ts": self.formatTime(record),
"level": record.levelname,
"logger": record.name,
"msg": record.getMessage(),
"request_id": getattr(record, "request_id", "-"),
}
)


handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
handler.addFilter(RequestIdFilter()) # handler filter: applies to records from ALL loggers
logging.basicConfig(level=logging.INFO, handlers=[handler])
log = logging.getLogger("orders.service")


async def charge_card() -> None:
log.info("charging card") # no request_id passed explicitly
await asyncio.sleep(0.01)


async def handle_request(incoming_request_id: str | None) -> None:
token = request_id_var.set(incoming_request_id or uuid.uuid4().hex)
try:
log.info("request started")
await charge_card()
log.info("request finished")
finally:
request_id_var.reset(token)


async def main() -> None:
# Two concurrent requests: each task gets its own copy of the context.
await asyncio.gather(handle_request("req-abc"), handle_request(None))


asyncio.run(main())

Details that separate a good answer from a great one:

  • Put the filter on the handler, not the logger. Logger-level filters only run for records logged directly on that logger, not for records propagated up from child loggers like orders.service.
  • In a web framework, set the variable in middleware from the incoming X-Request-ID header (or generate one), echo it back in the response, and forward it on outgoing calls.
  • If you already run OpenTelemetry, the trace ID is your correlation ID. Read it from the current span (as in the Log Correlation section above) instead of inventing a second identifier, or keep both and log them side by side.
  • ThreadPoolExecutor workers do not inherit context automatically; see Q35.

Q27: How do you stop passwords, tokens and personal data from ending up in logs?​

Defence in layers, because any single layer will eventually miss something:

  1. Do not log raw objects. Log an explicit allowlist of fields (order_id, status) instead of request.json, headers or a whole ORM model. This is the control that matters most.
  2. Redact in-process with a logging filter as a safety net.
  3. Redact in the pipeline (OpenTelemetry Collector, Fluent Bit, Vector) so that services you do not control are also covered. For example, the Collector's attributes processor can delete or hash keys such as http.request.header.authorization.
  4. Restrict and expire. Short retention, role-based access to log search, and audit who queries what.
  5. Test it. A unit test that logs a fake password and asserts it never appears in output catches regressions.
import logging
import re

SENSITIVE_KEYS = {"password", "token", "access_token", "authorization", "secret", "api_key", "card_number"}
EMAIL_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
BEARER_RE = re.compile(r"(?i)bearer\s+[a-z0-9._~+/=-]+")
REDACTED = "[REDACTED]"


def scrub(value):
if isinstance(value, dict):
return {k: REDACTED if str(k).lower() in SENSITIVE_KEYS else scrub(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return type(value)(scrub(v) for v in value)
if isinstance(value, str):
value = BEARER_RE.sub("Bearer " + REDACTED, value)
return EMAIL_RE.sub("[EMAIL]", value)
return value


class RedactingFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
record.msg = scrub(record.msg)
if record.args:
record.args = scrub(record.args)
for key in SENSITIVE_KEYS & record.__dict__.keys(): # fields passed via extra=
setattr(record, key, REDACTED)
return True


handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(levelname)s %(name)s %(message)s"))
handler.addFilter(RedactingFilter())
logging.basicConfig(level=logging.INFO, handlers=[handler])
log = logging.getLogger("auth")

log.info("login attempt %s", {"email": "ann@example.com", "password": "hunter2"})
# INFO auth login attempt {'email': '[EMAIL]', 'password': '[REDACTED]'}
log.info("upstream call failed, headers=%s", {"Authorization": "Bearer eyJhbGciOi.abc.def"})
# INFO auth upstream call failed, headers={'Authorization': '[REDACTED]'}
log.info("reset link sent to bob@example.com")
# INFO auth reset link sent to [EMAIL]

Remember that traces leak too: auto-instrumentation can capture SQL statements with literal values, full URLs with query-string tokens, and request headers. Review what your instrumentation records, not just your log calls. Under GDPR and similar laws, a log line holding an email address is personal data with the same deletion and access obligations as the database.

Q28: Logging is slowing down our request path. How do you make it cheap?​

Logging cost comes from three places: building the message, formatting it, and writing it (I/O). The standard library handlers write synchronously on the calling thread, so a slow disk, a full pipe, or a network handler (HTTP, syslog over TCP) adds latency to every request.

Fixes, in order of impact:

  1. Move I/O off the request thread with QueueHandler and QueueListener. The request thread only enqueues a record; a background thread does the formatting and writing.
  2. Format lazily. log.debug("payload=%s", payload) only builds the string if DEBUG is enabled. log.debug(f"payload={payload}") always builds it, even when the record is discarded.
  3. Guard expensive computation with log.isEnabledFor(...).
  4. Log less. One structured event per request with twenty fields is cheaper and more useful than twenty one-line logs.
  5. In containers, write to stdout or stderr and let the node agent ship logs, rather than calling a vendor API from the app.
import atexit
import logging
import logging.handlers
import queue

log_queue: queue.Queue[logging.LogRecord] = queue.Queue(maxsize=10_000)

file_handler = logging.FileHandler("app.log")
file_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))

# The listener owns the slow handler and runs it on a background thread.
listener = logging.handlers.QueueListener(log_queue, file_handler, respect_handler_level=True)
listener.start()
atexit.register(listener.stop) # drains the queue on normal shutdown

root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(logging.handlers.QueueHandler(log_queue)) # cheap: just enqueues

log = logging.getLogger("api")


def expensive_debug_dump() -> str:
return "..." * 10_000


def handle(order_id: int) -> None:
# Lazy %-formatting: the string is only built if the record is emitted.
log.info("order processed id=%s", order_id)

# Guard truly expensive work behind a level check.
if log.isEnabledFor(logging.DEBUG):
log.debug("state dump %s", expensive_debug_dump())


for i in range(3):
handle(i)

The trade-off to name: with a bounded queue, a burst that fills it causes QueueHandler to fail to enqueue and the record is dropped (the handler reports the error). With an unbounded queue (maxsize=0 or negative), you never drop logs but a slow sink can grow memory until the process is OOM-killed. Most production systems prefer bounded queues and a metric that counts dropped records, because losing some logs is better than losing the process.

Q29: Should this piece of data be a metric, a log, a span attribute or a span event?​

A practical decision guide:

Question you will ask laterBest signalWhy
"How many / how fast / what percentage, over time?"MetricCheap to store and query for long periods; ideal for alerting. Labels must be low-cardinality.
"What happened inside this one request, and where was the time spent?"Span attributesHigh-cardinality values (user.id, order.id, feature_flag.variant) are fine on spans and make traces searchable.
"Something notable happened at a point in time during this request"Span eventTimestamped annotation inside the span (a retry, a cache miss, an exception), automatically correlated with the trace.
"What happened, outside any request or in detail we may need for audit?"LogBackground jobs, startup and config, security audit trails, full error context.

Rules of thumb:

  • If you would put an ID in a metric label, it belongs on a span or a log instead.
  • If you are writing log.info("cache miss") inside a traced request, a span event (span.add_event("cache.miss", {"cache.key_prefix": "user"})) is usually better, because it carries the trace context and timing for free.
  • Derive metrics from spans or logs when you can (for example, the Collector's span metrics connector produces RED metrics from spans), so the numbers and the examples come from the same source. Do this before sampling, or the counts will be wrong.
  • Audit logs (who did what, when) are a compliance artefact. Keep them separate from debug logs, with stricter retention and tamper protection.

OpenTelemetry Deep Dive​

Q30: What exactly is inside a W3C traceparent header?​

The W3C Trace Context specification defines two HTTP headers.

traceparent has four dash-separated, lowercase hex fields:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ │ │ │
│ │ │ └─ trace-flags (8 bits); bit 0 = "sampled"
│ │ └─ parent-id: 8 bytes (16 hex) = the caller's span ID
│ └─ trace-id: 16 bytes (32 hex), shared by every span in the trace
└─ version (currently 00)
  • An all-zero trace-id or parent-id is invalid, and the receiver must ignore the header.
  • The sampled flag tells downstream services what the caller decided. A ParentBased sampler honours it, so one sampling decision is made at the root and the whole trace is kept or dropped together.

tracestate carries vendor-specific key/value pairs (vendor1=opaque,vendor2=opaque) alongside traceparent, so several tracing systems can coexist without breaking the standard header.

OpenTelemetry's default propagator writes and reads both headers (plus baggage, see Q31):

import re

from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk.trace import TracerProvider

TRACEPARENT_RE = re.compile(r"^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$")

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("demo")

# Service A: start a span and inject its context into outgoing headers.
with tracer.start_as_current_span("GET /checkout (client)") as client_span:
headers: dict[str, str] = {}
inject(headers)

version, trace_id, parent_id, flags = TRACEPARENT_RE.match(headers["traceparent"]).groups()
print(f"version={version} trace_id={trace_id} parent_id={parent_id} sampled={int(flags, 16) & 1 == 1}")

# Service B: extract the context and continue the same trace.
with tracer.start_as_current_span("GET /checkout (server)", context=extract(headers)) as server_span:
same_trace = server_span.get_span_context().trace_id == client_span.get_span_context().trace_id
parent_is_client = server_span.parent.span_id == client_span.get_span_context().span_id
print(f"same trace: {same_trace}, parent is client span: {parent_is_client}")
# same trace: True, parent is client span: True

Interview follow-up: older systems use other formats, such as Zipkin's B3 headers (X-B3-TraceId, ...) or vendor-specific ones. During a migration you configure a composite propagator (for example OTEL_PROPAGATORS=tracecontext,baggage,b3multi, where B3 needs the separate opentelemetry-propagator-b3 package) so old and new services can still join traces.

Q31: What is OpenTelemetry Baggage, and when should you not use it?​

Baggage is a set of key/value pairs that travels with the request context across process boundaries, in the baggage HTTP header. Any downstream service can read it. Typical uses: tenant tier, experiment or feature-flag variant, or the originating client app. These are values that a deep downstream service needs for routing, sampling or labelling but cannot easily look up.

from opentelemetry import baggage, context
from opentelemetry.propagate import inject

ctx = baggage.set_baggage("tenant.tier", "enterprise")
ctx = baggage.set_baggage("experiment", "new-checkout", context=ctx)
token = context.attach(ctx)
try:
headers: dict[str, str] = {}
inject(headers)
print(headers["baggage"]) # tenant.tier=enterprise,experiment=new-checkout
print(baggage.get_baggage("tenant.tier")) # enterprise
finally:
context.detach(token)

Key points:

  • Baggage is not automatically added to spans, metrics or logs. It only propagates. To record it you copy it explicitly. The contrib packages include a baggage span processor that does this for spans.
  • Never put secrets or personal data in baggage. It is sent in plain headers to every downstream service, including third-party APIs your services call, unless you strip it at the edge.
  • Never trust incoming baggage from the internet. A client can set any value. Strip or overwrite it at the ingress, and treat it as a hint, not as an authorization input.
  • Keep it small. It is added to every outgoing request; the W3C Baggage spec sets size limits and many proxies cap total header size.

Q32: What is the difference between the OpenTelemetry API, the SDK, and the Collector? When do you run the Collector as an agent versus a gateway?​

ComponentWhat it isWho uses it
APIInterfaces for creating spans, metrics and logs. With no SDK installed, every call is a no-op.Library authors. A library can depend on the API without forcing any backend on its users.
SDKThe implementation: samplers, span processors, metric readers, exporters, resource detection.Application owners, configured once at startup.
CollectorA separate process that receives telemetry (OTLP and many other formats), processes it (batching, filtering, redaction, sampling) and exports it to one or more backends.Platform teams.

Collector deployment patterns:

  • Agent (sidecar or DaemonSet, one per node or pod): the app exports to localhost, so export is fast and survives brief backend outages. The agent adds host and Kubernetes metadata.
  • Gateway (a central, horizontally scaled pool): central place for tail sampling, credentials for vendor backends, org-wide redaction, and routing. Apps never hold vendor API keys.
  • Most large deployments use both: app → agent → gateway → backends.
# Minimal, production-shaped Collector pipeline
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318

processors:
memory_limiter: # first: refuse data before the Collector itself is OOM-killed
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 25
batch: {} # last: fewer, larger export requests

exporters:
otlp:
endpoint: tempo.observability.svc:4317
tls:
insecure: true # in-cluster example only; use TLS across networks

service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp]

A senior answer adds that the Collector is now critical infrastructure. It needs its own resource limits, horizontal scaling, and alerts on its internal metrics (refused and failed exports, queue size). An unmonitored Collector means a silent loss of all telemetry.

Q33: How does tail-based sampling work in the OpenTelemetry Collector, and what are its operational pitfalls?​

Head sampling decides at the root span, before anything is known about the request (for example, keep 10% at random). It is cheap and consistent but cannot prefer errors or slow requests, because they have not happened yet.

Tail sampling buffers all spans of a trace for a short time, then decides using the complete trace: keep every trace with an error, every trace slower than 2 s, and 5% of the rest.

processors:
tail_sampling:
decision_wait: 10s # how long to buffer a trace before deciding
num_traces: 100000 # traces held in memory at once
expected_new_traces_per_sec: 2000
policies: # a trace is kept if ANY policy says "sample"
- name: keep-errors
type: status_code
status_code:
status_codes: [ERROR]
- name: keep-slow
type: latency
latency:
threshold_ms: 2000
- name: baseline
type: probabilistic
probabilistic:
sampling_percentage: 5

The pitfall almost everyone hits: every span of a trace must reach the same Collector instance, or each instance decides on a partial trace. The fix is a two-tier setup: a first tier that routes by trace ID using the load-balancing exporter, and a second tier that runs tail_sampling.

# Tier 1: route spans so each trace lands on one tier-2 instance
exporters:
loadbalancing:
routing_key: traceID
protocol:
otlp:
tls:
insecure: true
resolver:
dns:
hostname: otel-sampler-headless.observability.svc.cluster.local

Other pitfalls to mention:

  • Memory: num_traces multiplied by spans per trace, held for decision_wait, is real RAM. Size and alert on it.
  • Late spans: a span that arrives after its trace has been decided and evicted from memory (long async work) can be dropped even though the rest of the trace was kept, leaving the trace incomplete. Set decision_wait to cover your normal trace duration, and use the processor's decision cache where your Collector version supports it.
  • Skewed metrics: if you compute RED metrics from spans after tail sampling, error rates look enormous because errors were kept at 100% and successes at 5%. Generate span metrics before the sampler.
  • Scaling the sampler tier reshuffles the trace-to-instance mapping, so expect a short burst of split traces during scale events.
  • Span kind tells the backend the span's role so it can build service maps and pair calls: SERVER (handles an incoming request), CLIENT (makes an outgoing request), PRODUCER (sends a message), CONSUMER (processes a message), INTERNAL (in-process work).
  • Span status is Unset, Ok or Error. The convention is that instrumentation sets Error on failure and otherwise leaves Unset. Ok is reserved for an application explicitly overriding a status. For HTTP server spans, a 4xx is normally not an error (the client made a mistake). For HTTP client spans, a 4xx is an error from the caller's point of view.
  • Span events are timestamped annotations inside a span (exception, retry, cache.miss).
  • Span links connect a span to other span contexts that are related but are not its parent.

Use links when there is no single parent. The classic case is a batch consumer that processes 100 messages produced by 100 different requests. One span cannot have 100 parents, but it can have 100 links:

from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.trace import Link, SpanKind, Status, StatusCode

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("billing-worker")


def produce(message_id: str) -> dict:
"""Each message is produced inside its own trace (e.g. a separate API request)."""
with tracer.start_as_current_span("invoices publish", kind=SpanKind.PRODUCER):
headers: dict[str, str] = {}
inject(headers)
return {"id": message_id, "headers": headers}


def process_batch(messages: list[dict]) -> None:
links = []
for message in messages:
producer_ctx = trace.get_current_span(extract(message["headers"])).get_span_context()
if producer_ctx.is_valid:
links.append(Link(producer_ctx, attributes={"messaging.message.id": message["id"]}))

# One new trace for the batch, linked to every producer trace.
with tracer.start_as_current_span("invoices process", kind=SpanKind.CONSUMER, links=links) as span:
span.set_attribute("messaging.batch.message_count", len(messages))
try:
span.add_event("batch.validated", {"valid": len(messages)})
# ... write invoices ...
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, type(exc).__name__))
raise
print(f"batch span has {len(span.links)} links") # batch span has 3 links


process_batch([produce(f"msg-{i}") for i in range(3)])

Links are also useful for long-running or scheduled work. A nightly job triggered by a request should not stretch that request's trace to eight hours; start a new trace and link back to the trigger.

Q35: Why do spans and log context go missing inside thread pools, and how do you fix it?​

OpenTelemetry's current-span tracking (and the request ID from Q26) lives in contextvars. The rules for when context is copied are subtle:

MechanismCopies the caller's context?
asyncio.create_task / asyncio.gather on coroutinesYes
asyncio.to_threadYes
loop.run_in_executor(...)No
ThreadPoolExecutor.submit / .mapNo
threading.Thread(target=...)No

When context is not copied, a span started in the worker has no parent, so it becomes a new root trace. The request's trace looks incomplete, and orphan single-span traces appear in the backend.

import contextvars
from concurrent.futures import ThreadPoolExecutor

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("report-service")


def fetch_part(part: int) -> str:
with tracer.start_as_current_span(f"fetch part {part}") as span:
return f"part {part}: parent={'OK' if span.parent else 'MISSING (new root trace)'}"


with tracer.start_as_current_span("GET /report"):
with ThreadPoolExecutor(max_workers=4) as pool:
# BROKEN: worker threads start with an empty context.
for line in pool.map(fetch_part, range(2)):
print("broken:", line)

# FIXED: copy the caller's context once per task and run the function inside it.
futures = [pool.submit(contextvars.copy_context().run, fetch_part, i) for i in range(2)]
for future in futures:
print("fixed: ", future.result())

Notes:

  • Call copy_context() once per task. A single Context object cannot be entered by two threads at the same time, so sharing one copy across concurrent tasks raises RuntimeError.
  • In async code, prefer asyncio.to_thread over run_in_executor for this reason.
  • The OpenTelemetry contrib project also ships a threading instrumentation that patches executors and threads to carry context, which is useful when you cannot change every call site.
  • The same bug appears with Celery, RQ or any worker that runs code "somewhere else". There you must propagate explicitly through message headers (see Q42).

Metrics, Alerting and Incident Response​

Q36: Why are average latencies misleading, and why can't you average percentiles?​

Averages hide the tail. If 99 requests take 10 ms and one takes 5 s, the mean is about 60 ms, which looks healthy. Yet 1 in 100 users waited 5 seconds, and a user who makes 50 requests per page view will very likely hit that tail. Use percentiles (p50, p95, p99) and also look at the maximum.

Percentiles cannot be averaged or summed. A p99 is a property of one distribution. The p99s of two pods say nothing reliable about the p99 of the combined traffic, especially when traffic is uneven:

import random
import statistics

random.seed(42)


def p99(samples: list[float]) -> float:
return statistics.quantiles(samples, n=100)[98]


busy_pod = [random.uniform(0.010, 0.050) for _ in range(10_000)] # healthy, high traffic
quiet_pod = [random.uniform(1.0, 3.0) for _ in range(50)] # degraded, low traffic

avg_of_p99s = (p99(busy_pod) + p99(quiet_pod)) / 2
true_p99 = p99(busy_pod + quiet_pod)

print(f"busy pod p99: {p99(busy_pod):.3f}s") # 0.050s
print(f"quiet pod p99: {p99(quiet_pod):.3f}s") # 2.967s
print(f"avg of p99s: {avg_of_p99s:.3f}s") # 1.508s <- what a naive dashboard shows
print(f"true fleet p99: {true_p99:.3f}s") # 0.050s <- the real fleet-wide p99

The averaged number (1.5 s) is wrong in both directions: it overstates the fleet p99 and it hides the fact that one pod is badly degraded. Two correct views exist, and you usually want both:

# CORRECT fleet-wide p99: aggregate the histogram buckets first, then compute the quantile
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# CORRECT per-pod view, to find the outlier
histogram_quantile(0.99, sum by (pod, le) (rate(http_request_duration_seconds_bucket[5m])))

# WRONG: averaging precomputed quantiles (e.g. from a Summary or a per-pod recording rule)
avg(http_request_duration_seconds{quantile="0.99"})

This is the practical reason to prefer histograms over summaries in a distributed system: histogram buckets can be summed across instances and then turned into a quantile, while summary quantiles are computed per process and cannot be aggregated.

Q37: How do you choose histogram buckets, and what are exponential or native histograms?​

histogram_quantile assumes observations are spread evenly within a bucket and interpolates. Your quantiles are therefore only as accurate as your bucket boundaries. If buckets are 0.5, 1, 2.5 and your SLO is "95% under 300 ms", you cannot tell 290 ms from 450 ms.

Guidelines for classic (explicit) buckets:

  • Put a boundary exactly at every SLO threshold (for example 0.3), so "fraction of requests under 300 ms" is an exact count rather than an estimate.
  • Cover the realistic range, from your fastest cache hits up to the client timeout. Anything above the last bucket only lands in +Inf.
  • Remember the cost: every bucket is a separate time series, multiplied by every label combination. Twelve buckets × 50 routes × 5 status codes × 3 methods is 9,000 series for one metric.

Exponential histograms (OpenTelemetry) and native histograms (Prometheus, enabled behind a feature flag in many versions) solve the "choose boundaries up front" problem. Bucket boundaries grow by a fixed ratio and the scale adjusts automatically to the data, which gives good relative accuracy across many orders of magnitude, stored compactly as one series instead of one per bucket.

In the OpenTelemetry SDK, you choose the aggregation with a View:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.metrics.view import (
ExplicitBucketHistogramAggregation,
ExponentialBucketHistogramAggregation,
View,
)

reader = InMemoryMetricReader() # use PeriodicExportingMetricReader(OTLPMetricExporter()) in production
provider = MeterProvider(
metric_readers=[reader],
views=[
# Explicit buckets with a boundary exactly at the 300 ms SLO threshold.
View(
instrument_name="http.server.request.duration",
aggregation=ExplicitBucketHistogramAggregation(
boundaries=(0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.5, 5.0)
),
),
# Exponential buckets: no boundaries to choose, good relative accuracy.
View(
instrument_name="db.client.operation.duration",
aggregation=ExponentialBucketHistogramAggregation(max_size=160),
),
],
)
metrics.set_meter_provider(provider)
meter = metrics.get_meter("checkout")

request_duration = meter.create_histogram("http.server.request.duration", unit="s")
db_duration = meter.create_histogram("db.client.operation.duration", unit="s")

for seconds in (0.12, 0.28, 0.31, 0.9):
request_duration.record(seconds, {"http.route": "/checkout"})
db_duration.record(seconds / 10, {"db.system": "postgresql"})

for scope_metrics in reader.get_metrics_data().resource_metrics[0].scope_metrics:
for metric in scope_metrics.metrics:
print(metric.name, type(metric.data).__name__)
# http.server.request.duration Histogram
# db.client.operation.duration ExponentialHistogram

Check that your backend supports exponential or native histograms end to end before switching. If one hop in the pipeline converts them back to classic buckets, you lose the benefit.

Q38: Symptom-based versus cause-based alerting: which alerts should wake a human?​

  • Symptom-based alerts fire on what users experience: error rate, latency, availability, freshness of data, SLO burn rate.
  • Cause-based alerts fire on internal conditions that might lead to user pain: CPU at 90%, one pod restarted, a replica lagging.

Page a human only when all of these are true:

  1. Users are affected now, or will be soon (for example, an SLO is burning fast, or a disk will be full within hours).
  2. It needs a human action that automation cannot take.
  3. It is urgent: it cannot wait until working hours.

Everything else becomes a ticket (needs action, not urgent) or a dashboard panel (useful context during an incident). Causes still matter. They just go on the dashboards you open after a symptom pages you.

groups:
- name: checkout-alerts
rules:
# PAGE: user-facing symptom, fast error-budget burn on a 99.9% SLO
- alert: CheckoutErrorBudgetFastBurn
expr: |
(
sum(rate(http_requests_total{service="checkout", status=~"5.."}[1h]))
/ sum(rate(http_requests_total{service="checkout"}[1h]))
) > (14.4 * 0.001)
and
(
sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m]))
/ sum(rate(http_requests_total{service="checkout"}[5m]))
) > (14.4 * 0.001)
labels:
severity: page
annotations:
summary: "Checkout is burning its monthly error budget about 14x too fast"
runbook_url: "https://runbooks.example.internal/checkout/error-budget-burn"

# TICKET: a cause that predicts future pain, handled in working hours
- alert: DiskFullWithin24h
expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24 * 3600) < 0
for: 30m
labels:
severity: ticket
annotations:
summary: "Root filesystem on {{ $labels.instance }} is predicted to fill within 24h"

Every page should link a runbook answering: what does this alert mean, how do I confirm user impact, what are the first three mitigation steps, and who do I escalate to. Review pages regularly. Any alert that was not actionable gets fixed, downgraded or deleted.

Q39: Our observability bill is growing faster than our traffic. How do you cut telemetry cost without going blind?​

Find out where the money goes first. Most bills are dominated by one of: log volume, metric cardinality (active series), or trace volume. Then apply the cheapest effective lever for each:

SignalLeverNotes
LogsDrop DEBUG in production; sample repetitive INFO; turn high-volume access logs into metricsKeep 100% of WARN/ERROR and all audit logs
LogsTiered retention: 7 days hot, 30–90 days in cheap object storageRehydrate only during investigations
MetricsRemove unbounded labels; drop unused metrics; recording rules for expensive dashboard queriesAudit with "which series were never queried in 30 days?"
TracesHead sampling for baseline volume plus tail sampling to keep errors and slow tracesGenerate RED metrics before sampling
AllDrop health checks, readiness probes and synthetic noise at the CollectorOften 10–30% of spans on busy services
# Collector: drop noise before it reaches the (paid) backend
processors:
filter/drop-noise:
error_mode: ignore
traces:
span:
- 'attributes["url.path"] == "/healthz"'
- 'attributes["url.path"] == "/readyz"'
logs:
log_record:
# drop DEBUG/TRACE, but keep records whose severity was never set (0)
- 'severity_number < SEVERITY_NUMBER_INFO and severity_number != SEVERITY_NUMBER_UNSPECIFIED'
# Prometheus scrape config: drop whole metric families nobody uses,
# and cap the number of samples a single target may produce
scrape_configs:
- job_name: api
sample_limit: 50000 # the scrape fails instead of silently flooding storage
static_configs:
- targets: ["api:8000"]
metric_relabel_configs:
- source_labels: [__name__]
regex: "python_gc_.*|process_virtual_memory_bytes"
action: drop

The principal-level part of the answer is governance, not just knobs: per-team cost showback, a cardinality budget per service enforced in CI, a default-off policy for DEBUG, and a quarterly review of dashboards and alerts nobody uses. Cost control done blindly removes exactly the data you need in the next incident. Always keep errors, slow traces and SLO metrics at full fidelity.

Q40: Walk me through running an incident and writing a blameless postmortem.​

During the incident, roles matter more than tools:

RoleResponsibility
Incident Commander (IC)Owns the response, sets priorities, makes decisions. Does not debug.
Operations / subject-matter leadsInvestigate and apply mitigations.
Communications leadStatus page, stakeholder and customer updates at a regular cadence.
ScribeTimestamped timeline of observations, decisions and actions.

The flow:

  1. Declare early and set a severity. It is cheap to downgrade and costly to declare late.
  2. Mitigate first, find the root cause later. Roll back, fail over, shed load or disable the feature flag. Restoring service beats understanding it.
  3. Communicate on a fixed cadence (for example, every 30 minutes), even if the update is "no change".
  4. Hand over explicitly when people rotate out; long incidents fail on fatigue.
  5. Close once impact has ended and is confirmed by the SLI, not by "the graph looks better".

After the incident, write a blameless postmortem. Blameless does not mean "no accountability". It means you assume people acted reasonably with the information and tools they had, and ask why the system allowed the mistake. People who fear blame hide information, and then the same incident happens again.

POSTMORTEM: Checkout errors, 2026-03-14 (SEV-2)

Summary : 18% of checkout requests failed for 42 minutes.
Impact : ~9,300 failed checkouts; 31% of monthly error budget consumed.
Detection : Error-budget burn alert, 6 minutes after the deploy. (Could it have been faster?)
Timeline (UTC) : 14:02 deploy v2.31 ... 14:08 page ... 14:21 rollback started ... 14:44 recovered
Root cause(s) : New DB index migration locked the orders table; the migration ran inside the deploy.
Contributing : No lock-timeout on migrations; canary had too little traffic to show it.
What went well : Fast rollback; clear status-page updates.
What went badly: 13 minutes to find the right dashboard.
Action items : [owner, due date, ticket] for each:
- Enforce lock_timeout on all migrations (P1)
- Run schema migrations separately from app deploys (P1)
- Link the checkout dashboard from the alert's runbook (P2)

Good action items are specific, owned, dated and tracked. "Be more careful" is not an action item. Interviewers also listen for contributing factors rather than one root cause: complex incidents rarely have a single cause.


Senior / Principal — Real World Observability Scenarios​

Q41: p99 latency jumps from 120 ms to 3 s, but there are no errors and CPU is fine​

Level: Senior

Scenario: At 10:00 the p99 latency of the orders API climbs from 120 ms to about 3 s. Error rate is flat at 0.1%, CPU sits at 35%, memory is normal, and the database's own query-latency dashboard is green. Traces of slow requests show a large gap between the start of the server span and the first database span, with nothing recorded in between.

Impact: Checkout feels broken for a slice of users and conversion drops, yet no alert fires because every alert is on errors or CPU.

Root Cause: Saturation of an un-instrumented resource. A new report endpoint holds database connections for several seconds. The SQLAlchemy pool (10 connections + 5 overflow) is exhausted, so ordinary requests queue inside the app waiting for a connection. The database is healthy because it only sees the queries that already have a connection. The wait happens before any DB span starts, which explains the "empty gap" in the traces. No error is raised because requests eventually get a connection before pool_timeout.

Fix:

  1. Short term: cap the report endpoint (separate small pool or a read replica), and lower pool_timeout so requests fail fast instead of queueing silently.
  2. Instrument the wait, not just the work: pool checkout time, connections in use, and queue depth for every bounded resource (DB pools, HTTP client pools, worker semaphores, thread pools).
  3. Add a span or span event around connection acquisition so the trace gap gets a name.
  4. Alert on saturation that predicts latency (pool usage above 90% for 5 minutes), not only on errors.
import time
from contextlib import contextmanager

from prometheus_client import Gauge, Histogram
from sqlalchemy import create_engine, text

engine = create_engine(
"postgresql+psycopg://app@orders-db/orders",
pool_size=10,
max_overflow=5,
pool_timeout=2, # fail fast instead of queueing for the default 30 s
)

POOL_IN_USE = Gauge("db_pool_connections_in_use", "Connections currently checked out")
POOL_IN_USE.set_function(lambda: engine.pool.checkedout())

POOL_WAIT = Histogram(
"db_pool_acquire_seconds",
"Time spent waiting to acquire a pooled DB connection",
buckets=(0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0),
)


@contextmanager
def db_connection():
start = time.perf_counter()
with engine.connect() as conn: # blocks here when the pool is exhausted
POOL_WAIT.observe(time.perf_counter() - start)
yield conn


def count_open_orders() -> int:
with db_connection() as conn:
return conn.execute(text("SELECT count(*) FROM orders WHERE status = 'open'")).scalar_one()
# Saturation alert: pool nearly full for 5 minutes (pool_size + max_overflow = 15)
max_over_time(db_pool_connections_in_use[5m]) / 15 > 0.9

# Where does the time go? p99 of waiting for a connection
histogram_quantile(0.99, sum by (le) (rate(db_pool_acquire_seconds_bucket[5m])))

Note that the acquire time also includes opening a brand-new connection when the pool grows, so a few slow samples after a restart are normal.

Workflow Diagram:

p99 up, errors flat, CPU flat
|
v
Open slow traces --> gap BEFORE first DB span (time spent waiting, not working)
|
v
Ask: "What bounded resource could a request be queueing on?"
DB pool? HTTP client pool? thread pool? lock? rate limiter?
|
v
db_pool_connections_in_use pinned at 15/15 <-- saturation found
|
v
Which code holds connections longest? --> new /reports endpoint
|
v
Mitigate (isolate pool, lower pool_timeout) --> add saturation alert + acquire-time metric

Interviewer Notes: The strong candidate recognises that "latency up, errors flat, CPU flat" is the textbook signature of queueing on a saturated resource, which is why USE-style saturation metrics exist. They read the gap in a trace as information. They also mention other usual suspects for this signature: GC pauses, CPU throttling from container limits (CPU usage looks low but cpu_cfs_throttled is high), lock contention, DNS resolution, and a noisy neighbour on shared storage. Continuous profiling with wall-clock (not just CPU) profiles is the tool that finds off-CPU waiting.

Red Flags: "The database is green, so it's not the database." Only alerting on errors and CPU. Raising pool_size without understanding who holds connections, which just moves the queue into the database. Averaging latency across endpoints and missing the problem entirely.


Q42: Traces break in two at the message queue​

Level: Senior

Scenario: An HTTP request creates an order and publishes an order_created event to Kafka. A worker consumes it, charges the card and sends an email. In the tracing backend, the API trace ends at "publish", and the worker's spans appear as hundreds of separate root traces with no connection to the request that caused them. When a customer reports "I paid twice", nobody can follow the request end to end.

Impact: Mean time to resolve for async bugs goes from minutes to hours. Service maps are missing the API → worker edge, and latency SLIs for the full flow ("order placed → confirmation email sent") are impossible to measure.

Root Cause: HTTP auto-instrumentation propagates traceparent through HTTP headers, but nothing writes trace context into the message. Common variants:

  • The worker process never had the messaging instrumentation installed or enabled.
  • A custom serializer wraps the payload and drops the message headers.
  • A transactional outbox stores the event in a database table, and a relay publishes it later from a different process, with a different (empty) context.

Fix: Treat the message as a carrier. Inject on produce, extract on consume, and use the correct span kinds. For the outbox pattern, store the serialized traceparent in an extra outbox column and inject it when the relay publishes.

import queue

from opentelemetry import trace
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
from opentelemetry.trace import SpanKind

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("orders")

broker: queue.Queue[dict] = queue.Queue() # stands in for Kafka/RabbitMQ/SQS


def publish(order_id: str) -> None:
with tracer.start_as_current_span("orders publish", kind=SpanKind.PRODUCER) as span:
span.set_attribute("messaging.destination.name", "orders")
headers: dict[str, str] = {}
inject(headers) # writes traceparent (and baggage) into the message headers
broker.put({"headers": headers, "body": {"order_id": order_id}})


def consume_one() -> None:
message = broker.get()
parent_ctx = extract(message["headers"]) # read them back on the consumer side
with tracer.start_as_current_span("orders process", context=parent_ctx, kind=SpanKind.CONSUMER) as span:
span.set_attribute("order.id", message["body"]["order_id"])
# ... charge card, send email: child spans join the same trace ...


with tracer.start_as_current_span("POST /orders", kind=SpanKind.SERVER):
publish("o-123")
consume_one() # all three spans now share one trace_id

With real brokers, Kafka record headers, RabbitMQ message properties and SQS message attributes are the carriers. OpenTelemetry contrib has instrumentations for common clients (for example Celery, kafka-python, confluent-kafka, pika, aio-pika) that do this for you once they are enabled in both producer and consumer processes.

Workflow Diagram:

BEFORE AFTER
------ -----
[POST /orders]--[publish] [POST /orders]--[publish]
| |
(message, no headers) (message + traceparent header)
| |
[process] <- new root trace [process] <- child of publish, same trace
[charge] [charge]
[email] [email]

Outbox variant:
request txn: INSERT orders + INSERT outbox(payload, traceparent)
relay: SELECT outbox -> inject stored traceparent -> publish

Interviewer Notes: Look for the parent versus link judgement. For one message processed by one consumer, parent-child is natural. For batch consumption, or when the consumer runs hours later, use a span link (Q34) so a single trace does not stretch across hours or have many parents. Also listen for testing: a strong candidate adds an integration test that publishes a message and asserts the consumer span has the producer's trace ID.

Red Flags: Putting the trace ID in the message body and parsing it by hand in every consumer. Expecting auto-instrumentation in the API to fix the worker. Not knowing that outbox relays and retries break context. Long-lived traces that stay "open" for hours.


Q43: The metrics bill jumped 6x after a release that added a user_id label​

Level: Senior / Principal

Scenario: A developer adds user_id and the raw request path as labels on api_requests_total "to debug a customer issue". Within two days the hosted metrics bill rises sixfold, query latency on the main dashboard goes from 2 s to timeouts, and the self-hosted Prometheus replica restarts with out-of-memory errors.

Impact: Direct cost, a degraded monitoring stack in the middle of the business day, and alerts that evaluate late or not at all, which means you are partly blind during any incident.

Root Cause: Cardinality. Each unique label combination is a separate time series. With 400k active users, 30 routes whose raw paths contain IDs, 5 methods and 6 status codes, the theoretical maximum runs into hundreds of millions of series. Even the fraction that actually occurs is orders of magnitude more than before. Hosted vendors typically bill per active series, so cost scales directly with cardinality. There was no guardrail in code review, CI or the scrape config.

Fix:

  1. Stop the bleeding: revert the label or roll back. As an emergency measure at the pipeline, drop the whole metric (action: drop on __name__). Do not simply labeldrop user_id, because removing a label without re-aggregating leaves many series with identical label sets and produces collisions and wrong values.
  2. Fix the instrumentation: use route templates (/orders/{order_id}), and bound every label to a known set.
  3. Move per-user detail to traces and logs, where high cardinality is expected and cheap.
  4. Guardrails: sample_limit per scrape target, a cardinality dashboard, a CI check that fails new labels named like *_id, and an alert on active-series growth.
from prometheus_client import Counter

# VULNERABLE TO CARDINALITY EXPLOSION (do not do this):
# Counter("api_requests_total", "API requests", ["user_id", "path"])
# -> one series per user per raw path, e.g. path="/orders/918273"

KNOWN_METHODS = {"GET", "POST", "PUT", "PATCH", "DELETE"}
KNOWN_PLANS = {"free", "pro", "enterprise"}

API_REQUESTS = Counter(
"api_requests_total",
"API requests",
["route", "method", "status_class", "plan"],
)


def record_request(route_template: str, method: str, status: int, plan: str) -> None:
"""Every label is bounded: templates, enums and buckets only."""
method = method.upper()
API_REQUESTS.labels(
route=route_template, # "/orders/{order_id}", never "/orders/918273"
method=method if method in KNOWN_METHODS else "OTHER",
status_class=f"{status // 100}xx",
plan=plan if plan in KNOWN_PLANS else "other",
).inc()


record_request("/orders/{order_id}", "get", 200, "pro")
record_request("/orders/{order_id}", "PROPFIND", 405, "trial-2026") # -> method="OTHER", plan="other"
# Which metric names own the most series right now?
topk(10, count by (__name__) ({__name__=~".+"}))

# Which label is exploding inside one metric?
count(count by (user_id) (api_requests_total))

# Alert: total head series grew more than 50% in a day (Prometheus self-metric)
prometheus_tsdb_head_series > 1.5 * (prometheus_tsdb_head_series offset 1d)

The first query is itself expensive on a large server; run it against a single Prometheus or use your vendor's cardinality-explorer view.

Workflow Diagram:

Bill / OOM alert
|
v
topk(count by __name__) --> api_requests_total is 85% of series
|
v
count by (label) --> user_id: 400k values, path: 2M values
|
+--> Emergency: revert, or drop the metric at scrape/Collector
|
+--> Fix: route templates + bounded enums; per-user data -> spans/logs
|
+--> Prevent: sample_limit, CI label lint, series-growth alert, team cardinality budget

Interviewer Notes: A principal answer turns this into a platform guardrail rather than a one-off fix: shared instrumentation helpers that only accept route templates, per-team series budgets with showback, and an explanation for developers of where per-user questions should be answered (traces with user.id attribute, or a wide-event store). Bonus for knowing that exemplars (Q25) give "an example slow request" without any per-request label.

Red Flags: "Just add more memory to Prometheus." Using labeldrop without realising it creates duplicate series. Banning all labels instead of bounding them. Not knowing that path (raw URL) is as dangerous as user_id.


Q44: A 3 a.m. alert storm: 180 pages in four minutes​

Level: Senior / Principal

Scenario: At 03:07 the primary Postgres instance fails over to a replica, which takes about 90 seconds. During that window every service that uses the database fires its own alerts: high error rate, high latency, readiness probe failures, queue-backlog growth, synthetic check failures. The on-call engineer's phone receives 180 notifications. The one alert that says what is actually happening, PostgresPrimaryDown, is number 97 in the list.

Impact: The responder spends ten minutes triaging noise instead of confirming the failover. Two other engineers are paged unnecessarily. After weeks of this, the team starts muting the pager, which is how real incidents get missed.

Root Cause:

  • Alerts are cause-based and per-instance, so one root cause fans out into one alert per pod per symptom.
  • Alertmanager is not grouping related alerts and has no inhibition rules that express "if the database is down, do not page for things that depend on it".
  • Severity is not tied to user impact: readiness flaps and queue backlog page at the same level as an SLO burn.

Fix:

  1. Group by service and cluster, so 40 pods give one notification.
  2. Inhibit dependent alerts while a known upstream cause is firing.
  3. Page on symptoms (SLO burn on the user-facing journeys) and route cause-level alerts to a ticket or chat channel.
  4. Add a short for: duration to flap-prone alerts, so a 90-second automatic failover does not page at all if users were not meaningfully affected.
# alertmanager.yml (excerpt)
route:
receiver: team-chat
group_by: ["alertname", "cluster", "service"]
group_wait: 30s # wait to collect related alerts into one notification
group_interval: 5m
repeat_interval: 4h
routes:
- matchers: ['severity="page"']
receiver: oncall-pager

inhibit_rules:
# While the DB primary is down, silence alerts from services that depend on it.
- source_matchers: ['alertname="PostgresPrimaryDown"']
target_matchers: ['depends_on="orders-db"']
equal: ["cluster"]

receivers:
- name: oncall-pager
webhook_configs:
- url: "http://pager-bridge.observability.svc:8080/alerts"
- name: team-chat
webhook_configs:
- url: "http://chat-bridge.observability.svc:8080/alerts"
# Prometheus rule: dependency label lets the inhibit rule above work
- alert: OrdersHighErrorRate
expr: |
sum by (cluster, service) (rate(http_requests_total{service="orders", status=~"5.."}[5m]))
/ sum by (cluster, service) (rate(http_requests_total{service="orders"}[5m])) > 0.05
for: 3m
labels:
severity: ticket
depends_on: orders-db

Workflow Diagram:

DB failover (root cause)
|
+--> PostgresPrimaryDown ------------------------------> PAGE (1 notification)
|
+--> orders errors x40 pods --+
+--> payments latency x25 --+-- depends_on=orders-db --> INHIBITED
+--> readiness flaps x60 --+
+--> queue backlog x30 --+
|
+--> Checkout SLO fast burn? --- only if users really affected ---> PAGE (grouped)

Interviewer Notes: Strong answers mention measuring the pager itself: pages per on-call shift, percentage of pages that were actionable, and time to acknowledge. Google's SRE guidance suggests keeping on-call load to a small number of incidents per shift. Also good: periodic "alert review" meetings where every non-actionable page is deleted or downgraded. Principal candidates discuss dependency modelling (service catalogue → depends_on labels generated automatically) rather than hand-maintained inhibit rules, and caution that over-broad inhibition can hide a second, unrelated failure.

Red Flags: "Just raise all the thresholds." Paging on every pod individually. No distinction between page and ticket. Silencing the noisy alerts permanently instead of fixing why they fire. Not knowing what grouping and inhibition are.


Q45: During the outage, the logs we needed are missing​

Level: Senior

Scenario: The payments service crash-loops for 20 minutes. Afterwards, the log platform shows almost nothing from the crashing pods for that window: no stack trace, no "starting up" lines, nothing from the last seconds before each crash. Logs from healthy services look complete.

Impact: Root cause analysis is guesswork. The postmortem cannot explain why the pods died, so the same failure happens again a week later.

Root Cause: Several independent ways logs vanish at exactly the worst moment:

  1. Buffered stdout. print() output from Python is block-buffered when stdout is not a terminal (as in containers). On a crash, the buffer dies with the process. (The logging module's StreamHandler flushes after each record, so this mostly affects print and custom writers.)
  2. SIGKILL leaves no time to flush. An OOM kill is SIGKILL: no handler, no atexit, no final log line. In-process queues (Q28) and batching exporters lose whatever they held.
  3. Kubelet log rotation. Container logs on the node are rotated by size. A crash-looping pod that logs a huge stack trace in a tight loop can rotate files away before the node's log shipper reads them.
  4. The shipper is overwhelmed. The node agent's memory buffer fills or it hits a rate limit, and it drops data. That happens during incidents, when every service logs more.

Fix:

  • Set PYTHONUNBUFFERED=1 (or run python -u), and use logging rather than print.
  • Handle SIGTERM: flush log queues and telemetry providers before exiting.
  • Do not depend on the app's logs to diagnose how it died. Kubernetes records OOMKilled and exit codes in the pod status (kubectl describe pod, kubectl logs --previous), and kube-state-metrics exports them as metrics.
  • Give the node agent a disk-backed buffer, alert on its drop counters, and size kubelet log rotation for burst volume.
ENV PYTHONUNBUFFERED=1
import logging
import signal
import sys

from opentelemetry import trace

log = logging.getLogger("payments")


def _graceful_shutdown(signum: int, _frame) -> None:
log.warning("received signal %s, flushing telemetry before exit", signal.Signals(signum).name)
provider = trace.get_tracer_provider()
if hasattr(provider, "shutdown"): # SDK TracerProvider: flushes BatchSpanProcessor
provider.shutdown()
logging.shutdown() # flushes and closes every logging handler
sys.exit(0)


signal.signal(signal.SIGTERM, _graceful_shutdown)
# KubeletConfiguration (node level): more room before rotation discards logs
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
containerLogMaxSize: 50Mi
containerLogMaxFiles: 5

Workflow Diagram:

app writes log --> [stdout buffer] --> container runtime --> /var/log/pods/*.log --> node agent --> backend
| | |
lost on crash rotated away buffer full /
(PYTHONUNBUFFERED) before being read rate-limited
(rotation size) (disk buffer +
drop alerts)
Independent evidence that survives the crash:
pod status (OOMKilled, exit code), kubectl logs --previous, kube-state-metrics, node events

Interviewer Notes: The insight to listen for is that your telemetry pipeline fails in correlation with your outages, so you need both pipeline hardening and independent signals (orchestrator state, metrics) that do not depend on the dying process. Strong candidates also suggest a "log pipeline health" dashboard (ingest rate per service, agent drops) and a game-day that kills a pod deliberately to check the last lines arrive.

Red Flags: Assuming the logging vendor lost the data without checking the node. Not knowing that an OOM kill cannot be caught. Adding more logging to a crash-looping service, which makes rotation and agent drops worse.


Q46: The telemetry pipeline itself took production down​

Level: Principal

Scenario: The tracing backend has a partial outage. Within minutes, API latency rises from 80 ms to 10 s and requests begin to time out, even though the API, its database and its dependencies are all healthy. Investigation shows the services were configured with SimpleSpanProcessor, plus a custom logging handler that POSTs every log record to the vendor's HTTP intake, and both run synchronously on the request thread. A separate team's Collector, meanwhile, was OOM-killed repeatedly because it buffered without a memory limit.

Impact: A full customer-facing outage caused by a non-critical dependency. Worse, the outage of the observability backend meant the team had little visibility into the outage it caused.

Root Cause: Telemetry was fail-closed instead of fail-open. Every request waited for an export round trip, and export calls had long or no timeouts. With no bounded buffers and no memory limiter, back-pressure from the backend turned into latency and memory growth in the applications and the Collector.

Fix: Telemetry must never be able to hurt the thing it observes:

  1. Export asynchronously with bounded queues: BatchSpanProcessor, QueueHandler for logs. When the queue is full, drop telemetry; never block requests.
  2. Short timeouts on every exporter.
  3. Export to a local agent (sidecar or DaemonSet) rather than a remote vendor endpoint from each process.
  4. Give the Collector a memory_limiter, horizontal scaling, and a persistent sending queue if losing data during backend outages is unacceptable.
  5. Count what you drop and alert on it, so "fail-open" never silently becomes "no data for a week".
  6. Test it: a chaos experiment that blackholes the telemetry backend should leave application latency unchanged.
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# BEFORE (do not use in production request paths):
# provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint=REMOTE_VENDOR)))
# -> every span.end() blocks the request until the export finishes or times out

exporter = OTLPSpanExporter(
endpoint="http://localhost:4317", # local agent, not a remote vendor endpoint
timeout=5, # seconds; never let one export hang for long
)
processor = BatchSpanProcessor(
exporter,
max_queue_size=4096, # bounded: when full, new spans are DROPPED, requests are not blocked
schedule_delay_millis=2000,
max_export_batch_size=512,
export_timeout_millis=5000,
)
provider = TracerProvider()
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
# Collector: protect itself, and buffer to disk while the backend is down
extensions:
file_storage:
directory: /var/lib/otelcol/queue

processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
spike_limit_percentage: 25
batch: {}

exporters:
otlp:
endpoint: vendor-gateway.example.com:4317
timeout: 10s
sending_queue:
enabled: true
storage: file_storage # persistent queue survives Collector restarts
retry_on_failure:
enabled: true
max_elapsed_time: 10m

service:
extensions: [file_storage]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp]

(The receivers: block is omitted for brevity; it is the same as in Q32.)

Workflow Diagram:

FAIL-CLOSED (before) FAIL-OPEN (after)
request --> handler --> span.end() request --> handler --> span.end()
| |
sync export to vendor enqueue (bounded, O(1))
| |
vendor slow = request slow background thread --> local agent
|
Collector: memory_limiter -> disk queue
|
backend down? retry later / drop + count

Interviewer Notes: A principal candidate frames observability as a dependency with a failure mode and applies the same resilience patterns used for any dependency: bulkheads, timeouts, bounded queues, load shedding. They know the trade-off between dropping telemetry and blocking, and choose dropping, with a metric for dropped data. They also note that "observability of the observability" (Collector internal metrics, synthetic spans that verify end-to-end ingestion) needs an independent alerting path.

Red Flags: SimpleSpanProcessor or synchronous HTTP log handlers in production. Unbounded in-memory buffers. No exporter timeouts. Monitoring the telemetry pipeline only through the same pipeline.


Q47: Pods are OOM-killed every few days and nobody noticed for a month​

Level: Senior

Scenario: A Python API's pods restart roughly every 72 hours. Each restart drops in-flight requests, causing a small burst of 502s that is too short to trip the error-rate alert. The team discovers the pattern only when a customer complains about intermittent failed uploads. Memory graphs show a slow, steady sawtooth: memory climbs linearly for three days, then falls to baseline at each restart.

Impact: Recurring user-facing errors, silent for weeks. Each restart is also a small cold-start latency spike, and the leak means capacity planning numbers are wrong.

Root Cause:

  • The leak: an unbounded module-level cache (a dict keyed by customer and day) that is never evicted.
  • The monitoring gap: no alert on container restarts or OOM kills, and the error-rate alert's for: window was longer than each burst. Restarts "heal" the symptom, so the leak is invisible on short dashboards.

Fix:

  1. Detect: alert on restarts with reason OOMKilled, and use predict_linear to warn before the limit is reached.
  2. Find: compare tracemalloc snapshots, or use a memory profiler (continuous profiling with allocation profiles is ideal), to see which line is accumulating memory.
  3. Fix the code: bound the cache (functools.lru_cache(maxsize=...) or a TTL cache), or move it to Redis with expiry.
  4. Keep the guardrail: a long-window memory panel (7 days) on the service dashboard so sawtooth patterns are visible.
# Restarted because of OOM in the last hour (kube-state-metrics)
increase(kube_pod_container_status_restarts_total[1h]) > 0
and on (namespace, pod, container)
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1

# Predicted to hit the memory limit within 4 hours
max by (namespace, pod, container) (
predict_linear(container_memory_working_set_bytes{container="api"}[6h], 4 * 3600)
)
> on (namespace, pod, container)
max by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory", container="api"})
import tracemalloc

_report_cache: dict[str, bytes] = {} # the bug: an unbounded module-level cache


def render_report(customer_id: int, day: int) -> bytes:
key = f"{customer_id}:{day}"
if key not in _report_cache:
_report_cache[key] = bytes(2_000) # pretend this is a rendered PDF fragment
return _report_cache[key]


tracemalloc.start(10) # keep 10 frames per allocation for useful tracebacks
baseline = tracemalloc.take_snapshot()

for day in range(50): # simulate traffic over time
for customer_id in range(200):
render_report(customer_id, day)

current = tracemalloc.take_snapshot()
for stat in current.compare_to(baseline, "lineno")[:3]:
print(stat)
# leak.py:9: size=19.6 MiB (+19.6 MiB), count=10001 (+10001), average=2054 B <- the cache values
# leak.py:7: size=452 KiB (+452 KiB), count=10000 (+10000), average=46 B <- the cache keys

tracemalloc itself adds memory and CPU overhead, so in production enable it only on one pod, or behind an admin-only switch, for a limited time.

Workflow Diagram:

Weekly view of memory: /|/|/|/| (sawtooth = leak + restart)
|
v
kube_pod_container_status_last_terminated_reason = OOMKilled
|
v
tracemalloc snapshot diff on one pod (or allocation profile)
|
v
Top growth: module-level dict in render_report
|
v
Bound the cache --> deploy --> flat memory line
|
v
Keep alerts: OOMKilled restarts + predict_linear to the limit

Interviewer Notes: Listen for the monitoring lesson as much as the debugging: self-healing hides problems. Restarts, retries and autoscaling all mask regressions, so you monitor the healing mechanisms themselves (restart counts, retry rates, scale events). Good candidates know Python-specific causes: unbounded caches, global lists used as queues, reference cycles holding large objects, C-extension leaks that tracemalloc cannot see (then use RSS-level or native profilers), and memory fragmentation.

Red Flags: "Just raise the memory limit." Scheduling a nightly restart as the permanent fix. Looking only at a 1-hour dashboard window. Not knowing that an OOM kill gives the app no chance to log.


Q48: You are asked to roll out OpenTelemetry across 200 services in two quarters​

Level: Principal

Scenario: The company has 200 services in Python, Java, Go and Node.js. Some use a vendor's proprietary agent, some emit Zipkin B3 headers, and many have no tracing at all. Metrics are Prometheus. Leadership wants "end-to-end tracing everywhere" and to reduce vendor lock-in, without a big-bang migration or an outage.

Impact: Getting this wrong means two half-working systems, broken traces at every boundary between old and new, duplicate bills, and teams who learn to ignore tracing because it never shows the full picture.

Root Cause (of failed attempts elsewhere): Treating it as "install a library in every repo". The hard parts are organisational and at the boundaries: propagation compatibility, consistent naming, a pipeline that can route to old and new backends during the transition, and measuring adoption.

Fix: A phased plan:

  1. Pipeline first. Deploy Collector agents and a gateway. Point the gateway at the current vendor, so nothing changes for users of the old tool. Now every service only needs to speak OTLP to localhost.
  2. Propagation compatibility. Configure services to read and write both W3C and B3 during migration, so mixed old/new hops still join traces.
  3. Golden path. A thin internal package per language that sets resource attributes, propagators, sampler, exporter and log correlation, with pinned semantic-convention versions. Teams import one thing.
  4. Zero-code first, manual later. Start with auto-instrumentation for frameworks, HTTP clients, DB drivers and messaging. Add business spans only on critical journeys.
  5. Measure adoption, not effort. Track "% of requests at the edge whose trace reaches the database tier" and "% of services reporting a correct service.name".
  6. Switch backends last, at the gateway, possibly dual-exporting for a period.
# Python service, zero-code instrumentation
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install # installs instrumentations for libraries it detects

export OTEL_SERVICE_NAME=billing-api
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=payments,deployment.environment.name=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.10
export OTEL_PROPAGATORS=tracecontext,baggage,b3multi # b3multi requires opentelemetry-propagator-b3

opentelemetry-instrument uvicorn billing.app:app --host 0.0.0.0 --port 8000

With pre-fork servers such as Gunicorn, the OpenTelemetry docs recommend initialising the SDK in a post_fork hook, because background exporter threads started before the fork do not survive into the worker processes.

Workflow Diagram:

Phase 1: Collectors everywhere Phase 2: Golden-path SDK Phase 3: Backend choice
------------------------------ ------------------------ -----------------------
services --(old agent)--> vendor services --OTLP--> agent gateway --> new backend
+ agent/gateway deployed -----------> agent --> gateway --> vendor --> gateway --> old vendor (dual,
(no app change yet) propagators: tracecontext+b3 then removed)
adoption dashboard per team

Interviewer Notes: The principal signal is sequencing and risk management: pipeline before SDKs, compatibility before cut-over, adoption metrics, and a reversible backend switch. Strong candidates name the boundary problems: load balancers and API gateways that must forward traceparent, service meshes that generate their own spans, and third-party services that will never propagate (start a new trace and link it). They also budget for sampling and cost from day one, and treat the golden-path library as a product with an owner, versioning and a changelog.

Red Flags: A big-bang cutover. Each team choosing its own attribute names. Ignoring B3 or vendor headers so every mixed hop breaks the trace. Measuring success as "number of repos with the OTel dependency added".


Q49: The error budget is exhausted mid-quarter, but product wants to ship a major feature​

Level: Principal

Scenario: The checkout service has a 99.9% availability SLO over 30 days. Two incidents plus a slow-burning bug have consumed 127% of the budget only 40% of the way into the window. The product director wants to launch a big new feature next week for a marketing campaign. The SRE team says the error-budget policy requires a feature freeze. Both escalate to you.

Impact: If you ship anyway, you risk more user-facing failures during a high-traffic campaign and you teach everyone that SLOs are decoration. If you freeze rigidly with no discussion, product loses trust in the SLO process and starts routing around it.

Root Cause: The organisation has SLOs but the error-budget policy was never agreed in advance by product and engineering leadership, so every breach becomes a negotiation. The specific reliability problems (two incident causes, one slow bug) have not yet been fixed.

Fix: Use the data and a pre-agreed policy, with an explicit escalation path:

  1. Quantify: budget consumed, burn rate now, and what is causing it. Is the slow burn still ongoing?
  2. Apply the policy: over 100% consumed means reliability work takes priority until the burn is fixed. That is not "no changes forever"; it is "the next work is the fixes".
  3. Offer a risk-managed path: ship the feature behind a flag, dark-launch it, canary to 1% with automatic rollback on SLO burn, after the top reliability fixes land.
  4. Escalate exceptions explicitly: if leadership chooses to launch anyway, record it as a conscious business decision with an owner.
  5. Fix the process: get the error-budget policy signed off by product and engineering leadership before the next quarter.
from dataclasses import dataclass


@dataclass(frozen=True)
class ErrorBudget:
slo_target: float # e.g. 0.999
total_requests: int # in the SLO window so far
failed_requests: int
window_elapsed_fraction: float # 0.5 = halfway through the 30-day window

@property
def allowed_failures(self) -> float:
return (1 - self.slo_target) * self.total_requests

@property
def budget_consumed(self) -> float:
return self.failed_requests / self.allowed_failures if self.allowed_failures else float("inf")

def policy(self) -> str:
if self.budget_consumed >= 1.0:
return "FREEZE: only reliability fixes and security patches ship"
if self.budget_consumed > self.window_elapsed_fraction * 1.5:
return "CAUTION: burning faster than planned; extra review and canaries for risky changes"
return "NORMAL: ship features"


checkout = ErrorBudget(slo_target=0.999, total_requests=48_000_000, failed_requests=61_000, window_elapsed_fraction=0.4)
print(f"budget consumed: {checkout.budget_consumed:.0%}") # budget consumed: 127%
print(checkout.policy()) # FREEZE: only reliability fixes and security patches ship

Workflow Diagram:

Budget > 100% consumed
|
v
Is the burn still ongoing? --yes--> fix it first (top priority, all hands)
|no
v
Policy says: reliability work before features
|
v
Product needs launch? --> risk-reduced path:
| fixes land -> feature flag -> 1% canary -> auto-rollback on burn
v
Leadership overrides? --> documented exception, named owner, post-launch review
|
v
Retro: was the SLO right? was the policy pre-agreed? update both

Interviewer Notes: The best answers are about alignment, not arithmetic. An error budget is a tool for product and engineering to agree how to trade velocity against reliability before there is a conflict. Strong candidates also question the SLO itself: if users did not actually notice, perhaps the SLO is too strict or measures the wrong thing; if they did, the freeze is justified. They distinguish budget burned by the team's own changes from budget burned by a dependency outage, which many policies treat differently.

Red Flags: "The SLO is a hard rule, no discussion." Or the opposite: "Ship it, SLOs are just numbers." Proposing to change the SLO target retroactively to make the problem disappear. No pre-agreed policy and no record of exceptions.


Q50: Intermittent 502s at the load balancer that never appear in application logs​

Level: Senior

Scenario: About 0.05% of requests to an API behind a cloud application load balancer return 502 Bad Gateway. The errors are random, spread across all endpoints, and slightly more frequent at low-traffic hours. The application's logs, traces and metrics show no errors at all, and its own SLI reports 100% success.

Impact: Real users get random failures (mobile clients often surface them as "something went wrong"). The SLO dashboard says everything is fine, so the issue is dismissed for weeks.

Root Cause: An idle keep-alive timeout mismatch. The load balancer keeps idle connections to the backend open for up to its idle timeout (60 seconds is a common default on AWS ALB). The app server closes idle keep-alive connections much sooner (for example, Uvicorn's default timeout_keep_alive is 5 seconds). Occasionally the load balancer sends a new request on a connection at the same moment the backend closes it. The request never reaches the application, so the application has nothing to log, and the load balancer returns 502. Low traffic makes it more likely, because connections sit idle long enough to hit the backend's timeout.

Fix:

  1. Make the backend's keep-alive timeout longer than the load balancer's idle timeout, for every hop (LB → reverse proxy → app server).
  2. Measure availability at the edge. Use load-balancer metrics and access logs as the source for the availability SLI, since the app cannot count requests it never received.
  3. In load-balancer access logs, the giveaway is an LB status of 502 with no target status code.
# Run Uvicorn with a keep-alive timeout above the load balancer's idle timeout (60 s here)
import uvicorn

if __name__ == "__main__":
uvicorn.run("app:app", host="0.0.0.0", port=8000, timeout_keep_alive=75)
# gunicorn.conf.py, when running Uvicorn workers under Gunicorn
bind = "0.0.0.0:8000"
workers = 4
# Newer Uvicorn releases moved the Gunicorn worker to the separate `uvicorn-worker` package;
# older ones ship it as "uvicorn.workers.UvicornWorker".
worker_class = "uvicorn_worker.UvicornWorker"
keepalive = 75 # seconds; passed to Uvicorn as its keep-alive timeout; must exceed the LB idle timeout
# Edge-based availability SLI: count what the user saw, not what the app saw.
# lb_http_responses_total is a placeholder: substitute the metric your load balancer
# exporter or cloud integration actually provides.
1 - (
sum(rate(lb_http_responses_total{code=~"5.."}[5m]))
/ sum(rate(lb_http_responses_total[5m]))
)

Workflow Diagram:

t=0s LB --request--> app (connection kept alive)
t=5s app: idle 5s >= timeout_keep_alive --> sends FIN
t=5.0s LB (idle timeout 60s) reuses the connection for a NEW request
|
v
request hits a closing socket --> RST / no response
|
v
LB returns 502; app never saw the request --> no app log, no app metric

Fix: app keep-alive (75s) > LB idle timeout (60s) ==> the LB always closes first

Interviewer Notes: This scenario tests two things. First, networking knowledge of connection reuse and who closes first. Second, and more important, the observability lesson: an SLI measured inside the service is blind to failures in front of it. Strong candidates propose measuring at the closest point to the user (load balancer, CDN, or real-user monitoring) and cross-checking app-level and edge-level request counts; a gap between them is a signal in itself.

Red Flags: "No errors in our logs, so it must be the client's network." Adding retries at the client without finding the cause. Lowering the load balancer's idle timeout below the backend's without understanding why it works. Trusting an app-side SLI as the user's experience.