Monitoring & Observability: Grafana, Prometheus, SigNoz, OpenTelemetry (2026 Edition)
Total Questions: 70 Curriculum: Core Fundamentals (100), Intermediate (120), Advanced (80)
Core Fundamentals
Q1: What is Prometheus and how does it work?
Technical Answer: Prometheus is a time-series database designed for monitoring and alerting. It uses a pull-based model where it scrapes metrics from instrumented applications at regular intervals. Data is stored locally and queried using PromQL.
Architecture Overview:
- Prometheus Server: Scrapes and stores metrics
- Client Libraries: Instrument application code
- Alertmanager: Handles alerts
- Push Gateway: For short-lived jobs
Code Example:
from prometheus_client import Counter, Histogram, start_http_server
import time
import random
# Define metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint'])
REQUEST_DURATION = Histogram('http_request_duration_seconds', 'Request duration')
# Instrument code
@REQUEST_DURATION.time()
def handle_request(method, endpoint):
REQUEST_COUNT.labels(method=method, endpoint=endpoint).inc()
return process_request()
# Expose metrics endpoint
if __name__ == '__main__':
start_http_server(8000) # Metrics available at :8000/metrics
while True:
handle_request('GET', '/api/users')
time.sleep(1)
# prometheus.yml scraping config
scrape_configs:
- job_name: "my-app"
static_configs:
- targets: ["localhost:8000"]
scrape_interval: 15s
🔍 Interviewer Notes: Testing understanding of metrics collection and time-series concepts.
❌ Common Mistakes:
- Too many unique label combinations (cardinality explosion)
- Scraping too frequently (resource waste)
- Not securing the metrics endpoint
Q2: What are the core Prometheus metric types?
Technical Answer: Prometheus has four metric types: Counter (monotonically increasing), Gauge (arbitrary values), Histogram (value distributions), and Summary (quantile calculations).
Comparison Table:
| Metric Type | Use Case | Example |
|---|---|---|
| Counter | Events, totals | HTTP requests, errors |
| Gauge | Current state | CPU usage, queue size |
| Histogram | Distributions | Request latencies |
| Summary | Quantiles | Response time p95/p99 |
Code Example:
// Go example showing all types
var (
// Counter - only increases
httpRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total HTTP requests",
},
[]string{"method", "code"},
)
// Gauge - can go up/down
queueSize = prometheus.NewGauge(
prometheus.GaugeOpts{
Name: "queue_size",
Help: "Current queue size",
},
)
// Histogram - distribution buckets
requestDuration = prometheus.NewHistogram(
prometheus.HistogramOpts{
Name: "request_duration_seconds",
Help: "Request duration distribution",
Buckets: prometheus.DefBuckets,
},
)
)
🔍 Interviewer Notes: Testing metric type selection for different scenarios.
❌ Common Mistakes:
- Using Counter when value can decrease
- Too many histogram buckets
- High cardinality labels
Q3: What is PromQL and how to write queries?
Technical Answer: PromQL is Prometheus's query language for time-series data. It supports instant queries, range queries, and aggregations. Understanding selectors, operators, and aggregation functions is essential.
Key Query Patterns:
# Instant query - current value
http_requests_total{job="api"}
# Range query - values over time
rate(http_requests_total[5m])
# Aggregation - sum by label
sum by(method) (rate(http_requests_total[5m]))
# Filtering - specific conditions
http_requests_total{status=~"5.."}
# Rate calculations - per-second average
rate(http_requests_total[1m])
# Percentiles from histogram
histogram_quantile(0.95, rate(request_duration_bucket[5m]))
Common Queries:
# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m])) * 100
# Top endpoints by latency
topk(5, rate(request_duration_sum[5m]) / rate(request_duration_count[5m]))
🔍 Interviewer Notes: Testing ability to write meaningful observability queries.
❌ Common Mistakes:
- Using
increase()instead ofrate()for gauges - Not using proper time ranges
- Complex queries causing performance issues
Q4: How does Grafana work with Prometheus?
Technical Answer: Grafana visualizes Prometheus metrics through dashboards. It connects via Prometheus data source, runs PromQL queries, and renders panels like graphs, tables, and single stats. Grafana also supports alerting through unified alerting.
Dashboard JSON Snippet:
{
"dashboard": {
"title": "API Metrics",
"panels": [
{
"title": "Request Rate",
"type": "graph",
"targets": [
{
"expr": "rate(http_requests_total[5m])",
"legendFormat": "{{method}} {{code}}"
}
]
},
{
"title": "95th Percentile Latency",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.95, rate(request_duration_bucket[5m]))"
}
]
}
]
}
}
Variables Configuration:
# Template variables for dynamic dashboards
variables:
- name: job
label: Service
query: label_values(http_requests_total, job)
- name: interval
label: Refresh Interval
options: [30s, 1m, 5m, 15m]
🔍 Interviewer Notes: Testing dashboard design and variable usage skills.
❌ Common Mistakes:
- Overly complex dashboards
- Not setting proper refresh intervals
- Missing variable defaults
Q5: What is OpenTelemetry and how does it differ from Prometheus?
Technical Answer: OpenTelemetry (OTel) is an observability framework that standardizes telemetry collection (traces, metrics, logs). Unlike Prometheus which focuses on metrics, OTel provides a complete observability stack with vendor-neutral APIs and SDKs.
Key Differences:
| Aspect | OpenTelemetry | Prometheus |
|---|---|---|
| Scope | Traces + Metrics + Logs | Metrics only |
| Protocol | OTLP, Jaeger, Zipkin | Prometheus Protocol |
| Processing | Collector pipeline | Direct scraping |
| Backend | Vendor-neutral | Prometheus ecosystem |
Instrumentation Example:
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Trace setup
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
@tracer.start_as_current_span("process_order")
def process_order(order_id):
span = trace.get_current_span()
span.set_attribute("order.id", order_id)
# Process order logic
return result
# Metrics setup
metrics.set_meter_provider(MeterProvider())
meter = metrics.get_meter(__name__)
request_counter = meter.create_counter(
"http_requests",
description="HTTP request count"
)
🔍 Interviewer Notes: Testing understanding of modern observability stack and vendor neutrality.
❌ Common Mistakes:
- Not using semantic conventions
- Collecting too much data (cost explosion)
- Mixing OTel traces with Prometheus metrics incorrectly
Q6: What is SigNoz and how does it compare to Grafana?
Technical Answer: SigNoz is an open-source observability platform that uses OpenTelemetry as its data source. Unlike Grafana which requires separate backends, SigNoz provides unified storage for traces, metrics, and logs with built-in correlation and alerting.
Architecture Comparison:
# SigNoz architecture
components:
- otel-collector: Receives OTLP data
- query-service: Processes queries
- alertmanager: Handles alerts
- frontend: React UI
- clickhouse: Unified storage
# Grafana typical setup
components:
- prometheus: Metrics storage
- tempo: Traces storage
- loki: Logs storage
- grafana: Visualization
- mimir: Optional metrics scale
SigNoz Setup:
# docker-compose.yml snippet
services:
signoz:
image: signoz/signoz:latest
ports:
- "3301:3301" # UI
environment:
- CLICKHOUSE_URL=clickhouse:8123
otel-collector:
image: otel/opentelemetry-collector
command: ["--config=/etc/otel-collector-config.yaml"]
🔍 Interviewer Notes: Testing modern observability platform knowledge.
❌ Common Mistakes:
- Not understanding unified vs separate backends
- Missing correlation between signals
- Underestimating storage requirements
Q7: How to implement distributed tracing?
Technical Answer: Distributed tracing tracks requests across service boundaries using spans. Each service creates spans that link to parent spans, forming a trace tree. Context propagation through headers maintains trace continuity.
Trace Context Flow:
Client -> Service A -> Service B -> Service C
w3c traceparent header
Implementation Example:
from opentelemetry import trace
from opentelemetry.trace import SpanKind
from opentelemetry.instrumentation.requests import RequestsInstrumentor
tracer = trace.get_tracer(__name__)
def call_downstream_service(data):
# Child span inherits parent context
with tracer.start_as_current_span(
"downstream-service-call",
kind=SpanKind.CLIENT
) as span:
span.set_attribute("http.url", "http://service-b/api")
response = requests.get("http://service-b/api", params=data)
span.set_attribute("http.status_code", response.status_code)
return response.json()
# Context automatically propagated via headers
@tracer.start_as_current_span("main-handler")
def main_request_handler(req):
result = call_downstream_service(req.data)
return result
HTTP Header Propagation:
# Required headers for trace continuity
# traceparent: version-traceid-parentid-flags
# tracestate: optional vendor-specific data
🔍 Interviewer Notes: Testing understanding of trace propagation and span semantics.
❌ Common Mistakes:
- Not propagating context across async boundaries
- Missing span attributes
- Incorrect span kinds (SERVER vs CLIENT)
Q8: What are Service Level Objectives (SLOs) and how to define them?
Technical Answer: SLOs are target values for service reliability measured over time. They define what "good" looks like and drive alerting decisions. Common SLOs include availability, latency percentiles, and error rates.
SLO Definition Process:
slos:
- name: api-availability
description: "API should be available 99.9% of the time"
indicator:
ratio_metric:
numerator:
query: 'sum(increase(http_requests_total{status=~"5.."}[5m]))'
denominator:
query: "sum(increase(http_requests_total[5m]))"
objective:
- value: 0.999
target: 0.95 # 95% of time SLO should be met
alerting:
name: api-burn-rate
ratio_rule:
query: |
sum(increase(http_requests_total{status=~"5.."}[1m]))
/ sum(increase(http_requests_total[1m]))
threshold: 14.4 # ~2% error budget per hour
duration: 1m
Burn Rate Calculation:
# Fast burn rate (1 hour window)
(sum(increase(errors[1m])) / sum(increase(requests[1m]))) * 60
# Slow burn rate (multi-day window)
(sum(increase(errors[1h])) / sum(increase(requests[1h]))) * 24
🔍 Interviewer Notes: Testing reliability engineering concepts.
❌ Common Mistakes:
- Too many SLOs (alert fatigue)
- Unrealistic targets
- Not understanding error budget burn rates
Q9: How to set up alerting best practices?
Technical Answer: Effective alerting requires actionable alerts with low noise. Use multi-window burn rate alerts, proper grouping, and clear runbooks. Avoid paging for non-critical issues and implement progressive notification escalation.
Alert Rule Example:
groups:
- name: api-alerts
rules:
- alert: HighErrorRate
expr: |
# Page immediately on fast burn
(sum(increase(http_requests_total{status=~"5.."}[1m]))
/ sum(increase(http_requests_total[1m]))) > 0.02
for: 2m
labels:
severity: page
team: backend
annotations:
summary: "High error rate in API"
description: "{{ $value }}% error rate over 2 minutes"
runbook: "https://internal/wiki/api-troubleshooting"
- alert: WarningErrorRate
expr: |
# Warning on slow burn (24h window)
(sum(increase(http_requests_total{status=~"5.."}[1h))
/ sum(increase(http_requests_total[1h))) > 0.001
for: 30m
labels:
severity: warning
team: backend
Notification Channels:
receivers:
- name: critical
pagerduty_configs:
- routing_key: PRIMARY_KEY
slack_configs:
- channel: "#incidents"
send_resolved: true
- name: warning
slack_configs:
- channel: "#alerts"
send_resolved: false
🔍 Interviewer Notes: Testing operational maturity and alert design.
❌ Common Mistakes:
- Alerting on symptoms rather than causes
- Noisiness causing alert fatigue
- Missing clear remediation steps
Q10: How to monitor microservices effectively?
Technical Answer: Microservices monitoring requires distributed tracing for request flow, golden signals (latency, traffic, errors, saturation) for health, and service-level dashboards for troubleshooting. Correlation between logs, metrics, and traces is essential.
Golden Signals Dashboard:
# Latency - p95 of request duration
histogram_quantile(0.95,
rate(http_request_duration_seconds_bucket[5m])
)
# Traffic - requests per second
sum(rate(http_requests_total[5m]))
# Errors - rate of 5xx responses
sum(rate(http_requests_total{status=~"5.."}[5m]))
# Saturation - resource utilization
avg(container_cpu_usage_percent) by (service)
Service Mesh Integration:
# Istio metrics for service mesh
prometheus_metrics:
# Request volume by source and destination
istio_requests_total:
query: "sum(rate(istio_requests_total[5m])) by (source_workload, destination_workload)"
# Request duration
istio_request_duration:
query: "histogram_quantile(0.95, sum(rate(istio_request_duration_milliseconds_bucket[5m])) by (le, destination_workload))"
🔍 Interviewer Notes: Testing holistic monitoring approach for distributed systems.
❌ Common Mistakes:
- Only monitoring aggregates (missing outliers)
- No cross-service correlation
- Not instrumenting at service boundaries
Q11: How to implement distributed tracing across microservices?
Technical Answer: Distributed tracing follows requests across service boundaries using trace contexts propagated via HTTP headers. Trace context includes trace ID, span ID, and sampling decisions. Each service creates spans for operations.
OpenTelemetry Collector Setup:
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
memory_limiter:
limit_mib: 400
attributes:
actions:
- key: service.version
value: "1.0.0"
action: insert
exporters:
otlp/jaeger:
endpoint: jaeger-collector:4317
tls:
insecure: true
prometheus:
endpoint: "0.0.0.0:8889"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, attributes]
exporters: [otlp/jaeger]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
Python Service Instrumentation:
from opentelemetry import trace, propagate
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
import requests
# Initialize tracing
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer(__name__)
# Instrument HTTP client
RequestsInstrumentor().instrument()
# Service endpoint
@tracer.start_as_current_span("get_user_profile")
def get_user_profile(user_id):
span = trace.get_current_span()
span.set_attribute("user_id", user_id)
# Make downstream call with trace context
headers = {}
propagate.inject(headers) # Inject trace context
response = requests.get(
f"http://order-service/orders?userId={user_id}",
headers=headers
)
span.set_attribute("order_count", len(response.json()))
return response.json()
# HTTP handler
@tracer.start_as_current_span("http_request")
def handle_request(request):
# Extraction happens automatically with instrumentation
user_data = get_user_profile(request.user_id)
return {"user": user_data}
Jaeger Query Examples:
# Find slow traces
traces{service="order-service", operation="get_orders", duration > 1000}
# Error traces
traces{service="payment-service", error="true"}
# Trace comparison
traces{service="api", http.status_code=~"5.."}
🔍 Interviewer Notes: Testing distributed tracing implementation skills.
❌ Common Mistakes:
- Not propagating trace context across async boundaries
- Too much sampling cost
- Missing database spans
Q12: How to implement alerting with multi-window burn rates?
Technical Answer: Multi-window burn rate alerts detect both fast and slow error budget consumption. Fast burn rates trigger pages (critical), slow burn rates trigger warnings (investigation needed). Multiple windows provide layered defense.
Alert Rules:
groups:
- name: service-slos
rules:
# Fast burn - page immediately (2% error budget per hour)
- alert: ErrorBudgetFastBurn
expr: |
sum(rate(http_requests_total{job="api", status=~"5.."}[1m]))
/
sum(rate(http_requests_total{job="api"}[1m]))
/
0.02 # 2% per hour target
> 14.4 # 2% * 60 minutes / 1 minute = 14.4
for: 1m
labels:
severity: page
team: backend
annotations:
summary: "Fast error budget burn rate detected"
# Medium burn - warning (2% per 6 hours)
- alert: ErrorBudgetMediumBurn
expr: |
sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
/
0.02
> 1.2 # 2% * 360 min / 5 min = 14.4, but we use 6h window
for: 10m
labels:
severity: warning
# Slow burn - investigation (2% per day)
- alert: ErrorBudgetSlowBurn
expr: |
sum(increase(http_requests_total{job="api", status=~"5.."}[1h]))
/
sum(increase(http_requests_total{job="api"}[1h]))
/
0.02
> 1
for: 2h
labels:
severity: info
Burn Rate Calculations:
| Window | Rate | Interpretation |
|---|---|---|
| 1 min | 14.4x | 100% budget in 4 minutes |
| 5 min | 1.2x | 100% budget in 40 minutes |
| 1 hour | 1x | 100% budget in 24 hours |
| 6 hours | 4x | 100% budget in 6 hours |
🔍 Interviewer Notes: Testing SLO design and alert strategy.
❌ Common Mistakes:
- Too many overlapping alerts
- Not tuning for service-specific traffic patterns
- Missing "no data" alerts
Q13: How to correlate logs, metrics, and traces?
Technical Answer: Correlation uses trace IDs propagated across signals. Structured logging with trace context enables linking logs to spans. Metric labels with trace context help identify which requests caused issues.
Correlated Instrumentation:
import logging
import uuid
from opentelemetry import trace, context
# Configure structured logging
class TracingFormatter(logging.Formatter):
def format(self, record):
span = trace.get_current_span()
if span and span.get_span_context().trace_id:
record.trace_id = format(span.get_span_context().trace_id, '032x')
record.span_id = format(span.get_span_context().span_id, '016x')
return super().format(record)
logger = logging.getLogger()
handler = logging.StreamHandler()
handler.setFormatter(TracingFormatter(
'%(asctime)s %(trace_id)s %(span_id)s %(message)s'
))
logger.addHandler(handler)
# Handler with correlation
def handle_request(request):
span = trace.get_current_span()
request_id = str(uuid.uuid4())
logger.info(
"Processing request started",
extra={
"request_id": request_id,
"user_id": request.user_id,
"endpoint": request.path
}
)
span.set_attribute("request_id", request_id)
try:
result = process_request(request)
logger.info("Request completed", extra={"request_id": request_id})
return result
except Exception as e:
logger.error(
"Request failed",
extra={"request_id": request_id, "error": str(e)}
)
raise
Grafana Unified Correlation:
# Find traces for slow requests
topk(10,
rate(http_request_duration_seconds_sum[5m])
/ rate(http_request_duration_seconds_count[5m])
* on(trace_id) group_left(log_line)
(count by(trace_id) (log_entries{level="error"}))
)
🔍 Interviewer Notes: Testing observability signal correlation.
❌ Common Mistakes:
- Missing trace context in logs
- Not standardizing correlation fields
- Expensive correlation queries
Q14: How to implement synthetic monitoring and chaos engineering?
Technical Answer: Synthetic monitoring simulates user journeys to detect issues before real users. Chaos engineering intentionally breaks systems to test resilience. Both require careful design to avoid false alerts and production disruption.
Synthetic Monitoring Script:
import requests
import time
from datetime import datetime
def synthetic_check():
start = time.time()
errors = []
# User journey: Login -> Search -> Add to Cart -> Checkout
try:
# Step 1: Login
session = requests.Session()
login_resp = session.post(
"https://api.example.com/auth/login",
json={"email": "test@example.com", "password": "test123"}
)
if login_resp.status_code != 200:
errors.append(f"Login failed: {login_resp.status_code}")
# Step 2: Search products
search_resp = session.get(
"https://api.example.com/products",
params={"query": "laptop", "limit": 10}
)
if search_resp.status_code != 200:
errors.append(f"Search failed: {search_resp.status_code}")
# Step 3: Check inventory
if search_resp.status_code == 200:
product_id = search_resp.json()[0]["id"]
inventory_resp = session.get(f"https://api.example.com/products/{product_id}")
if inventory_resp.status_code != 200:
errors.append(f"Inventory check failed")
except Exception as e:
errors.append(f"Journey failed: {str(e)}")
duration = time.time() - start
# Report to monitoring
report_metric("synthetic_journey_duration", duration)
if errors:
report_error("synthetic_journey_failed", "\n".join(errors))
return len(errors) == 0
# Run every minute
if __name__ == "__main__":
synthetic_check()
Chaos Experiment:
# chaos-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: HTTPChaos
metadata:
name: api-500-cascade
spec:
mode: one
selector:
namespaces:
- api
labelSelectors:
"app": "payment-service"
target: Request
port: 8080
path: "/process"
method: "POST"
delay: "10ms"
code: 500
duration: "1m"
---
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: network-partition-db
spec:
action: partition
mode: fixed
selector:
namespaces: ["database"]
labelSelectors:
"app": "mongodb"
partition:
- selector:
labelSelectors:
"app": "api"
direction: to
duration: "30s"
🔍 Interviewer Notes: Testing proactive reliability practices.
❌ Common Mistakes:
- Running chaos during peak hours
- Not having rollback plans
- Synthetic checks too simple
Q15: How to build executive dashboards for system health?
Technical Answer: Executive dashboards focus on business outcomes, SLIs/SLOs, and user impact. Avoid technical metrics; show user-perceived reliability, revenue impact, and customer satisfaction. Clear status indicators enable quick decisions.
Dashboard Structure:
{
"title": "Service Health Executive Dashboard",
"panels": [
{
"title": "Overall System Status",
"type": "stat",
"targets": [
{
"expr": "sum(up{job=~\"api|web|worker\"}) / count(up{job=~\"api|web|worker\"}) * 100",
"legendFormat": "Uptime %"
}
],
"thresholds": [
{ "value": 99.9, "color": "green" },
{ "value": 99.5, "color": "yellow" },
{ "value": 0, "color": "red" }
]
},
{
"title": "Customer-Impacting Errors (Last 24h)",
"type": "graph",
"targets": [
{
"expr": "sum(increase(customer_facing_errors_total[24h]))",
"legendFormat": "Errors"
}
]
},
{
"title": "Revenue at Risk",
"type": "gauge",
"targets": [
{
"expr": "sum(revenue_impacting_incidents_active) * 10000",
"legendFormat": "Estimated hourly loss"
}
]
},
{
"title": "SLI Compliance Calendar",
"type": "heatmap",
"targets": [
{
"expr": "100 - (sum(increase(error_budget_burn_rate[30d])) / 100)",
"legendFormat": "SLO % Achieved"
}
]
}
]
}
Alert Summary Panel:
# Active incidents by severity
count by(severity) (alerting_rules_alerts{alertstate="firing"})
# Mean time to resolution
avg(incident_resolution_time_seconds)
# Incident frequency trend
increase(incidents_total[7d])
🔍 Interviewer Notes: Testing stakeholder communication skills.
❌ Common Mistakes:
- Too much technical detail
- No business context
- Overwhelming color use
Senior / Principal — Real World Scenarios
Q16: Alert fatigue — too many alerts, engineers ignore them
Level: Senior
Scenario: PagerDuty fires 200 alerts per night. Most are low-severity or flapping. Engineers start ignoring pages. A real outage at 3 AM is missed because it looked like another noisy alert.
Impact: Missed incidents. Slow response. Burnout. Revenue loss.
Root Cause: Alerts based on static thresholds without symptom-based tuning. No differentiation between "warning" and "critical." No suppression of known flapping or expected maintenance.
Simple explanation: A car alarm goes off every time a leaf lands on the hood. After a week, nobody checks when it really means a break-in.
Technical Answer:
- Use symptom-based alerting (user-visible impact) not resource-based (CPU > 80% alone isn't an alert).
- Separate warning (ticket) from critical (page). Page only on user impact or imminent SLO breach.
- Use alert grouping/dependencies so one root cause produces one page, not 50.
- Silence alerts during known maintenance windows.
- Tune or delete alerts that never lead to action (if nobody acts, remove it).
Code/Syntax Justification:
# Bad: alerts on every CPU spike
ALERT HighCPU
IF node_cpu_seconds_total{mode="system"} > 0.8
FOR 1m
# Good: alerts on user-impacting latency sustained over window
ALERT HighAPILatency
IF histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2
FOR 10m
LABELS { severity: "critical" }
Workflow Diagram:
Metric crosses threshold
|
v
Static threshold alert Symptom-based alert
| |
v v
Page engineer (low value) Evaluate: user impact?
|
+-- no impact --> warning ticket
|
+-- user impact --> page
🔍 Behind the Question: Alert fatigue, symptom-based vs resource-based, severity, alert grouping.
❌ The Red Flags: Paging on CPU/memory alone; no severity levels; no alert deduplication; alerts nobody acts on.
Q17: Silent failure — metrics stop flowing but no alert fires
Level: Principal
Scenario: The Prometheus scraper stops receiving metrics from a service (network change, scrape config typo, service moved). Dashboards show flatlines or "no data." No alert fires because the alert queries return empty (not error).
Impact: Blind spot. A real outage is invisible because metrics are missing.
Root Cause: Absence of data is not treated as an error. Alerts query existing metrics; when metrics disappear, the query returns nothing and the alert condition is never true.
Simple explanation: A security camera stops recording. The screen shows blank. Nobody notices because "nothing is happening" — but nothing is happening because the camera is off.
Technical Answer:
- Use
absent()orabsent_over_time()to alert when a metric disappears. - Use
upmetric (Prometheus scrape target health) to alert when a target stops being scraped. - Use dead man's switch: service sends a periodic heartbeat metric; alert if heartbeat missing.
- Monitor scrape duration and target health with
prometheus_target_interval_length_secondsandscrape_samples_scraped.
Code/Syntax Justification:
# Alert when service stops being scraped
ALERT TargetDown
IF up{job="api"} == 0
FOR 2m
LABELS { severity: "critical" }
# Alert when a critical metric stops being emitted
ALERT MissingRequests
IF absent(rate(http_requests_total[5m]))
FOR 5m
Workflow Diagram:
Service stops emitting metrics
|
v
Without absent(): With absent() / up == 0:
query returns empty query detects absence
alert never fires alert fires -> page
|
v
Blind spot Detected
🔍 Behind the Question: Absent metrics, dead man's switch, scrape target health, observability blind spots.
❌ The Red Flags: No alert on metric absence; not monitoring up; dashboards showing stale data mistaken for healthy.
Q18: Dashboard shows "green" while users report errors — metric coverage gap
Level: Principal
Scenario: Infrastructure metrics (CPU, memory, disk) are all green. Application error rate metric is not instrumented. Users report 500 errors but the dashboard says everything is healthy.
Impact: Incident detected by users, not by monitoring. Mean time to detect (MTTD) is high.
Root Cause: Monitoring covers infrastructure, not application/business metrics. No error rate, no latency percentiles, no business KPI (checkout conversion) instrumented.
Simple explanation: Checking the engine temperature and oil level, but not whether the car is actually moving. The car could be broken down and you wouldn't know from the gauges.
Technical Answer:
- Instrument RED metrics: Rate, Errors, Duration (per endpoint).
- Instrument business KPIs: checkout completion, sign-up conversion, payment success rate.
- Use SLOs/SLIs: define error budget (e.g., 99.9% of requests < 200ms).
- Alert on SLO burn rate (fast burn = page, slow burn = ticket).
- Correlate infrastructure metrics with application metrics in the same dashboard.
Code/Syntax Justification:
# Error rate per endpoint (RED - Errors)
ALERT HighErrorRate
IF sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m])) > 0.01
FOR 5m
# SLO burn rate
ALERT SLOBurnFast
IF sum(rate(slo_burn_rate[30m])) > 14.4 # 1% error budget burned in 30 min
FOR 10m
Workflow Diagram:
Users report 500 errors
|
v
Infra metrics green App metrics instrumented
(CPU, memory OK) (error rate, latency)
| |
v v
"Everything healthy" Alert fires on error rate
before users complain
🔍 Behind the Question: RED metrics, SLO/SLI, MTTD, business KPI instrumentation, metric coverage.
❌ The Red Flags: Only infrastructure metrics; no error rate; no SLO; discovering outages from users.
Q19: Thundering herd on Prometheus — too many scrapes or rules
Level: Senior
Scenario: Prometheus scrapes 2000 targets every 15 seconds. Recording rules with complex window functions run constantly. Prometheus CPU spikes, scrape latency increases, and some scrapes time out.
Impact: Prometheus itself becomes slow. Metric gaps. Alerting delayed.
Root Cause: Too many targets, too frequent scraping, or expensive recording rules (large [] windows, many labels, group_by on high-cardinality labels).
Simple explanation: One person trying to call 2000 friends every 15 seconds. They can't keep up. Calls drop.
Technical Answer:
- Use service discovery instead of static scrape configs; scrape only what's needed.
- Increase scrape interval for non-critical metrics (e.g., 1m instead of 15s).
- Optimize recording rules: pre-aggregate, avoid high-cardinality
group_by, limit window size. - Use recording rules for frequently queried aggregations instead of ad-hoc queries.
- Shard Prometheus by team/environment if scale demands it.
Code/Syntax Justification:
scrape_configs:
- job_name: "api"
scrape_interval: 30s # not 15s unless needed
static_configs:
- targets: ["api:8080"]
# Expensive rule (avoid):
# sum(rate(http_requests_total[5m])) by (method, endpoint, status, instance, pod, node) # too many labels
# Better: pre-aggregate at write time
record: job:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (job, endpoint)
Workflow Diagram:
2000 targets every 15s Optimized:
Prometheus CPU spikes 500 targets every 60s
scrape timeouts recording rules pre-aggregate
| |
v v
Metric gaps Prometheus healthy, timely alerts
🔍 Behind the Question: Scrape efficiency, recording rules, cardinality, Prometheus scaling.
❌ The Red Flags: Scraping every 15s unnecessarily; high-cardinality labels in rules; one Prometheus for everything.
Q20: No correlation between metrics, logs, and traces — slow incident triage
Level: Principal
Scenario: An API latency spike occurs. Metrics show latency up. Logs show errors but no request ID match. Traces exist but no way to link to the metric spike. The team spends 2 hours correlating manually.
Impact: High MTTR. Engineer frustration. Prolonged outage.
Root Cause: No correlation IDs propagated across metrics, logs, and traces. No OpenTelemetry instrumentation. No centralized correlation (trace_id in logs and metrics).
Simple explanation: Three detectives each have a clue (metric, log, trace) but no shared case number. They can't combine their findings.
Technical Answer:
- Instrument with OpenTelemetry: trace_id propagated in HTTP headers (
traceparent). - Include
trace_idin structured logs so logs can be filtered by trace. - Emit custom metrics tagged with relevant dimensions (endpoint, error code) and link to traces via exemplars.
- Use a correlated observability stack (Grafana + Tempo + Loki + Prometheus) with trace_id drill-down.
- Define a service-level correlation standard (all services emit trace_id, span_id).
Code/Syntax Justification:
# OpenTelemetry: propagate trace context
from opentelemetry import trace
from opentelemetry.propagate import inject
headers = {}
inject(headers) # adds traceparent
resp = httpx.get("https://downstream.internal", headers=headers)
# Structured log with trace_id
logger.error("payment failed", extra={"trace_id": trace.get_current_span().get_span_context().trace_id})
Workflow Diagram:
Without correlation: With OpenTelemetry correlation:
metric spike metric spike -> trace_id
logs (no ID) logs filtered by trace_id
traces (no link to metric) traces linked to metric spike
| |
v v
2 hours manual correlation seconds to root cause
🔍 Behind the Question: OpenTelemetry, trace_id propagation, correlated observability, MTTR.
❌ The Red Flags: No trace propagation; logs without trace_id; metrics/logs/traces in separate silos.
Senior / Principal — Observability Real World Scenarios (Continued)
Q21: Metric cardinality explosion — Prometheus OOM
Level: Senior
Scenario: A team instruments an HTTP counter with customer_id as a label. They have 1 million customers. Prometheus creates 1 million time series for that one metric. Prometheus OOM-kills.
Impact: Prometheus down. All monitoring blind. Alerting broken.
Root Cause: High cardinality label (customer_id) creates a time series per unique value. Each unique combination of labels = new time series in memory.
Simple explanation: A librarian creates a new index card for every single book instead of one card per genre. Millions of cards. The filing cabinet overflows.
Technical Answer:
- Never use high-cardinality values (user IDs, order IDs, IP addresses, emails) as Prometheus labels.
- Use recording rules to pre-aggregate high-cardinality metrics.
- Use histograms instead of counters per value.
- Store high-cardinality data in a time-series DB designed for it (e.g., VictoriaMetrics, M3DB) or in logs with trace_id for lookup.
- Monitor
prometheus_tsdb_head_seriesand alert on series count growth.
Code/Syntax Justification:
# Bad: high cardinality label
HTTP_REQUESTS_TOTAL.labels(customer_id=customer_id).inc() # 1M series
# Good: low cardinality, put customer_id in logs/traces
HTTP_REQUESTS_TOTAL.labels(endpoint="/api/order", method="POST").inc()
# Log: {"customer_id": "abc-123", "trace_id": "xyz", "endpoint": "/api/order"}
Workflow Diagram:
High cardinality label: Low cardinality + logs:
customer_id = label (1M values) endpoint = label (10 values)
| |
v v
1M time series in Prometheus 10 time series + customer_id in logs
| |
v v
Prometheus OOM Prometheus healthy, logs searchable by customer_id
🔍 Behind the Question: Prometheus cardinality, label design, recording rules, TSDB limits.
❌ The Red Flags: User IDs/Order IDs as labels; no cardinality monitoring; not using recording rules.
Q22: SLO burn rate — alerting on error budget
Level: Senior
Scenario: A service has an SLO of 99.9% availability. The team alerts on every 5xx error. During a minor blip, they get paged at 3 AM for 0.1% errors — within error budget. Alert fatigue.
Impact: Paged for non-issues. Alert fatigue. Real issues missed.
Root Cause: Alerting on raw error rate instead of error budget burn rate. Not distinguishing between "within budget" and "burning budget too fast."
Simple explanation: You have a $100/week coffee budget. You get alerted every time you spend $2. You should only be alerted if you're spending so fast you'll exceed $100 by week's end.
Technical Answer:
- Define SLI (service level indicator) and SLO (service level objective).
- Calculate error budget = 1 - SLO (e.g., 0.1% errors allowed for 99.9%).
- Alert on burn rate: fast burn (e.g., 14.4x budget in 1 hour = page), slow burn (e.g., 2x budget in 7 days = ticket).
- Use
rate()andavg_over_time()in PromQL to calculate burn rate. - Stop paging on individual errors; page on budget exhaustion trend.
Code/Syntax Justification:
# Fast burn alert: 1% error budget burned in 1 hour (14.4x normal rate)
ALERT SLOBurnFast
IF sum(rate(slo_burn_rate[1h])) > 14.4
FOR 5m
LABELS { severity: "critical" }
# Slow burn alert: 5% budget burned in 7 days
ALERT SLOBurnSlow
IF sum(rate(slo_burn_rate[7d])) > 1.0
FOR 1h
LABELS { severity: "warning" }
Workflow Diagram:
Raw error rate alerting: SLO burn rate alerting:
5xx error occurs 5xx error occurs
| |
v v
Page immediately Calculate burn rate
| |
v v
Alert fatigue (paged for 0.1%) Within budget? -> no alert
Burn rate too high? -> page
🔍 Behind the Question: SLO/SLI, error budget, burn rate, alert fatigue, PromQL rate calculations.
❌ The Red Flags: Paging on individual errors; no SLO defined; not distinguishing fast vs slow burn.
Q23: Logging volume — log explosion and cost
Level: Senior
Scenario: An application logs every request with full request/response body. At 10k RPS, log volume is 500 GB/day. Log storage (Elasticsearch/Splunk) cost is $10k/month. Team can't afford retention.
Impact: High logging cost. Short retention. Can't investigate old incidents.
Root Cause: Logging too much (full bodies, every request). No log sampling or level filtering.
Simple explanation: You photocopy every page of every book you read. You run out of shelf space. You should photocopy only key pages.
Technical Answer:
- Use structured logging (JSON) with appropriate levels (debug/info/warn/error).
- Don't log request/response bodies for every request; sample or log only errors.
- Use log sampling: log 1% of successful requests, 100% of errors.
- Set retention policies (30 days hot, 90 days cold, delete after).
- Use log shipping filters to drop noisy logs before they reach storage.
Code/Syntax Justification:
import logging
import random
logger = logging.getLogger("app")
def handle_request(request):
# Log 1% of successes, 100% of errors
if random.random() < 0.01 or response.status_code >= 400:
logger.info("request", extra={
"method": request.method,
"path": request.path,
"status": response.status_code,
"trace_id": get_trace_id()
})
Workflow Diagram:
Log everything: Log sampling + levels:
500 GB/day 1% of successes + 100% of errors
| |
v v
$10k/month storage cost ~5 GB/day
Short retention Longer retention, lower cost
🔍 Behind the Question: Log volume, structured logging, log sampling, retention, cost optimization.
❌ The Red Flags: Logging full request/response bodies for every request; no log levels; no retention policy.
Q24: Alert routing — wrong team gets paged
Level: Principal
Scenario: A database latency alert pages the frontend team. They can't fix it. The DBA team doesn't get paged. Incident drags on for 2 hours.
Impact: Slow MTTR. Wrong team engaged. Customer impact prolonged.
Root Cause: Alert routing not aligned with ownership. No clear ownership mapping for services/metrics.
Simple explanation: A pipe bursts in the kitchen. You call the electrician. They can't fix it. The plumber isn't called. Water keeps flooding.
Technical Answer:
- Define service ownership (team per service) in a catalog.
- Route alerts by service/team labels in Prometheus Alertmanager.
- Use
routeblocks withmatchandmatch_reon labels (team, service, severity). - Set up escalation rules (primary -> secondary -> manager) with timeouts.
- Test alert routing regularly (game days).
Code/Syntax Justification:
# Alertmanager routing
route:
- match:
team: "backend"
severity: "critical"
receiver: "backend-pager"
repeat_interval: 15m
- match:
team: "frontend"
severity: "critical"
receiver: "frontend-pager"
- match:
service: "database"
receiver: "dba-team"
Workflow Diagram:
Wrong team paged: Correct routing:
DB latency alert DB latency alert (label: team=dba)
| |
v v
Frontend team paged DBA team paged immediately
| |
v v
Frontend can't fix DBA fixes in 30 min
2 hour MTTR 30 min MTTR
🔍 Behind the Question: Alert routing, ownership mapping, Alertmanager routes, escalation, MTTDR.
❌ The Red Flags: All alerts to one team; no service ownership; no escalation rules; not testing routing.
Q25: Dashboard design — too many metrics, no signal
Level: Senior
Scenario: A Grafana dashboard has 50 panels. During an incident, engineers can't find the relevant metric. Too much noise, not enough signal.
Impact: Slow incident investigation. Engineers miss the key metric.
Root Cause: Dashboard built for "everything" not for "incident response." No hierarchy or signal prioritization.
Simple explanation: A cockpit with 500 buttons. During an emergency, the pilot can't find the brake. Needs only the critical buttons front and center.
Technical Answer:
- Use the "Four Golden Signals": latency, traffic, errors, saturation (per service).
- Organize dashboards: overview (top 4 signals) -> service detail -> infrastructure -> dependencies.
- Use status lights (green/yellow/red) for quick health assessment.
- Link dashboards (drill-down) from overview to detail.
- Annotate dashboards with deploy events and incidents.
Code/Syntax Justification:
// Golden signals panel
{
"title": "Request Rate (Traffic)",
"targets": [{"expr": "sum(rate(http_requests_total[5m])) by (service)"}],
"thresholds": [
{"color": "green", "value": null},
{"color": "red", "value": 10000}
]
}
Workflow Diagram:
50-panel dashboard: Four Golden Signals dashboard:
Everything visible Latency | Traffic | Errors | Saturation
| |
v v
Can't find key metric during incident One glance: service healthy or not
Slow investigation Drill-down to detail if needed
🔍 Behind the Question: Four Golden Signals, dashboard hierarchy, signal vs noise, drill-down, MTTDR.
❌ The Red Flags: 50+ panels on one dashboard; no hierarchy; no status lights; everything treated equally.
Q26: Log aggregation — ELK stack performance
Level: Senior
Scenario: An ELK stack (Elasticsearch, Logstash, Kibana) is slow. Queries take minutes. Indexing falls behind. The cluster is red.
Impact: Logs unavailable during incidents. Can't investigate. MTTDR high.
Root Cause: Elasticsearch overloaded. Too many indices, too many shards, insufficient resources, or unbounded log volume.
Simple explanation: A library with 10 million books and no catalog. Finding one book takes hours. The librarian is overwhelmed.
Technical Answer:
- Use ILM (Index Lifecycle Management): hot (fast) -> warm (slower) -> cold (frozen) -> delete.
- Right-size shards (20-50 GB per shard). Too many small shards = overhead.
- Use log sampling to reduce volume.
- Increase Elasticsearch heap (max 50% of RAM, max ~31 GB due to compressed OOPs).
- Consider ClickHouse, VictoriaLogs, or Loki for log-only use cases (cheaper than ES).
Code/Syntax Justification:
// ILM policy
{
"policy": {
"phases": {
"hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "7d" } } },
"delete": { "actions": { "delete": { "min_age": "30d" } } }
}
}
}
Workflow Diagram:
Unmanaged indices: ILM-managed indices:
Indices grow forever Hot (7d) -> Warm -> Cold -> Delete (30d)
| |
v v
Cluster red, slow queries Bounded size, predictable performance
🔍 Behind the Question: ELK performance, ILM, shard sizing, log volume, retention.
❌ The Red Flags: No ILM; too many small shards; unbounded log volume; Elasticsearch heap misconfigured.
Q27: Synthetic monitoring — proactive detection
Level: Senior
Scenario: A checkout flow breaks. Users report it 30 minutes later. The team didn't know until users complained. No synthetic monitoring.
Impact: Customer-impacting outage undetected. Revenue loss. MTTDR starts with user complaint.
Simple explanation: You don't know your store is closed until a customer calls to say the door is locked. A synthetic customer would have found it earlier.
Technical Answer:
- Use synthetic monitoring (Playwright, Puppeteer, Grafana k6) to simulate critical user journeys (login, checkout, search).
- Run from multiple geographic locations and devices.
- Alert on synthetic failure before real users are affected.
- Combine with RUM (Real User Monitoring) for actual user experience.
- Alert on synthetic error rate > threshold (e.g., 5% of synthetic checks fail).
Code/Syntax Justification:
# Playwright synthetic test
async def test_checkout(page):
await page.goto("https://shop.example.com")
await page.click("text=Add to Cart")
await page.click("text=Checkout")
await page.fill("input[name=email]", "test@example.com")
await page.click("text=Place Order")
await expect(page.locator("text=Order Confirmed")).to_be_visible()
Workflow Diagram:
No synthetic: Synthetic monitoring:
Users find the bug Synthetic user runs every 5 min
| |
v v
30 min before team knows Synthetic fails at 2:05 PM
| |
v v
Revenue loss Alert at 2:06 PM, fix by 2:30 PM
🔍 Behind the Question: Synthetic monitoring, proactive detection, user journeys, RUM, MTTDR.
❌ The Red Flags: No synthetic monitoring; relying on user reports; not testing critical journeys.
Q28: Error tracking — Sentry aggregation
Level: Senior
Scenario: An application throws 10,000 errors/hour. Sentry shows 10,000 separate issues. Engineers can't find the root cause. All look unique due to different stack traces.
Impact: Noise. Root cause hidden. Engineers waste time on duplicates.
Root Cause: Errors not aggregated by root cause. Dynamic values (IDs, timestamps) in error messages/fingerprints make each error look unique.
Simple explanation: 100 people report "the door is stuck." Each says it differently ("door 3 stuck," "door #3 won't open"). You think they're 100 different doors.
Technical Answer:
- Use Sentry fingerprinting to group errors by root cause (exclude dynamic values).
- Sanitize error messages (remove IDs, emails, IPs).
- Use
before_sendcallback to modify/filter events. - Aggregate by
transaction+exception type+frame. - Alert on new issue or spike in existing issue, not every event.
Code/Syntax Justification:
import sentry_sdk
def before_send(event, hint):
# Remove dynamic values from fingerprint
if "exception" in event:
for exc in event["exception"]["values"]:
exc["value"] = re.sub(r"\d+", "[ID]", exc["value"])
return event
sentry_sdk.init(before_send=before_send, traces_sample_rate=0.1)
Workflow Diagram:
Unaggregated errors: Aggregated errors:
10,000 separate issues 5 root causes
| |
v v
Engineers overwhelmed One fix addresses 10,000 errors
🔍 Behind the Question: Error tracking, fingerprinting, aggregation, before_send, noise reduction.
❌ The Red Flags: No fingerprinting; dynamic values in error messages; alerting on every event.
Q29: Distributed tracing — missing spans
Level: Principal
Scenario: A distributed trace has gaps. Span A (API gateway) -> Span C (database) but Span B (auth service) is missing. Can't tell where time was spent.
Impact: Incomplete trace. Can't identify slow service. MTTDR high.
Root Cause: Span context not propagated between services. Missing traceparent header. Service not instrumented.
Simple explanation: A relay race. Runner A passes baton to Runner C, but Runner B never got it. The race has a gap.
Technical Answer:
- Use OpenTelemetry SDK in all services. Propagate
traceparent/tracestateheaders. - Instrument all HTTP clients and servers automatically.
- For async message queues (Kafka, SQS), propagate context in message headers.
- Verify with
trace_idcorrelation across services. - Use Jaeger/Tempo/Zipkin to visualize traces and find gaps.
Code/Syntax Justification:
from opentelemetry import propagate
from opentelemetry.trace import get_current_span
import httpx
# Propagate context to downstream service
ctx = propagate.get_current_context()
headers = {}
propagate.inject(headers) # adds traceparent
resp = httpx.get("https://downstream.internal", headers=headers)
Workflow Diagram:
No context propagation: Full context propagation:
API gateway (trace_id: xyz) API gateway (trace_id: xyz)
| |
v v
Auth service (no trace_id) Auth service (trace_id: xyz)
| |
v v
Database (trace_id: xyz) Database (trace_id: xyz)
Gap: can't see auth latency Complete trace: all services visible
🔍 Behind the Question: Distributed tracing, context propagation, traceparent, OpenTelemetry, trace visualization.
❌ The Red Flags: No trace propagation; services not instrumented; missing spans in async queues.
Q30: Log levels — debug in production
Level: Senior
Scenario: A developer leaves logger.debug() statements in production code. Debug logs flood the system at high volume. Log storage cost spikes. Important error logs are buried.
Impact: Log noise. Cost. Important signals buried.
Root Cause: Debug-level logging left enabled in production. No log level governance.
Simple explanation: You leave all the faucets open in your house. Water floods the floor. You can't find the leak.
Technical Answer:
- Set production log level to
infoorwarn.debugonly in development. - Use structured logging so level can be filtered at ingestion.
- Don't use
print()— use logging framework with levels. - Review log levels in code review (no debug in production).
- Use dynamic log level adjustment (can raise to debug temporarily for an incident).
Code/Syntax Justification:
import logging
logging.basicConfig(level=logging.INFO) # production: INFO
logger.debug("detailed debug info") # not shown in production
logger.info("request processed") # shown
logger.error("payment failed") # shown
Workflow Diagram:
Debug in production: Correct log levels:
DEBUG floods logs INFO/WARN/ERROR only
| |
v v
Errors buried in noise Errors visible, cost controlled
🔍 Behind the Question: Log levels, production logging, debug governance, structured logging.
❌ The Red Flags: Debug logging in production; print() instead of logging; no log level review.
Q31: (continued) — Error budget consumption alert
Level: Principal
Scenario: A service consumes its monthly error budget in 3 days. The team doesn't know until month-end. No burn rate alert.
Impact: Error budget exhausted early. No reliability guardrail for the rest of the month.
Simple explanation: You have a $300/month dining budget. You spend it all in 3 days. You don't realize until month-end. No food for the rest of the month.
**Technical Answer:
- Track error budget consumption continuously (not just at month-end).
- Alert when budget burn rate exceeds threshold (e.g., 50% of monthly budget used in 3 days).
- Use error budget policy: if budget exhausted, stop deploying new features (focus on reliability).
- Visualize budget remaining on dashboard (burn-down chart).
- Review burn rate in weekly SRE meeting.
Code/Syntax Justification:
# Error budget consumed: 50% of monthly budget in 3 days (burn rate > expected)
ALERT ErrorBudgetDepletion
IF sum(rate(slo_error_budget_consumed[3d])) > 0.5 / (3 * 86400)
FOR 10m
LABELS { severity: "warning" }
Workflow Diagram:
No budget tracking: Error budget tracking:
Spend budget in 3 days Budget burn rate tracked
| |
v v
No guardrail for rest of month Alert at 50% budget in 3 days
Stop deploying features
🔍 Behind the Question: Error budget, burn rate, reliability guardrail, feature freeze policy.
❌ The Red Flags: No error budget tracking; deploying features when budget exhausted; not reviewing burn rate.
Q32: Metric retention — downsampling and storage cost
Level: Senior
Scenario: Prometheus retains all raw metrics for 1 year. Storage cost is high. Queries over long ranges are slow. The team only needs raw data for 30 days and aggregated for 1 year.
Impact: High storage cost. Slow long-range queries.
Root Cause: No retention/downsampling policy. All data kept at full resolution forever.
Simple explanation: You keep every grocery receipt forever in your wallet. Wallet is too full. You only need last month's receipts for taxes.
Technical Answer:
- Set Prometheus retention (e.g.,
--storage.tsdb.retention.time=30dfor raw data). - Use recording rules to downsample to 5m/1h/1d aggregates for long-term storage.
- Store aggregates in a cheaper TSDB or object storage.
- Use Thanos or Cortex for long-term retention with downsampling.
- Alert on storage usage growth.
Code/Syntax Justification:
# Prometheus recording rule for downsampling
groups:
- name: http_requests_aggregated
rules:
- record: job:http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (job)
- record: job:http_requests:rate1h
expr: sum(rate(http_requests_total[1h])) by (job)
Workflow Diagram:
Raw data forever: Retention + downsampling:
1 year raw metrics Raw: 30 days
| |
v v
High storage cost Aggregated: 1 year (5m/1h/1d)
Slow long-range queries Fast queries, low cost
🔍 Behind the Question: Metric retention, downsampling, recording rules, Thanos/Cortex, storage cost.
❌ The Red Flags: All raw data kept forever; no downsampling; not monitoring storage growth.
Q33: Blackbox exporter — proactive endpoint probing
Level: Senior
Scenario: An API endpoint returns 500 for external users but internal health checks pass (localhost). The team doesn't know until users complain. No external probing.
Impact: Customer-impacting outage undetected internally.
Simple explanation: You test the stove by touching it (it's hot inside). You don't check if the food is actually cooked (external view).
Technical Answer:
- Use Prometheus Blackbox Exporter to probe endpoints from outside the cluster.
- Probe HTTP, TCP, DNS, ICMP, SSL certificate expiry.
- Probe from multiple locations (different VPCs, regions, or external).
- Alert on probe failure or slow response (probe duration).
- Combine with internal health checks (inside view) + blackbox (outside view).
Code/Syntax Justification:
# Blackbox exporter config
modules:
http_probe:
prober: http
timeout: 5s
http:
preferred_ip_protocol: ip4
method: GET
# Prometheus scrape config
scrape_configs:
- job_name: blackbox
metrics_path: /probe
params:
module: [http_probe]
static_configs:
- targets:
- https://api.example.com/health
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox-exporter:9115
Workflow Diagram:
Internal health check only: Internal + Blackbox probing:
localhost:200 OK Internal: 200 OK
| |
v v
External users get 500 External probe: 500 -> alert
| |
v v
Undetected until users complain Detected proactively
🔍 Behind the Question: Blackbox exporter, external probing, inside vs outside view, SSL expiry probing.
❌ The Red Flags: Only internal health checks; no external probing; not probing from multiple locations.
Q34: Log correlation — trace_id in logs
Level: Principal
Scenario: An error occurs in a microservices architecture. Logs from 5 services are in Elasticsearch. Can't tell which log lines belong to the same request. Can't trace the flow.
Impact: Can't reconstruct request flow. Slow debugging.
Root Cause: No trace_id in logs. Logs from different services not correlated.
Simple explanation: 5 people each write notes about a project. No one writes the project ID. You can't assemble the full picture.
Technical Answer:
- Propagate trace_id (from OpenTelemetry/W3C trace context) into all service logs.
- Use structured logging (JSON) with
trace_id,span_id,service_namefields. - Configure log shipper (Fluent Bit, Vector) to add trace_id automatically.
- Query Elasticsearch/Kibana by
trace_idto reconstruct full request flow. - Link logs to traces in Grafana (trace_id -> Jaeger/Tempo).
Code/Syntax Justification:
import structlog
from opentelemetry import trace
logger = structlog.get_logger()
def handle_request():
span = trace.get_current_span()
trace_id = format(span.get_span_context().trace_id, "032x")
logger.error("payment failed", trace_id=trace_id, service="payments")
Workflow Diagram:
Logs without trace_id: Logs with trace_id:
Service A: "error" Service A: trace_id=xyz, "error"
Service B: "error" Service B: trace_id=xyz, "error"
Service C: "error" Service C: trace_id=xyz, "error"
| |
v v
Can't correlate Query: trace_id=xyz -> full flow
🔍 Behind the Question: Log correlation, trace_id, structured logging, Kibana query, trace-log linking.
❌ The Red Flags: No trace_id in logs; unstructured logs; can't correlate across services.
Q35: Recording rules — pre-aggregation for dashboard performance
Level: Senior
Scenario: A Grafana dashboard queries sum(rate(http_requests_total[5m])) by (service) over 30 days. The query scans millions of samples. Dashboard loads in 30 seconds.
Impact: Slow dashboards. Poor user experience. Prometheus load.
Root Cause: Computing aggregations on raw data at query time. No pre-computed recording rules.
Simple explanation: You calculate the sum of every receipt every time you want to know your monthly spending. You should keep a running total.
Technical Answer:
- Use Prometheus recording rules to pre-compute expensive aggregations.
- Store as new metrics (e.g.,
job:http_requests:rate5m). - Dashboard queries the pre-computed metric instead of raw data.
- Schedule recording rules at appropriate intervals (1m, 5m, 15m).
- Monitor rule evaluation duration.
Code/Syntax Justification:
# Recording rule
groups:
- name: compute
rules:
- record: service:http_request_rate:rate5m
expr: sum(rate(http_requests_total[5m])) by (service)
labels:
env: production
Workflow Diagram:
Query-time aggregation: Pre-computed recording rule:
Query scans raw data (30 days) Query reads pre-computed metric
| |
v v
30 second dashboard load < 1 second dashboard load
🔍 Behind the Question: Recording rules, pre-aggregation, dashboard performance, PromQL.
❌ The Red Flags: Expensive queries on raw data at dashboard load; no recording rules; slow dashboards.
Q36: Anomaly detection — baseline vs spike
Level: Principal
Scenario: A metric normally ranges 100-500 RPS. Suddenly spikes to 5000 RPS (flash sale). Static threshold (1000 RPS) alerts every flash sale (false positive). Doesn't alert on subtle degradation (600 RPS when normal is 100).
Impact: False positives during expected events. Missed subtle degradations.
Root Cause: Static threshold doesn't account for expected variation or baseline shifts.
Simple explanation: You set your home thermostat to alert above 75°F. It alerts every summer day (expected). It doesn't alert at 72°F in winter (unexpected for winter).
Technical Answer:
- Use anomaly detection (Holt-Winters, Prophet, or ML-based) to learn baseline.
- Alert on deviation from expected baseline, not absolute value.
- Use seasonality (daily/weekly patterns) in baseline.
- For known events (flash sales), suppress alerts or use higher thresholds.
- Combine static absolute thresholds (hard limits) with anomaly detection (baseline deviation).
Code/Syntax Justification:
# Holt-Winters anomaly detection (Prometheus)
predict_linear(http_requests_total[1h], 3600) # predict next hour
# Compare actual vs predicted
abs(http_requests_total - predict_linear(http_requests_total[6h], 3600)) > threshold
Workflow Diagram:
Static threshold: Anomaly detection:
Alert at 1000 RPS Learn baseline (daily pattern)
| |
v v
Flash sale = false positive Flash sale = expected (no alert)
Subtle degradation = missed 600 RPS at 3 AM = anomaly (alert)
🔍 Behind the Question: Anomaly detection, baseline, seasonality, static vs dynamic thresholds, false positives.
❌ The Red Flags: Static thresholds only; no seasonality; alerting on expected events; missing subtle degradation.
Q37: SLO dashboard — reliability communication
Level: Principal
Scenario: A stakeholder asks "Is the service reliable?" The team shows a CPU dashboard. CPU is fine but error rate is high. Can't answer the question.
Impact: Can't communicate reliability. Misaligned expectations between engineering and business.
Simple explanation: A car's engine temperature is normal but the brakes are broken. Saying "the car is fine" because engine temp is OK is misleading.
**Technical Answer:
- Define SLOs (service level objectives) for user-facing reliability (availability, latency, correctness).
- Create SLO dashboard showing: current compliance, error budget remaining, burn rate.
- Use SLIs (service level indicators) that measure user experience, not just infrastructure.
- Review SLO compliance in weekly SRE meeting.
- Use error budget policy: if budget exhausted, pause feature deployments.
Code/Syntax Justification:
# Availability SLO: successful requests / total requests
slo_availability = sum(rate(http_requests_total{status=~"2.."}[5m])) /
sum(rate(http_requests_total[5m]))
# Error budget remaining
error_budget_remaining = 1 - (1 - slo_availability) / (1 - 0.999)
Workflow Diagram:
CPU dashboard: SLO dashboard:
"Engine temp normal" "99.95% availability (SLO: 99.9%)"
| |
v v
Can't answer reliability question Error budget: 50% remaining
Clear answer to stakeholder
🔍 Behind the Question: SLO/SLI, error budget, reliability communication, user-facing metrics.
❌ The Red Flags: Infrastructure metrics as reliability proxy; no SLO defined; can't answer "is the service reliable?"
Q38 (continued): (next) Q37 — Log sampling
Level: Senior
Scenario: A high-traffic API logs every request. At 50k RPS, log ingestion cost is prohibitive. The team can't afford to keep all logs.
Impact: High logging cost. Can't retain logs long enough for investigation.
Root Cause: No log sampling. Every request logged at full detail.
Simple explanation: You record every conversation you have. Storage is full. You should record only important conversations.
**Technical Answer:
- Use log sampling: log 1% of successful requests, 100% of errors and slow requests (>1s).
- Sample at the application level (before logging) or at the log shipper.
- Keep trace_id in sampled logs for correlation.
- Monitor log volume and cost per GB.
- Use structured logging for efficient parsing and filtering.
Code/Syntax Justification:
import random
import structlog
logger = structlog.get_logger()
def log_request(request, response, duration):
if response.status_code >= 400 or duration > 1.0 or random.random() < 0.01:
logger.info("request", method=request.method, path=request.path,
status=response.status_code, duration=duration,
trace_id=get_trace_id())
Workflow Diagram:
Log all requests: Log sampling:
50k RPS x full detail 1% of successes + 100% of errors/slow
| |
v v
Prohibitive cost Manageable cost, key events logged
🔍 Behind the Question: Log sampling, ingestion cost, error/slow priority, structured logging.
❌ The Red Flags: No log sampling; logging every request at full detail; not monitoring log volume.
Q39: (continued): Q38 — Metric naming conventions
Level: Senior
Scenario: Different teams name metrics differently: http_requests, http_requests_total, api_request_count. Can't compare or aggregate across services. Dashboard queries break.
Impact: Inconsistent metrics. Can't build cross-service dashboards. Query errors.
Simple explanation: One team calls it "soda," another calls it "pop," another calls it "soft drink." You can't order the same thing consistently.
**Technical Answer:
- Adopt a metric naming convention:
<namespace>_<subsystem>_<metric>_<unit>(e.g.,http_server_requests_total). - Use Prometheus naming conventions (snake_case,
_totalfor counters,_secondsfor durations). - Document conventions in a shared style guide.
- Enforce with linting (Prometheus lint) or review.
- Use consistent labels:
service,endpoint,method,status_code.
Code/Syntax Justification:
# Good naming convention
http_server_requests_total{service="api", endpoint="/orders", method="POST", status="200"}
http_server_request_duration_seconds{service="api", endpoint="/orders"}
# Bad: inconsistent
api_request_count{}
requests{}
api_latency_ms{}
Workflow Diagram:
Inconsistent naming: Standardized naming:
http_requests{} http_server_requests_total{}
api_request_count{} http_server_requests_total{}
requests{} http_server_requests_total{}
| |
v v
Can't aggregate across services Can aggregate: sum(rate(http_server_requests_total)) by (service)
🔍 Behind the Question: Metric naming, conventions, consistency, Prometheus best practices.
❌ The Red Flags: Inconsistent metric names; no naming convention; labels not standardized.
Q40: (continued): Q40 — Log retention policy
Level: Senior
Scenario: Logs are kept forever. Storage cost grows unbounded. Compliance requires deleting logs after 90 days (GDPR). No retention enforcement.
Impact: High storage cost. Compliance violation. Risk of retaining PII too long.
Root Cause: No log retention policy. Indices never deleted.
Simple explanation: You keep every piece of mail you've ever received. House is full of paper. You should recycle after 90 days.
**Technical Answer:
- Define retention tiers: hot (7 days, fast search), warm (30 days, slower), cold (90 days, frozen), delete after.
- Use ILM (Index Lifecycle Management) in Elasticsearch or retention policies in Loki/CloudWatch.
- Automate deletion (don't rely on manual cleanup).
- Audit deletion logs for compliance.
- Monitor storage cost and retention compliance.
**Code/Syntax Justification:
// Elasticsearch ILM delete after 90 days
{
"policy": {
"phases": {
"delete": { "min_age": "90d", "actions": { "delete": {} } }
}
}
}
Workflow Diagram:
Logs forever: Retention policy:
Storage grows unbounded Hot (7d) -> Warm (30d) -> Cold (90d) -> Delete
| |
v v
Compliance violation Compliant, bounded cost
GDPR risk (PII retained too long) PII deleted on schedule
🔍 Behind the Question: Log retention, ILM, GDPR compliance, storage cost, automated deletion.
❌ The Red Flags: Logs kept forever; no retention enforcement; GDPR violations; manual deletion.
Q41: (continued): Q41 — Error rate SLO vs infrastructure metrics
Level: Principal
Scenario: A stakeholder asks if the service is reliable. The team shows infrastructure metrics (CPU, memory, disk) — all green. But users are seeing 5% error rates. Infrastructure metrics don't reflect user experience.
Impact: Misleading reliability picture. User-impacting issues missed.
Simple explanation: A restaurant's kitchen equipment is all working (fridge, oven, stove). But 5% of meals are wrong. Kitchen metrics say "fine," customers say "broken."
**Technical Answer:
- Define user-facing SLIs: error rate, latency (p95/p99), availability, correctness.
- Alert on user-facing SLIs, not just infrastructure metrics.
- Infrastructure metrics (CPU, memory) are for capacity planning, not reliability SLOs.
- Correlate infrastructure metrics with user-facing SLIs (e.g., high CPU -> high error rate).
- Use RED metrics (Rate, Errors, Duration) for services.
**Code/Syntax Justification:
# User-facing error rate SLI
error_rate = sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))
# Alert on user impact, not CPU
ALERT HighUserErrorRate
IF error_rate > 0.01
FOR 5m
Workflow Diagram:
Infrastructure metrics only: User-facing SLIs:
CPU: 30% (green) Error rate: 5% (red)
Memory: 40% (green) Latency p99: 5s (red)
| |
v v
"System healthy" "Users experiencing errors"
Miss user-impacting issue Clear user impact signal
🔍 Behind the Question: User-facing SLIs vs infrastructure metrics, RED metrics, reliability communication.
❌ The Red Flags: Infrastructure metrics as sole reliability signal; no user-facing SLOs; missing user-impacting issues.
Q42: (continued): Q42 — Trace sampling rate
Level: Senior
Scenario: A service traces 100% of requests. At 50k RPS, trace storage cost is prohibitive. The team can't afford to keep all traces.
Impact: High tracing cost. Can't retain traces for root cause analysis.
Root Cause: 100% trace sampling. No tail-based or rate-based sampling.
Simple explanation: You record every phone call. Storage is full. You should record only important calls (errors, slow calls, random sample).
**Technical Answer:
- Use tail-based sampling: decide to keep or discard a trace after it completes (based on status, duration, errors).
- Keep 100% of errors, 100% of slow requests (>1s), 1% of successful fast requests.
- Use OpenTelemetry tail-based sampling processor.
- Balance: enough traces for debugging, not too many for cost.
- Monitor trace volume and storage cost.
**Code/Syntax Justification:
# OpenTelemetry tail-sampling processor
processors:
tail_sampling:
policies:
- always_on: # keep all errors
- latency_threshold:
threshold: 1000ms # keep slow traces
- probabilistic:
sampling_percentage: 5 # keep 5% of others
Workflow Diagram:
100% trace sampling: Tail-based sampling:
All traces stored Keep all errors + slow + 5% sample
| |
v v
Prohibitive cost Targeted traces, manageable cost
🔍 Behind the Question: Trace sampling, tail-based sampling, sampling probability, tracing cost.
❌ The Red Flags: 100% trace sampling; no tail-based sampling; not monitoring trace volume.
Q43: (continued): Q43 — Correlation ID propagation
Level: Principal
Scenario: A request passes through 7 services. Each service generates its own correlation ID. Can't trace the request end-to-end. Logs from different services can't be joined.
Impact: Can't debug distributed issues. No end-to-end visibility.
Root Cause: Each service generates its own ID instead of propagating the incoming one.
Simple explanation: A package changes tracking numbers at every warehouse. You can't track it from origin to destination.
**Technical Answer:
- Propagate the incoming correlation/trace ID from the entry point through all services.
- Use W3C trace context (
traceparentheader) orX-Request-ID. - If no incoming ID, generate one at the edge (API gateway).
- Log correlation ID in every service log.
- Query logs by correlation ID to reconstruct full request flow.
**Code/Syntax Justification:
from opentelemetry import propagate
from opentelemetry.trace import get_current_span
import httpx
# Propagate incoming context to downstream
headers = {}
propagate.inject(headers) # includes traceparent with trace_id
resp = httpx.get("https://downstream.internal", headers=headers)
# Log with trace_id
trace_id = format(get_current_span().get_span_context().trace_id, "032x")
logger.info("downstream call", trace_id=trace_id, url="https://downstream.internal")
Workflow Diagram:
New ID at each service: Propagated correlation ID:
Service A: id=A Edge: trace_id=xyz
Service B: id=B Service A: trace_id=xyz
Service C: id=C Service B: trace_id=xyz
| |
v v
Can't trace end-to-end Query trace_id=xyz -> full flow
🔍 Behind the Question: Correlation ID propagation, W3C trace context, edge ID generation, log correlation.
❌ The Red Flags: New ID at each service; no edge ID generation; can't correlate across services.
Q44: (continued): Q44 — Dashboard auto-refresh and staleness
Level: Senior
Scenario: A Grafana dashboard shows "green" but data is stale (last updated 2 hours ago). The team thinks the system is healthy but it's actually down. No staleness detection.
Impact: False sense of security. Incidents undetected because dashboard looks normal but is stale.
Simple explanation: A clock stopped at 3:00. You look at it and think it's 3:00. It's actually 6:00. You're late.
**Technical Answer:
- Set dashboard auto-refresh interval (e.g., 30 seconds).
- Use "no data" handling in Grafana (show red/alert when no data).
- Use
absent()orlast()> threshold alerts for staleness detection. - Show "last updated" timestamp prominently on dashboard.
- Alert when critical metrics stop updating (data freshness alert).
**Code/Syntax Justification:
# Staleness alert: metric hasn't updated in 10 minutes
ALERT MetricStale
IF absent(http_requests_total[10m])
FOR 5m
LABELS { severity: "warning" }
Workflow Diagram:
Stale dashboard: Fresh dashboard + staleness alert:
Dashboard shows "green" Dashboard shows "green" + last updated timestamp
| |
v v
System actually down Alert: metric stale for 10 min
False sense of security Detected proactively
🔍 Behind the Question: Dashboard staleness, auto-refresh, absent() alert, data freshness, false sense of security.
❌ The Red Flags: No auto-refresh; no staleness detection; dashboard looks green but data is old.
Q45: (continued): Q45 — Error classification
Level: Senior
Scenario: All 5xx errors are treated the same. A "database connection pool exhausted" (retryable) and a "null pointer exception" (bug) trigger the same alert and response. Engineers waste time on non-actionable errors.
Impact: Alert fatigue. Wasted engineering time. Real bugs missed.
Root Cause: No error classification. All errors grouped together.
Simple explanation: A "check engine" light comes on. It could be a loose gas cap (non-actionable) or engine failure (critical). Same light for both.
**Technical Answer:
- Classify errors by type: retryable (transient), bug (code error), configuration, dependency failure.
- Use error codes or exception types to classify.
- Alert only on actionable errors (bugs, config issues). Retryable errors handled by retry logic.
- Track error rate by classification in dashboard.
- Review error classification in post-mortems.
**Code/Syntax Justification:
# Error classification
class ErrorClassification:
RETRYABLE = "retryable" # DB connection, network timeout
BUG = "bug" # NullPointer, IndexError
CONFIG = "config" # Missing env var, wrong flag
DEPENDENCY = "dependency" # Downstream service error
def classify_error(exc):
if isinstance(exc, (ConnectionError, TimeoutError)):
return ErrorClassification.RETRYABLE
if isinstance(exc, (NullPointerException, IndexError)):
return ErrorClassification.BUG
return ErrorClassification.UNKNOWN
Workflow Diagram:
All errors equal: Classified errors:
DB pool exhausted = alert Retryable -> handled by retry, no alert
Null pointer = alert Bug -> alert + fix
| |
v v
Engineer wakes up for retryable Engineer wakes up for bugs only
error (non-actionable) Alert fatigue reduced
🔍 Behind the Question: Error classification, actionable vs non-actionable alerts, error codes, post-mortem review.
❌ The Red Flags: All errors alert the same; no error classification; retryable errors wake engineers.
Q46: (continued): Q46 — Trace waterfall analysis
Level: Principal
Scenario: A trace shows 10 spans. Span 4 (database query) takes 80% of total time. The team was optimizing span 2 (application logic) which takes 5%. Optimization effort wasted.
Impact: Optimization effort on wrong span. No latency improvement.
Root Cause: Not analyzing trace waterfall. Optimizing based on assumption, not data.
Simple explanation: You optimize your morning routine by brushing teeth faster (2 minutes saved). But your commute is 45 minutes. Wrong optimization target.
**Technical Answer:
- Use trace waterfall visualization (Jaeger/Tempo/Zipkin) to see span durations and dependencies.
- Identify the critical path (longest chain of dependent spans).
- Optimize spans on the critical path, not parallel spans.
- Look for spans with high self-time (not waiting on children).
- Use trace comparison (before/after optimization) to measure improvement.
**Code/Syntax Justification:
# Trace analysis: find critical path
# In Jaeger/Tempo UI: view trace waterfall
# Span 4 (db query) = 4s self-time, 4s total (critical path)
# Span 2 (app logic) = 0.5s self-time (not critical)
Workflow Diagram:
Trace waterfall analysis:
Span 1 (API) -> Span 2 (app logic, 0.5s) -> Span 3 (cache, 0.1s)
-> Span 4 (db query, 4s) <-- CRITICAL PATH
Optimize Span 4 (db query), not Span 2
🔍 Behind the Question: Trace waterfall, critical path, span self-time, optimization targeting.
❌ The Red Flags: Optimizing based on assumption; not viewing trace waterfall; optimizing non-critical spans.
Q47: (continued): Q47 — Log parsing and structured logging
Level: Senior
Scenario: Logs are unstructured text: "Error processing order 123 for customer abc at 2026-01-15 10:30:00". Can't query by order_id, customer, or time range efficiently. Regex parsing is fragile.
Impact: Slow log investigation. Can't filter/aggregate. Regex breaks on format changes.
Simple explanation: You write notes in free-form text. Finding "all notes about order 123" requires reading every note. Structured notes with fields would be searchable.
**Technical Answer:
- Use structured logging (JSON) with consistent fields:
timestamp,level,service,trace_id,order_id,customer_id,message. - Parse at ingestion (Fluent Bit, Vector, Logstash) into structured fields.
- Query by fields:
order_id: "123"instead of regex. - Enforce schema in code review (all logs must be structured).
- Use consistent field names across services.
**Code/Syntax Justification:
# Structured logging (JSON)
logger.info("order processed", {
"order_id": "123",
"customer_id": "abc",
"amount": 99.99,
"status": "success",
"trace_id": "xyz789"
})
# Query: order_id:"123" AND status:"success"
Workflow Diagram:
Unstructured logs: Structured logs:
"Error processing order 123..." {"order_id": "123", "status": "error", ...}
| |
v v
Regex parsing (fragile) Query by field: order_id:"123"
Slow investigation Fast, reliable investigation
🔍 Behind the Question: Structured logging, JSON logs, log parsing, field-based querying, schema enforcement.
❌ The Red Flags: Unstructured text logs; regex parsing; inconsistent field names across services.
Q48: (continued): Q48 — Metric labeling best practices
Level: Senior
Scenario: A team labels metrics with endpoint=/api/users/123 (includes ID). This creates high cardinality. Another team labels with environment=prod-us-east-1a (includes AZ). Both are bad practices.
Impact: Cardinality explosion. Prometheus performance degraded.
Root Cause: Labels used for values that should be in log messages or trace attributes.
Simple explanation: You label every book by its exact page count. Millions of label combinations. You should label by genre, not page count.
**Technical Answer:
- Labels should have low cardinality (limited set of values):
service,environment,endpoint,method,status_code. - Never label with IDs, timestamps, IP addresses, or unbounded strings.
- Use log fields for high-cardinality data (order_id, customer_id).
- Use trace attributes for request-specific data.
- Review label usage in code review.
**Code/Syntax Justification:
# Good labels (low cardinality)
http_requests_total{service="api", endpoint="/api/users", method="GET", status="200"}
# Bad labels (high cardinality)
http_requests_total{service="api", endpoint="/api/users/123", customer_id="abc-456"}
Workflow Diagram:
High-cardinality labels: Low-cardinality labels:
endpoint=/api/users/123 endpoint=/api/users
customer_id=abc-456 status=200
| |
v v
1M time series 10 time series
Prometheus OOM Prometheus healthy
🔍 Behind the Question: Label cardinality, label best practices, high-cardinality values, Prometheus design.
❌ The Red Flags: IDs/timestamps/IPs as labels; unbounded label values; not reviewing label usage.
Q49: (continued): Q49 — SLO burn rate visualization
Level: Principal
Scenario: A stakeholder asks "How much error budget do we have left?" The team doesn't have a visualization. They calculate manually in a spreadsheet. Takes days. Decision delayed.
Impact: Slow reliability decisions. Can't react to budget burn quickly.
Simple explanation: You have a water tank. You don't have a gauge. You have to measure with a cup every day. Slow and inaccurate.
**Technical Answer:
- Create SLO burn-down dashboard: error budget remaining over time.
- Show: budget consumed, budget remaining, burn rate (fast/slow), time to exhaustion.
- Use PromQL recording rules for SLI and error budget.
- Visualize in Grafana with stat panels and time series.
- Review in weekly SRE meeting; link to error budget policy.
**Code/Syntax Justification:
# Error budget remaining (for 99.9% SLO)
error_budget_remaining = 1 - (sum(rate(http_requests_total{status=~"5.."}[30d])) /
sum(rate(http_requests_total[30d]))) / 0.001
Workflow Diagram:
Manual spreadsheet: SLO burn-down dashboard:
Calculate monthly Real-time: budget remaining, burn rate
| |
v v
Decision delayed Immediate visibility
No guardrail Error budget policy enforced
🔍 Behind the Question: SLO visualization, burn-down dashboard, error budget remaining, reliability communication.
❌ The Red Flags: No SLO visualization; manual spreadsheet; no burn rate tracking; delayed decisions.
Q50: (continued): Q50 — Observability maturity assessment
Level: Principal
Scenario: A company wants to improve observability but doesn't know where to start. They have metrics but no traces, logs in silos, no SLOs. Need a roadmap.
Impact: Scattered observability efforts. Wasted investment. No clear priority.
Simple explanation: You want to get fit but don't know your current level. You need an assessment before a training plan.
**Technical Answer:
- Use an observability maturity model:
- Level 1: Metrics only (CPU, memory)
- Level 2: Metrics + Logs (centralized)
- Level 3: Metrics + Logs + Traces (correlated)
- Level 4: SLOs + Error Budgets + Alerting
- Level 5: Proactive (anomaly detection, synthetic, chaos engineering)
- Assess current state per service/team.
- Prioritize: SLOs first (define what reliability means), then traces (distributed systems), then log correlation.
- Create roadmap with quarterly milestones.
- Measure progress: MTTDR, alert fatigue, SLO coverage.
**Code/Syntax Justification:
# Maturity assessment checklist
maturity = {
"metrics": True, # L1
"centralized_logs": True, # L2
"distributed_traces": False, # L3 - gap
"slo_defined": False, # L4 - gap
"error_budget_alerting": False, # L4 - gap
"anomaly_detection": False, # L5 - gap
}
# Priority: define SLOs (L4) before traces (L3) because SLOs define what to trace
Workflow Diagram:
Maturity assessment:
L1 Metrics -> L2 Logs -> L3 Traces -> L4 SLOs -> L5 Proactive
| | | | |
Assess current state -> Identify gaps -> Create roadmap -> Measure progress
🔍 Behind the Question: Observability maturity model, roadmap, prioritization, MTTDR, SLO coverage.
❌ The Red Flags: No maturity assessment; scattered observability; no SLOs; investing in traces before defining reliability goals.
Senior / Principal — Observability Real World Scenarios (Continued)
Q51: SLI/SLO/SLA — choosing the right metric to alert on
Level: Senior
Scenario: A stakeholder asks "Is the service reliable?" The team shows CPU and memory dashboards — all green. But users are seeing 5% error rates. Infrastructure metrics don't reflect user experience.
Impact: Misleading reliability picture. User-impacting issues missed.
Root Cause: Infrastructure metrics (CPU, memory, disk) are for capacity planning, not reliability SLOs. No user-facing SLIs defined.
Simple explanation: A restaurant's kitchen equipment is all working (fridge, oven, stove). But 5% of meals are wrong. Kitchen metrics say "fine," customers say "broken."
Technical Answer:
- Define user-facing SLIs: error rate, latency (p95/p99), availability, correctness.
- Alert on user-facing SLIs, not just infrastructure metrics.
- Infrastructure metrics (CPU, memory) are for capacity planning, not reliability SLOs.
- Correlate infrastructure metrics with user-facing SLIs (e.g., high CPU → high error rate).
- Use RED metrics (Rate, Errors, Duration) for services.
Code/Syntax Justification:
# User-facing error rate SLI
error_rate = sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))
# Alert on user impact, not CPU
ALERT HighUserErrorRate
IF error_rate > 0.01
FOR 5m
Workflow Diagram:
Infrastructure metrics only: User-facing SLIs:
CPU: 30% (green) Error rate: 5% (red)
Memory: 40% (green) Latency p99: 5s (red)
| |
v v
"System healthy" "Users experiencing errors"
Miss user-impacting issue Clear user impact signal
🔍 Behind the Question: User-facing SLIs vs infrastructure metrics, RED metrics, reliability communication.
❌ The Red Flags: Infrastructure metrics as sole reliability signal; no user-facing SLOs; missing user-impacting issues.
Q52: Error budget burn rate — alerting on budget consumption
Level: Principal
Scenario: A service consumes its monthly error budget in 3 days. The team doesn't know until month-end. No burn rate alert.
Impact: Error budget exhausted early. No reliability guardrail for the rest of the month.
Simple explanation: You have a $300/month dining budget. You spend it all in 3 days. You don't realize until month-end. No food for the rest of the month.
Technical Answer:
- Track error budget consumption continuously (not just at month-end).
- Alert when budget burn rate exceeds threshold (e.g., 50% of monthly budget used in 3 days).
- Use error budget policy: if budget exhausted, stop deploying new features (focus on reliability).
- Visualize budget remaining on dashboard (burn-down chart).
- Review burn rate in weekly SRE meeting.
Code/Syntax Justification:
# Error budget consumed: 50% of monthly budget in 3 days (burn rate > expected)
ALERT ErrorBudgetDepletion
IF sum(rate(slo_error_budget_consumed[3d])) > 0.5 / (3 * 86400)
FOR 10m
LABELS { severity: "warning" }
Workflow Diagram:
No budget tracking: Error budget tracking:
Spend budget in 3 days Budget burn rate tracked
| |
v v
No guardrail for rest of month Alert at 50% budget in 3 days
Stop deploying features
🔍 Behind the Question: Error budget, burn rate, reliability guardrail, feature freeze policy.
❌ The Red Flags: No error budget tracking; deploying features when budget exhausted; not reviewing burn rate.
Q53: Alert routing — wrong team gets paged
Level: Principal
Scenario: A database latency alert pages the frontend team. They can't fix it. The DBA team doesn't get paged. Incident drags on for 2 hours.
Impact: Slow MTTDR. Wrong team engaged. Customer impact prolonged.
Root Cause: Alert routing not aligned with ownership. No clear ownership mapping for services/metrics.
Simple explanation: A pipe bursts in the kitchen. You call the electrician. They can't fix it. The plumber isn't called. Water keeps flooding.
Technical Answer:
- Define service ownership (team per service) in a catalog.
- Route alerts by service/team labels in Prometheus Alertmanager.
- Use
routeblocks withmatchandmatch_reon labels (team, service, severity). - Set up escalation rules (primary → secondary → manager) with timeouts.
- Test alert routing regularly (game days).
Code/Syntax Justification:
# Alertmanager routing
route:
- match:
team: "backend"
severity: "critical"
receiver: "backend-pager"
repeat_interval: 15m
- match:
team: "frontend"
severity: "critical"
receiver: "frontend-pager"
- match:
service: "database"
receiver: "dba-team"
Workflow Diagram:
Wrong team paged: Correct routing:
DB latency alert DB latency alert (label: team=dba)
| |
v v
Frontend team paged DBA team paged immediately
| |
v v
Frontend can't fix DBA fixes in 30 min
2 hour MTTR 30 min MTTR
🔍 Behind the Question: Alert routing, ownership mapping, Alertmanager routes, escalation, MTTDR.
❌ The Red Flags: All alerts to one team; no service ownership; no escalation rules; not testing routing.
Q54: Dashboard design — too many metrics, no signal
Level: Senior
Scenario: A Grafana dashboard has 50 panels. During an incident, engineers can't find the relevant metric. Too much noise, not enough signal.
Impact: Slow incident investigation. Engineers miss the key metric.
Root Cause: Dashboard built for "everything" not for "incident response." No hierarchy or signal prioritization.
Simple explanation: A cockpit with 500 buttons. During an emergency, the pilot can't find the brake. Needs only the critical buttons front and center.
Technical Answer:
- Use the Four Golden Signals: latency, traffic, errors, saturation (per service).
- Organize dashboards: overview (top 4 signals) → service detail → infrastructure → dependencies.
- Use status lights (green/yellow/red) for quick health assessment.
- Link dashboards (drill-down) from overview to detail.
- Annotate dashboards with deploy events and incidents.
Code/Syntax Justification:
// Golden signals panel
{
"title": "Request Rate (Traffic)",
"targets": [{"expr": "sum(rate(http_requests_total[5m])) by (service)"}],
"thresholds": [
{"color": "green", "value": null},
{"color": "red", "value": 10000}
]
}
Workflow Diagram:
50-panel dashboard: Four Golden Signals dashboard:
Everything visible Latency | Traffic | Errors | Saturation
| |
v v
Can't find key metric during incident One glance: service healthy or not
Slow investigation Drill-down to detail if needed
🔍 Behind the Question: Four Golden Signals, dashboard hierarchy, signal vs noise, drill-down, MTTDR.
❌ The Red Flags: 50+ panels on one dashboard; no hierarchy; no status lights; everything treated equally.
Q55: Log aggregation — ELK stack performance
Level: Senior
Scenario: An ELK stack (Elasticsearch, Logstash, Kibana) is slow. Queries take minutes. Indexing falls behind. The cluster is red.
Impact: Logs unavailable during incidents. Can't investigate. MTTDR high.
Root Cause: Elasticsearch overloaded. Too many indices, too many shards, insufficient resources, or unbounded log volume.
Simple explanation: A library with 10 million books and no catalog. Finding one book takes hours. The librarian is overwhelmed.
Technical Answer:
- Use ILM (Index Lifecycle Management): hot (fast) → warm (slower) → cold (frozen) → delete.
- Right-size shards (20-50 GB per shard). Too many small shards = overhead.
- Use log sampling to reduce volume.
- Increase Elasticsearch heap (max 50% of RAM, max ~31 GB due to compressed OOPs).
- Consider ClickHouse, VictoriaLogs, or Loki for log-only use cases (cheaper than ES).
Code/Syntax Justification:
// ILM policy
{
"policy": {
"phases": {
"hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "7d" } } },
"delete": { "actions": { "delete": { "min_age": "30d" } } }
}
}
}
Workflow Diagram:
Unmanaged indices: ILM-managed indices:
Indices grow forever Hot (7d) -> Warm -> Cold -> Delete (30d)
| |
v v
Cluster red, slow queries Bounded size, predictable performance
🔍 Behind the Question: ELK performance, ILM, shard sizing, log volume, retention.
❌ The Red Flags: No ILM; too many small shards; unbounded log volume; Elasticsearch heap misconfigured.
Q56: Synthetic monitoring — proactive detection
Level: Senior
Scenario: A checkout flow breaks. Users report it 30 minutes later. The team didn't know until users complained. No synthetic monitoring.
Impact: Customer-impacting outage undetected. Revenue loss. MTTDR starts with user complaint.
Simple explanation: You don't know your store is closed until a customer calls to say the door is locked. A synthetic customer would have found it earlier.
Technical Answer:
- Use synthetic monitoring (Playwright, Puppeteer, Grafana k6) to simulate critical user journeys (login, checkout, search).
- Run from multiple geographic locations and devices.
- Alert on synthetic failure before real users are affected.
- Combine with RUM (Real User Monitoring) for actual user experience.
- Alert on synthetic error rate > threshold (e.g., 5% of synthetic checks fail).
Code/Syntax Justification:
# Playwright synthetic test
async def test_checkout(page):
await page.goto("https://shop.example.com")
await page.click("text=Add to Cart")
await page.click("text=Checkout")
await page.fill("input[name=email]", "test@example.com")
await page.click("text=Place Order")
await expect(page.locator("text=Order Confirmed")).to_be_visible()
Workflow Diagram:
No synthetic: Synthetic monitoring:
Users find the bug Synthetic user runs every 5 min
| |
v v
30 min before team knows Synthetic fails at 2:05 PM
| |
v v
Revenue loss Alert at 2:06 PM, fix by 2:30 PM
🔍 Behind the Question: Synthetic monitoring, proactive detection, user journeys, RUM, MTTDR.
❌ The Red Flags: No synthetic monitoring; relying on user reports; not testing critical journeys.
Q57: Error tracking — Sentry aggregation
Level: Senior
Scenario: An application throws 10,000 errors/hour. Sentry shows 10,000 separate issues. Engineers can't find the root cause. All look unique due to different stack traces.
Impact: Noise. Root cause hidden. Engineers waste time on duplicates.
Root Cause: Errors not aggregated by root cause. Dynamic values (IDs, timestamps) in error messages/fingerprints make each error look unique.
Simple explanation: 100 people report "the door is stuck." Each says it differently ("door 3 stuck," "door #3 won't open"). You think they're 100 different doors.
Technical Answer:
- Use Sentry fingerprinting to group errors by root cause (exclude dynamic values).
- Sanitize error messages (remove IDs, emails, IPs).
- Use
before_sendcallback to modify/filter events. - Aggregate by
transaction+exception type+frame. - Alert on new issue or spike in existing issue, not every event.
Code/Syntax Justification:
import sentry_sdk
def before_send(event, hint):
# Remove dynamic values from fingerprint
if "exception" in event:
for exc in event["exception"]["values"]:
exc["value"] = re.sub(r"\d+", "[ID]", exc["value"])
return event
sentry_sdk.init(before_send=before_send, traces_sample_rate=0.1)
Workflow Diagram:
Unaggregated errors: Aggregated errors:
10,000 separate issues 5 root causes
| |
v v
Engineers overwhelmed One fix addresses 10,000 errors
🔍 Behind the Question: Error tracking, fingerprinting, aggregation, before_send, noise reduction.
❌ The Red Flags: No fingerprinting; dynamic values in error messages; alerting on every event.
Q58: Distributed tracing — missing spans
Level: Principal
Scenario: A distributed trace has gaps. Span A (API gateway) → Span C (database) but Span B (auth service) is missing. Can't tell where time was spent.
Impact: Incomplete trace. Can't identify slow service. MTTDR high.
Root Cause: Span context not propagated between services. Missing traceparent header. Service not instrumented.
Simple explanation: A relay race. Runner A passes baton to Runner C, but Runner B never got it. The race has a gap.
Technical Answer:
- Use OpenTelemetry SDK in all services. Propagate
traceparent/tracestateheaders. - Instrument all HTTP clients and servers automatically.
- For async message queues (Kafka, SQS), propagate context in message headers.
- Verify with
trace_idcorrelation across services. - Use Jaeger/Tempo/Zipkin to visualize traces and find gaps.
Code/Syntax Justification:
from opentelemetry import propagate
from opentelemetry.trace import get_current_span
import httpx
# Propagate context to downstream service
ctx = propagate.get_current_context()
headers = {}
propagate.inject(headers) # adds traceparent
resp = httpx.get("https://downstream.internal", headers=headers)
Workflow Diagram:
No context propagation: Full context propagation:
API gateway (trace_id: xyz) API gateway (trace_id: xyz)
| |
v v
Auth service (no trace_id) Auth service (trace_id: xyz)
| |
v v
Database (trace_id: xyz) Database (trace_id: xyz)
Gap: can't see auth latency Complete trace: all services visible
🔍 Behind the Question: Distributed tracing, context propagation, traceparent, OpenTelemetry, trace visualization.
❌ The Red Flags: No trace propagation; services not instrumented; missing spans in async queues.
Q59: Log levels — debug in production
Level: Senior
Scenario: A developer leaves logger.debug() statements in production code. Debug logs flood the system at high volume. Log storage cost spikes. Important error logs are buried.
Impact: Log noise. Cost. Important signals buried.
Root Cause: Debug-level logging left enabled in production. No log level governance.
Simple explanation: You leave all the faucets open in your house. Water floods the floor. You can't find the leak.
Technical Answer:
- Set production log level to
infoorwarn.debugonly in development. - Use structured logging so level can be filtered at ingestion.
- Don't use
print()— use logging framework with levels. - Review log levels in code review (no debug in production).
- Use dynamic log level adjustment (can raise to debug temporarily for an incident).
Code/Syntax Justification:
import logging
logging.basicConfig(level=logging.INFO) # production: INFO
logger.debug("detailed debug info") # not shown in production
logger.info("request processed") # shown
logger.error("payment failed") # shown
Workflow Diagram:
Debug in production: Correct log levels:
DEBUG floods logs INFO/WARN/ERROR only
| |
v v
Errors buried in noise Errors visible, cost controlled
🔍 Behind the Question: Log levels, production logging, debug governance, structured logging.
❌ The Red Flags: Debug logging in production; print() instead of logging; no log level review.
Q60: Log sampling
Level: Senior
Scenario: A high-traffic API logs every request. At 50k RPS, log ingestion cost is prohibitive. The team can't afford to keep all logs.
Impact: High logging cost. Can't retain logs long enough for investigation.
Root Cause: No log sampling. Every request logged at full detail.
Simple explanation: You record every conversation you have. Storage is full. You should record only important conversations.
Technical Answer:
- Use log sampling: log 1% of successful requests, 100% of errors and slow requests (>1s).
- Sample at the application level (before logging) or at the log shipper.
- Keep trace_id in sampled logs for correlation.
- Monitor log volume and cost per GB.
- Use structured logging for efficient parsing and filtering.
Code/Syntax Justification:
import random
import structlog
logger = structlog.get_logger()
def log_request(request, response, duration):
if response.status_code >= 400 or duration > 1.0 or random.random() < 0.01:
logger.info("request", method=request.method, path=request.path,
status=response.status_code, duration=duration,
trace_id=get_trace_id())
Workflow Diagram:
Log all requests: Log sampling:
50k RPS x full detail 1% of successes + 100% of errors/slow
| |
v v
Prohibitive cost Manageable cost, key events logged
🔍 Behind the Question: Log sampling, ingestion cost, error/slow priority, structured logging.
❌ The Red Flags: No log sampling; logging every request at full detail; not monitoring log volume.
Q61: Metric naming conventions
Level: Senior
Scenario: Different teams name metrics differently: http_requests, http_requests_total, api_request_count. Can't compare or aggregate across services. Dashboard queries break.
Impact: Inconsistent metrics. Can't build cross-service dashboards. Query errors.
Root Cause: Labels used for values that should be in log messages or trace attributes.
Simple explanation: One team calls it "soda," another calls it "pop," another calls it "soft drink." You can't order the same thing consistently.
Technical Answer:
- Adopt a metric naming convention:
<namespace>_<subsystem>_<metric>_<unit>(e.g.,http_server_requests_total). - Use Prometheus naming conventions (snake_case,
_totalfor counters,_secondsfor durations). - Document conventions in a shared style guide.
- Enforce with linting (Prometheus lint) or review.
- Use consistent labels:
service,endpoint,method,status_code.
Code/Syntax Justification:
# Good naming convention
http_requests_total{service="api", endpoint="/users", method="GET", status="200"}
# Bad labels (high cardinality)
http_requests_total{service="api", endpoint="/api/users/123", customer_id="abc-456"}
Workflow Diagram:
High-cardinality labels: Low-cardinality labels:
endpoint=/api/users/123 endpoint=/api/users
customer_id=abc-456 status=200
| |
v v
1M time series 10 time series
Prometheus OOM Prometheus healthy
🔍 Behind the Question: Metric naming, conventions, consistency, Prometheus best practices.
❌ The RedFlags: IDs/timestamps/IPs as labels; unbounded label values; not reviewing label usage.
Q62: Log retention policy
Level: Senior
Scenario: Logs are kept forever. Storage cost grows unbounded. Compliance requires deleting logs after 90 days (GDPR). No retention enforcement.
Impact: High storage cost. Compliance violation. Risk of retaining PII too long.
Root Cause: No log retention policy. Indices never deleted.
Simple explanation: You keep every piece of mail you've ever received. House is full of paper. You should recycle after 90 days.
Technical Answer:
- Define retention tiers: hot (7 days, fast search), warm (30 days, slower), cold (90 days, frozen), delete after.
- Use ILM (Index Lifecycle Management) in Elasticsearch or retention policies in Loki/CloudWatch.
- Automate deletion (don't rely on manual cleanup).
- Audit deletion logs for compliance.
- Monitor storage cost and retention compliance.
Code/Syntax Justification:
// Elasticsearch ILM delete after 90 days
{
"policy": {
"phases": {
"delete": { "min_age": "90d", "actions": { "delete": {} } }
}
}
}
Workflow Diagram:
Logs forever: Retention policy:
Storage grows unbounded Hot (7d) -> Warm (30d) -> Cold (90d) -> Delete
| |
v v
Compliance violation Compliant, bounded cost
GDPR risk (PII retained too long) PII deleted on schedule
🔍 Behind the Question: Log retention, ILM, GDPR compliance, storage cost, automated deletion.
❌ The Red Flags: Logs kept forever; no retention enforcement; GDPR violations; manual deletion.
Q63: Error rate SLO vs infrastructure metrics
Level: Principal
Scenario: A stakeholder asks if the service is reliable. The team shows infrastructure metrics (CPU, memory, disk) — all green. But users are seeing 5% error rates. Infrastructure metrics don't reflect user experience.
Impact: Misleading reliability picture. User-impacting issues missed.
Root Cause: Infrastructure metrics (CPU, memory, disk) are for capacity planning, not reliability SLOs. No user-facing SLIs defined.
Simple explanation: A restaurant's kitchen equipment is all working (fridge, oven, stove). But 5% of meals are wrong. Kitchen metrics say "fine," customers say "broken."
Technical Answer:
- Define user-facing SLIs: error rate, latency (p95/p99), availability, correctness.
- Alert on user-facing SLIs, not just infrastructure metrics.
- Infrastructure metrics (CPU, memory) are for capacity planning, not reliability SLOs.
- Correlate infrastructure metrics with user-facing SLIs (e.g., high CPU → high error rate).
- Use RED metrics (Rate, Errors, Duration) for services.
Code/Syntax Justification:
# User-facing error rate SLI
error_rate = sum(rate(http_requests_total{status=~"5.."}[5m])) /
sum(rate(http_requests_total[5m]))
# Alert on user impact, not CPU
ALERT HighUserErrorRate
IF error_rate > 0.01
FOR 5m
Workflow Diagram:
Infrastructure metrics only: User-facing SLIs:
CPU: 30% (green) Error rate: 5% (red)
Memory: 40% (green) Latency p99: 5s (red)
| |
v v
"System healthy" "Users experiencing errors"
Miss user-impacting issue Clear user impact signal
🔍 Behind the Question: User-facing SLIs vs infrastructure metrics, RED metrics, reliability communication.
❌ The Red Flags: Infrastructure metrics as sole reliability signal; no SLOs defined; missing user-impacting issues.
Q64: Trace sampling rate
Level: Senior
Scenario: A service traces 100% of requests. At 50k RPS, trace storage cost is prohibitive. The team can't afford to keep all traces.
Impact: High tracing cost. Can't retain traces for root cause analysis.
Root Cause: 100% trace sampling. No tail-based or rate-based sampling.
Simple explanation: You record every phone call. Storage is full. You should record only important calls (errors, slow calls, random sample).
Technical Answer:
- Use tail-based sampling: decide to keep or discard a trace after it completes (based on status, duration, errors).
- Keep 100% of errors, 100% of slow requests (>1s), 1% of successful fast requests.
- Use OpenTelemetry tail-based sampling processor.
- Balance: enough traces for debugging, not too many for cost.
- Monitor trace volume and storage cost.
Code/Syntax Justification:
# OpenTelemetry tail-sampling processor
processors:
tail_sampling:
policies:
- always_on: # keep all errors
- latency_threshold:
threshold: 1000ms # keep slow traces
- probabilistic:
sampling_percentage: 5 # keep 5% of others
Workflow Diagram:
100% trace sampling: Tail-based sampling:
All traces stored Keep all errors + slow + 5% sample
| |
v v
Prohibitive cost Targeted traces, manageable cost
🔍 Behind the Question: Trace sampling, tail-based sampling, sampling probability, tracing cost.
❌ The Red Flags: 100% trace sampling; no tail-based sampling; not monitoring trace volume.
Q65: Correlation ID propagation
Level: Principal
Scenario: A request passes through 7 services. Each service generates its own correlation ID. Can't trace the request end-to-end. Logs from different services can't be joined.
Impact: Can't debug distributed issues. No end-to-end visibility.
Root Cause: Each service generates its own ID instead of propagating the incoming one.
Simple explanation: A package changes tracking numbers at every warehouse. You can't track it from origin to destination.
Technical Answer:
- Propagate the incoming correlation/trace ID from the entry point through all services.
- Use W3C trace context (
traceparentheader) orX-Request-ID. - If no incoming ID, generate one at the edge (API gateway).
- Log correlation ID in every service log.
- Query logs by correlation ID to reconstruct full request flow.
Code/Syntax Justification:
from opentelemetry import propagate
from opentelemetry.trace import get_current_span
import httpx
# Propagate incoming context to downstream
headers = {}
propagate.inject(headers) # includes traceparent with trace_id
resp = httpx.get("https://downstream.internal", headers=headers)
# Log with trace_id
trace_id = format(get_current_span().get_span_context().trace_id, "032x")
logger.info("downstream call", trace_id=trace_id, url="https://downstream.internal")
Workflow Diagram:
New ID at each service: Propagated correlation ID:
Service A: id=A Edge: trace_id=xyz
Service B: id=B Service A: trace_id=xyz
Service C: id=C Service B: trace_id=xyz
| |
v v
Can't trace end-to-end Query trace_id=xyz -> full flow
🔍 Behind the Question: Correlation ID propagation, W3C trace context, edge ID generation, log correlation.
❌ The RedFlags: New ID at each service; no edge ID generation; can't correlate across services.
Q66: Dashboard staleness
Level: Senior
Scenario: A Grafana dashboard shows "green" but data is stale (last updated 2 hours ago). The team thinks the system is healthy but it's actually down. No staleness detection.
Impact: False sense of security. Incidents undetected because dashboard looks normal but is stale.
Simple explanation: A clock stopped at 3:00. You look at it and think it's 3:00. It's actually 6:00. You're late.
Technical Answer:
- Set dashboard auto-refresh interval (e.g., 30 seconds).
- Use "no data" handling in Grafana (show red/alert when no data).
- Use
absent()orlast()> threshold alerts for staleness detection. - Show "last updated" timestamp prominently on dashboard.
- Alert when critical metrics stop updating (data freshness alert).
Code/Syntax Justification:
# Staleness alert: metric hasn't updated in 10 minutes
ALERT MetricStale
IF absent(http_requests_total[10m])
FOR 5m
LABELS { severity: "warning" }
Workflow Diagram:
Stale dashboard: Fresh dashboard + staleness alert:
Dashboard shows "green" Dashboard shows "green" + last updated timestamp
| |
v v
System actually down Alert: metric stale for 10 min
False sense of security Detected proactively
🔍 Behind the Question: Dashboard staleness, auto-refresh, absent() alert, data freshness, false sense of security.
❌ The Red Flags: No auto-refresh; no staleness detection; dashboard looks green but data is old.
Q67: Error classification
Level: Senior
Scenario: All 5xx errors are treated the same. A "database connection pool exhausted" (retryable) and a "null pointer exception" (bug) trigger the same alert and response. Engineers waste time on non-actionable errors.
Impact: Alert fatigue. Wasted engineering time. Real bugs missed.
Root Cause: No error classification. All errors grouped together.
Simple explanation: A "check engine" light comes on. It could be a loose gas cap (non-actionable) or engine failure (critical). Same light for both.
Technical Answer:
- Classify errors by type: retryable (transient), bug (code error), configuration, dependency failure.
- Use error codes or exception types to classify.
- Alert only on actionable errors (bugs, config issues). Retryable errors handled by retry logic.
- Track error rate by classification in dashboard.
- Review error classification in post-mortems.
Code/Syntax Justification:
# Error classification
class ErrorClassification:
RETRYABLE = "retryable" # DB connection, network timeout
BUG = "bug" # NullPointer, IndexError
CONFIG = "config" # Missing env var, wrong flag
DEPENDENCY = "dependency" # Downstream service error
def classify_error(exc):
if isinstance(exc, (ConnectionError, TimeoutError)):
return ErrorClassification.RETRYABLE
if isinstance(exc, (NullPointerException, IndexError)):
return ErrorClassification.BUG
return ErrorClassification.UNKNOWN
Workflow Diagram:
All errors equal: Classified errors:
DB pool exhausted = alert Retryable -> handled by retry, no alert
Null pointer = alert Bug -> alert + fix
| |
v v
Engineer wakes up for retryable Engineer wakes up for bugs only
error (non-actionable) Alert fatigue reduced
🔍 Behind the Question: Error classification, actionable vs non-actionable alerts, error codes, post-mortem review.
❌ The Red Flags: All errors alert the same; no error classification; retryable errors wake engineers.
Q68: Trace waterfall analysis
Level: Principal
Scenario: A trace shows 10 spans. Span 4 (database query) takes 80% of total time. The team was optimizing span 2 (application logic) which takes 5%. Optimization effort wasted.
Impact: Optimization effort on wrong span. No latency improvement.
Root Cause: Not analyzing trace waterfall. Optimizing based on assumption, not data.
Simple explanation: You optimize your morning routine by brushing teeth faster (2 minutes saved). But your commute is 45 minutes. Wrong optimization target.
Technical Answer:
- Use trace waterfall visualization (Jaeger/Tempo/Zipkin) to see span durations and dependencies.
- Identify the critical path (longest chain of dependent spans).
- Optimize spans on the critical path, not parallel spans.
- Look for spans with high self-time (not waiting on children).
- Use trace comparison (before/after optimization) to measure improvement.
Code/Syntax Justification:
# Trace analysis: find critical path
# In Jaeger/Tempo UI: view trace waterfall
# Span 4 (db query) = 4s self-time, 4s total (critical path)
# Span 2 (app logic) = 0.5s self-time (not critical)
Workflow Diagram:
Trace waterfall analysis:
Span 1 (API) -> Span 2 (app logic, 0.5s) -> Span 3 (cache, 0.1s)
-> Span 4 (db query, 4s) <-- CRITICAL PATH
Optimize Span 4 (db query), not Span 2
🔍 Behind the Question: Trace waterfall, critical path, span self-time, optimization targeting.
❌ The Red Flags: Optimizing based on assumption; not viewing trace waterfall; optimizing non-critical spans.
Q69: Log parsing and structured logging
Level: Senior
Scenario: Logs are unstructured text: "Error processing order 123 for customer abc at 2026-01-15 10:30:00". Can't query by order_id, customer, or time range efficiently. Regex parsing is fragile.
Impact: Slow log investigation. Can't filter/aggregate. Regex breaks on format changes.
Simple explanation: You write notes in free-form text. Finding "all notes about order 123" requires reading every note. Structured notes with fields would be searchable.
Technical Answer:
- Use structured logging (JSON) with consistent fields:
timestamp,level,service,trace_id,order_id,customer_id,message. - Parse at ingestion (Fluent Bit, Vector, Logstash) into structured fields.
- Query by fields:
order_id: "123"instead of regex. - Enforce schema in code review (all logs must be structured).
- Use consistent field names across services.
Code/Syntax Justification:
# Structured logging (JSON)
logger.info("order processed", {
"order_id": "123",
"customer_id": "abc",
"amount": 99.99,
"status": "success",
"trace_id": "xyz789"
})
# Query: order_id: "123" AND status: "success"
Workflow Diagram:
Unstructured logs: Structured logs:
"Error processing order 123..." {"order_id": "123", "status": "error", ...}
| |
v v
Regex parsing (fragile) Query by field: order_id:"123"
Slow investigation Fast, reliable investigation
🔍 Behind the Question: Structured logging, JSON logs, log parsing, field-based querying, schema enforcement.
❌ The Red Flags: Unstructured text logs; regex parsing; inconsistent field names across services.
Q70: Metric labeling best practices
Level: Senior
Scenario: A team labels metrics with endpoint=/api/users/123 (includes ID). This creates high cardinality. Another team labels with environment=prod-us-east-1a (includes AZ). Both are bad practices.
Impact: Cardinality explosion. Prometheus performance degraded.
Root Cause: Labels used for values that should be in log messages or trace attributes.
Simple explanation: You label every book by its exact page count. Millions of label combinations. You should label by genre, not page count.
Technical Answer:
- Labels should have low cardinality (limited set of values):
service,environment,endpoint,method,status_code. - Never label with IDs, timestamps, IP addresses, or unbounded strings.
- Use log fields for high-cardinality data (order_id, customer_id).
- Use trace attributes for request-specific data.
- Review label usage in code review.
Code/Syntax Justification:
# Good labels (low cardinality)
http_requests_total{service="api", endpoint="/api/users", method="GET", status="200"}
# Bad labels (high cardinality)
http_requests_total{service="api", endpoint="/api/users/123", customer_id="abc-456"}
Workflow Diagram:
High-cardinality labels: Low-cardinality labels:
endpoint=/api/users/123 endpoint=/api/users
customer_id=abc-456 status=200
| |
v v
1M time series 10 time series
Prometheus OOM Prometheus healthy
🔍 Behind the Question: Label cardinality, label best practices, high-cardinality values, Prometheus design.
❌ The Red Flags: IDs/timestamps/IPs as labels; unbounded label values; not reviewing label usage.