Skip to main content

OneHub Technical Interview Masterclass: System Design

Total Questions: 81 Curriculum: Core Fundamentals + Applied + Principal (2026 additions: idempotency, sagas, rate limiting, event sourcing, Kafka ordering)


🏗️ Section 1: Core Fundamentals​

Q1: What is the CAP Theorem, and how do you choose between CP and AP?​

Technical Answer: CAP states that in a distributed system with a network partition (P), you can only guarantee either Consistency (C) or Availability (A). CP (Consistency + Partition Tolerance) systems (like etcd or HBase) prioritize data correctness; if a partition occurs, they become unavailable until the data is synced. AP (Availability + Partition Tolerance) systems (like Cassandra or DynamoDB) stay available but might return stale data (eventual consistency).

Code/Syntax Justification: N/A (Conceptual/Architectural) 🔍 Behind the Question: Distributed systems trade-offs and consistency models. ❌ The Red Flags: Claiming a system can be "CAP" (all three) or not knowing which database fits which model.

Q2: Explain "Consistent Hashing" and why it is used in Load Balancing.​

Technical Answer: Consistent Hashing maps both data (keys) and nodes (servers) to a circular hash space (the "ring"). A key is assigned to the first node encountered moving clockwise. When a node is added or removed, only 1/n keys need to be remapped, compared to nearly all keys in traditional hash(key) % n modulo hashing. This minimizes reshuffling and avoids cache misses in distributed caches or data stores.

Code/Syntax Justification:

# Simplified Logic
hash_ring = {node_hash: node_ip}
target_node = min([h for h in hash_ring if h >= key_hash])

🔍 Behind the Question: Scalability and distributed state management. ❌ The Red Flags: Modulo hashing for dynamic clusters.

Q3: What is the difference between horizontal and vertical scaling?​

Technical Answer: Vertical scaling increases the resources of a single machine, such as CPU, RAM, or disk. Horizontal scaling adds more machines and distributes load across them. Vertical scaling is simpler but has a hard ceiling and can create a single point of failure; horizontal scaling is more resilient and can grow much further, but requires load balancing, replication, and coordination.

Code/Syntax Justification:

Vertical: 1 server -> bigger server
Horizontal: 1 server -> 10 servers behind a load balancer

🔍 Behind the Question: Testing practical scalability tradeoffs. ❌ The Red Flags: Claiming vertical scaling is always enough.

Q4: What is a load balancer and why is it essential in distributed systems?​

Technical Answer: A load balancer is a component that spreads incoming traffic across multiple backend instances. It improves availability, prevents overload, and can enable health-based routing. At the application layer, it may inspect HTTP headers; at the transport layer, it simply forwards TCP connections. Good load balancers support health checks, TLS termination, and session affinity when needed.

Code/Syntax Justification:

upstream api_backends {
server 10.0.0.1:8000;
server 10.0.0.2:8000;
}

server {
location / {
proxy_pass http://api_backends;
}
}

🔍 Behind the Question: Testing availability, routing, and failure isolation. ❌ The Red Flags: Thinking a load balancer just "makes things faster" without explaining fault tolerance.

Q5: What is caching and when should you avoid it?​

Technical Answer: Caching stores computed or fetched data in faster storage so repeated reads do not hit the source system. It improves latency and reduces backend load, but it introduces staleness, invalidation complexity, and memory overhead. You should avoid caching when data changes too frequently, when consistency is critical, or when cache maintenance is more expensive than recomputation.

Code/Syntax Justification:

cache_key = f"user:{user_id}"
value = redis.get(cache_key)
if not value:
value = db.fetch_user(user_id)
redis.setex(cache_key, 300, value)

🔍 Behind the Question: Testing read optimization and invalidation awareness. ❌ The Red Flags: Caching everything blindly or ignoring cache expiration.

Q6: What is consistency in distributed systems?​

Technical Answer: Consistency means that all clients observe the same data at the same time or within a defined model. In strong consistency, once a write is acknowledged, all later reads see it. In eventual consistency, replicas converge over time and temporary divergence is allowed. The right choice depends on product needs such as finance, messaging, or social feeds.

Code/Syntax Justification:

Strong: read-after-write always sees the latest value
Eventual: read may see stale value briefly, then converges

🔍 Behind the Question: Testing distributed data semantics. ❌ The Red Flags: Confusing consistency with availability.

Q7: What is a circuit breaker pattern?​

Technical Answer: A circuit breaker prevents repeated calls to a failing dependency. After enough failures, it opens and stops sending traffic for a cooldown period. This avoids wasting resources and prevents cascading failure. After the timeout, it allows a limited number of trial requests in a half-open state to determine recovery.

Code/Syntax Justification:

if breaker.state == "open":
raise ServiceUnavailable()
response = call_dependency()

🔍 Behind the Question: Testing resilience and fault containment. ❌ The Red Flags: Not knowing half-open state or retry storm risks.

Q8: What is idempotency and why does every write API need it?​

Technical Answer: Idempotency means repeating an operation produces the same result as running it once. Network retries, double clicks, and message redelivery can all duplicate writes. A client-sent idempotency key, stored with result (and TTL), lets the server return the stored response instead of executing twice — critical for payments and order creation.

Code/Syntax Justification:

# Server side: key on Idempotency-Key header
key = request.headers["Idempotency-Key"]
result = redis.get(f"idem:{key}")
if result:
return json.loads(result)
resp = db.create_order(payload) # may run once only
redis.setex(f"idem:{key}", 86400, json.dumps(resp))
return resp

🔍 Behind the Question: Distributed systems reliability and at-least-once semantics. ❌ The Red Flags: Assuming database unique constraints alone solve duplicates; no TTL on keys.

Q9: How do you handle distributed transactions (Saga pattern)?​

Technical Answer: A saga breaks one big transaction into local transactions with compensating actions. If a step fails, run reverse steps for already-committed work. Choreographed sagas emit events between services; orchestrated sagas use a central coordinator. Sagas trade atomicity for availability — eventual consistency across services.

Code/Syntax Justification:

# Orchestrated saga: step 2 fails -> compensate step 1
try:
order = create_order() # step 1
payment = process_payment(order) # step 2
except PaymentError:
cancel_order(order) # compensate step 1
return "order cancelled"
ship(order) # step 3

🔍 Behind the Question: Distributed consistency without 2PC. ❌ The Red Flags: Claiming 2PC everywhere; missing compensation handlers.

Q10: How do you design a rate limiter at scale?​

Technical Answer: Use a sliding window or token bucket stored in Redis (INCR + EXPIRE for fixed window, or sorted sets for sliding). Put the limiter at the gateway/API layer, keyed by user/IP. Distributed limiters need atomic operations — Redis Lua scripts. Return 429 with Retry-After, and queue or shed excess load gracefully.

Code/Syntax Justification:

# Fixed window: count requests per user per minute
key = f"rl:{user_id}:{int(time.time()//60)}"
count = redis.incr(key)
if count == 1:
redis.expire(key, 60)
if count > LIMIT:
raise HTTPException(429, "Too Many Requests")

🔍 Behind the Question: Protecting APIs from abuse; choosing algorithms. ❌ The Red Flags: In-memory-only counters in multi-instance setups; no 429 handling.

Q11: How do you design a notification system (email/SMS/push)?​

Technical Answer: Producers emit NotificationRequest to a queue (SQS/Kafka). A worker normalizes, templates, and fan-outs to provider channels. Track delivery state in a DB, retry with exponential backoff per channel, and enforce per-user rate limits to avoid spam. Use a template service so content changes don't require deploys.

Code/Syntax Justification:

# Producer -> Queue -> Worker
queue.send({"user_id": u, "type": "email", "template": "reset", "ctx": {...}})

def worker(msg):
for ch in channels_for(msg["type"]):
send_with_retry(ch, render(templates[msg["template"]], msg["ctx"]))

🔍 Behind the Question: Async decoupling, retries, and failure isolation. ❌ The Red Flags: Sending synchronously in the request path.

Q12: How do you design a URL shortener?​

Technical Answer: Store (short_key → long_url) in a DB with a unique index. Generate keys via base62 encoding of an auto-increment ID or random strings with collision retry. Redirect: 301/302 lookup with caching (Redis) + DB fallback. Handle high read: cache hot keys; writes are low volume. Analytics: async log click events to a queue.

Code/Syntax Justification:

short = base62_encode(next_id()) # id from DB sequence
db.set(short, long_url)
# read path
url = redis.get(short) or db.get(short)
return RedirectResponse(url)

🔍 Behind the Question: Read-heavy caching, key generation, analytics pipeline. ❌ The Red Flags: No caching layer; key collision handling omitted.

Q13: How do you design a chat system (1:1 and group)?​

Technical Answer: Gateway layer holds WebSocket connections. Messages go to a message queue, App Server persists to DB (or timeline store), and fan-out pushes to recipients via push service or their live connections. For groups, fan-out per member (or lazily on read). Ordering per conversation via sequence numbers. Presence: heartbeat + Redis pub/sub.

Code/Syntax Justification:

# Fan-out via queue per conversation
queue.send({"conv": conv_id, "msg": msg})
# Consumer pushes to connected members' WebSocket
for member in conn_registry.get(conv_id):
member.ws.send(json.dumps(msg))

🔍 Behind the Question: Real-time delivery, ordering, presence, scaling connections. ❌ The Red Flags: Storing all history in One DB with no partition strategy.

Q14: What is event sourcing and when does it make sense?​

Technical Answer: Store state changes as an append-only log of events instead of only the current state. Rebuilding state = replay events. Great for audit trails, financial ledgers, and features needing full history or temporal queries. Cost: eventual consistency, event schema versioning, and replay complexity — do not use everywhere.

Code/Syntax Justification:

class Account:
def apply(self, event):
{"deposited": self.balance += event.amount,
"withdrawn": self.balance -= event.amount}[event.type]
# current balance = fold(replay(events), Account())

🔍 Behind the Question: Aggregates, projections, snapshots, CQRS pairing. ❌ The Red Flags: Event sourcing for every CRUD feature; no snapshotting.

Q15: How do you back up and restore a database with zero downtime?​

Technical Answer: Use continuous WAL archiving (PostgreSQL PITR): base backup + WAL segments → restore to any point in time. Logical dumps (pg_dump) for small DBs/portable data. Restore practice: stand up replica from backup, verify, promote. Backups must be tested or they don't exist.

Code/Syntax Justification:

# Base backup + archiving (PostgreSQL)
pg_basebackup -D /backup/base -Ft -z
# WAL archiving enabled: archive_command copies WAL continuously
pg_restore -d newdb /backup/base.tar.gz

🔍 Behind the Question: Durability, RTO/RPO design, backup testing. ❌ The Red Flags: Weekly full dumps only — lose up to 7 days of data.

Q16: How do you design for high read throughput (cache-aside)?​

Technical Answer: Read: check cache, on miss read DB and populate cache with TTL. Write: update DB, invalidate (or update) cache. Cache-aside is simple and resilient (DB is source of truth), risking a cache stampede on miss — mitigate with single-flight/request coalescing. Evict on write to avoid stale reads.

Code/Syntax Justification:

def get(key):
val = cache.get(key)
if val is None:
val = db.query(key) # cache miss
cache.setex(key, 600, val) # TTL: 10 min
return val

🔍 Behind the Question: Cache consistency and stampede control. ❌ The Red Flags: No TTL; write-through with stale-data bugs; no invalidation on update.

Q17: How do you handle message ordering and duplicates in Kafka?​

Technical Answer: Ordering: same key → same partition guarantees per-key order; one partition per key group. Duplicates: producers can retry → duplicate messages. Consumers must be idempotent (dedupe by event ID in DB unique index or state store). Exactly-once processing (transactions) is possible but complex and slower — at-least-once + idempotent consumer is the common production pattern.

Code/Syntax Justification:

producer.send("orders", key=order_id, value=data) # same key = same partition
# Consumer: dedupe by unique constraint
INSERT INTO events (event_id, ...) VALUES (...) ON CONFLICT (event_id) DO NOTHING

🔍 Behind the Question: Partitioning, ordering guarantees, exactly-once semantics. ❌ The Red Flags: Ignoring ordering per key; non-idempotent consumers.

Q18: What is backpressure and how do you implement it?​

Technical Answer: Backpressure slows producers when consumers can't keep up, preventing unbounded queue growth and memory pressure. Implement via: bounded queues, concurrency limits, rate limits at producers, and pushback signals (503s, circuit breakers). For Kafka, monitor consumer lag and alert; for HTTP, use bounded thread pools + reject.

🔍 Behind the Question: Load shedding and resiliency under overload. ❌ The Red Flags: Unbounded queues that OOM under spike traffic.

Q19: How do you design search with typo tolerance (e.g., "teh" → "the")?​

Technical Answer: Use a search engine (Elasticsearch/OpenSearch) with analyzers: lowercase, stemming, and n-gram edge tokenization give fuzzy matching. Or Levenshtein distance within edit distance 1-2 over candidate terms. Combine exact-term boost with fuzzy scoring; index synonyms for domain terms.

Code/Syntax Justification:

# Elasticsearch fuzzy query
{"query": {"fuzzy": {"title": {"value": "teh", "fuzziness": "AUTO"}}}}

🔍 Behind the Question: Search relevance, analyzers, ranking. ❌ The Red Flags: Doing fuzzy matching by scanning the whole DB in SQL.

Q20: How do you measure and design for p99 latency?​

Technical Answer: p99 = 99th percentile latency; the tail, not the average. Optimize: reduce lock contention, limit network round-trips, cache hot paths, keep dependency timeouts tight, and use request coalescing. Add tail-latency dashboards, and run load tests at p99, not just p50. One slow dependency call can drag p99 if not bounded.

🔍 Behind the Question: Perf engineering, load testing, tail latency awareness. ❌ The Red Flags: Only reporting average; no timeout on slow dependencies.


🏗️ Section 2: Applied System Design​

Q21: How do you design a rate limiter?​

Technical Answer: Rate limiting controls how many requests a client can make in a time window. Common algorithms: fixed window, sliding window, token bucket, leaky bucket. Implementation: in-memory for single-instance apps, Redis for distributed systems (atomic counters, sorted sets for sliding windows). Return 429 Too Many Requests with Retry-After.

Code/Syntax Justification:

# Token bucket in Redis (distributed)
key = f"ratelimit:{user_id}:{window}"
current = redis.incr(key)
if current == 1:
redis.expire(key, 60)
if current > LIMIT:
raise HTTPException(429, "Retry after 30s")

🔍 Behind the Question: Protection against abuse, fair resource allocation. ❌ The Red Flags: Client-side rate limiting only; no distributed coordination.

Q22: How do you design a URL shortener (detailed)?​

Technical Answer: Write path: accept long URL, generate short key (Base62 of auto-increment ID or random 7 chars), store in DB with expiry, return short URL. Read path: look up key, redirect. Scale: cache hot URLs in Redis, use CDN for redirects, shard DB by key prefix.

Code/Syntax Justification:

def shorten(long_url):
short_key = base62_encode(redis.incr("counter"))
db.set(short_key, long_url, ex=86400)
return f"https://short.co/{short_key}"

🔍 Behind the Question: Key generation, cache-aside, expiry, analytics. ❌ The Red Flags: Sequential IDs exposed; no expiry; cache stampede on misses.

Q23: How do you design a notification system?​

Technical Answer: Separate notification creation from delivery. API accepts notification, stores in DB, enqueues to message broker. Workers consume, classify by type (email/SMS/push/in-app), and dispatch via provider APIs. Preferences and templates stored per user. Retry with exponential backoff + dead-letter for failures.

Code/Syntax Justification:

# Producer
notification = {"user_id": uid, "type": "email", "template": "welcome"}
queue.send("notifications", notification)

# Worker
for msg in queue.consume("notifications"):
provider = PROVIDERS[msg["type"]]
provider.send(msg["user_id"], msg["template"])

🔍 Behind the Question: Fan-out, retry, preference management, provider abstraction. ❌ The Red Flags: Synchronous email sending in the request path.

Q24: What is the Circuit Breaker pattern and when do you use it?​

Technical Answer: A circuit breaker stops calling a failing downstream service after a threshold of failures, giving it time to recover. States: closed (normal), open (failing fast), half-open (testing recovery). Prevents cascading failures and resource exhaustion.

Code/Syntax Justification:

class CircuitBreaker:
def call(self, fn, *args):
if self.state == "open":
if time() - self.opened_at > self.reset_timeout:
self.state = "half-open"
else:
raise CircuitOpenError()
try:
result = fn(*args)
self.on_success()
return result
except Exception:
self.on_failure()
raise

🔍 Behind the Question: Resilience, failure isolation, timeout management. ❌ The Red Flags: No timeout; retrying without backoff; ignoring downstream health.

Q25: How do you design a distributed cache?​

Technical Answer: Use a consistent hashing ring (Redis Cluster, Memcached) to distribute keys across nodes. Each node owns a token range. Clients route requests to the correct node. Replication per shard for HA. Eviction: LRU/LFU per shard. Monitoring: hit rate, eviction rate, memory pressure.

Code/Syntax Justification:

# Client-side consistent hashing
ring = ConsistentHashRing(nodes=["cache-1", "cache-2", "cache-3"])
node = ring.get_node(key)
client = cache_clients[node]
value = client.get(key)

🔍 Behind the Question: Data partitioning, replication, eviction policies. ❌ The Red Flags: Single cache node; no eviction strategy; no replication.

Q26: How do you design a web crawler?​

Technical Answer: Seed URLs → URL frontier (priority queue) → fetcher ( respects robots.txt, rate limits per domain) → parser (extract links + content) → dedupe (Bloom filter or seen-URL DB) → storage (DB + index). Politeness: per-domain crawl delay, concurrent limit per host. Scale: distributed fetchers, Kafka for URL queue, dedupe with Bloom filter + DB unique index.

Code/Syntax Justification:

frontier = PriorityQueue()
frontier.put(seed_url, priority=1.0)

while not frontier.empty():
url = frontier.get()
if url in seen: continue
seen.add(url)
html = fetch(url, delay=DOMAIN_DELAYS[domain])
links = parse_links(html)
for link in links:
frontier.put(link, priority=page_rank(link))

🔍 Behind the Question: Politeness, deduplication, scalability, storage. ❌ The Red Flags: No robots.txt handling; no per-domain rate limiting.

Q27: How do you design a file storage service like S3?​

Technical Answer: API: PUT/GET/DELETE/LIST objects. Storage: object store (S3, MinIO) with erasure coding or replication. Metadata in DB (filename, size, owner, ACL, version). Large files: multipart upload. CDN for hot objects. Consistency: read-after-write for new objects, eventual for overwrite/delete.

Code/Syntax Justification:

# Multipart upload
s3.create_multipart_upload(Bucket="bucket", Key="large_file")
parts = [s3.upload_part(Bucket="bucket", Key="large_file",
PartNumber=i, Body=chunk) for i, chunk in enumerate(chunks)]
s3.complete_multipart_upload(Bucket="bucket", Key="large_file",
MultipartUpload={"Parts": parts})

🔍 Behind the Question: Durability, scalability, multipart upload, CDN integration. ❌ The Red Flags: No multipart for large files; no versioning; single AZ only.

Q28: What is idempotency and why is it critical in distributed systems?​

Technical Answer: Idempotency means calling an operation multiple times produces the same result as calling it once. Critical for payment processing, retries, and message queues. Achieve via idempotency keys (client-generated UUID stored in DB unique index) or natural keys.

Code/Syntax Justification:

# Idempotent payment
INSERT INTO payments (idempotency_key, amount, status)
VALUES (%s, %s, 'pending')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING status;

🔍 Behind the Question: Safe retries, duplicate prevention, financial correctness. ❌ The Red Flags: Retrying without idempotency keys; double-charging users.

Q29: How do you design a leaderboard?​

Technical Answer: Use a sorted set data structure (Redis ZADD/ZRANGE) for real-time rankings. Score = player points. For massive scale, precompute top-K periodically and cache. Handle ties: secondary sort by timestamp (lower = better). Persist to DB for durability; Redis is cache.

Code/Syntax Justification:

# Update score
redis.zadd("leaderboard", {player_id: new_score})

# Get top 10
top = redis.zrevrange("leaderboard", 0, 9, withscores=True)

# Get rank
rank = redis.zrevrank("leaderboard", player_id)

🔍 Behind the Question: Real-time ranking, tie-breaking, persistence. ❌ The Red Flags: Sorting in SQL on every request; no caching.

Q30: How do you design a task scheduler (like Celery)?​

Technical Answer: API accepts task definition (function, args, schedule). Persistent queue (Redis/RabbitMQ) holds tasks. Workers pull tasks, execute, update status. Beat scheduler for periodic tasks. Result backend stores task state (pending/started/success/failure). Retries with exponential backoff; dead-letter for poison tasks.

Code/Syntax Justification:

# Enqueue task
task_id = celery.send_task("send_email", args=[user_id], countdown=300)

# Worker
@app.task(bind=True, max_retries=3)
def send_email(self, user_id):
try:
email_service.send(user_id)
except TemporaryError:
raise self.retry(countdown=60)

🔍 Behind the Question: Async execution, retries, monitoring, scheduling. ❌ The Red Flags: Synchronous task execution; no retry/backoff; no monitoring.

Q31: How do you design a recommendation system?​

Technical Answer: Offline: batch compute recommendations (collaborative filtering, content-based) nightly, store in cache. Online: fetch precomputed recommendations, blend with real-time signals (recent views, trends). Pipeline: data collection → feature store → model training → serving. Cold start: popular items, content features for new users.

Code/Syntax Justification:

# Online serving
user_recs = redis.zrange(f"recs:{user_id}", 0, 49)
trending = redis.zrange("trending:today", 0, 19)
final = blend(user_recs, trending, weights=[0.8, 0.2])

🔍 Behind the Question: Offline vs online, cold start, real-time signals. ❌ The Red Flags: Pure real-time computation; no fallback for new users.

Q32: How do you design an analytics pipeline?​

Technical Answer: Collect events via SDK/API → ingest service validates and enriches → event bus (Kafka) buffers → stream processors (Flink/Kafka Streams) aggregate → data warehouse (Snowflake/BigQuery) for analysis → BI tools (Looker/Metabase) for dashboards. Idempotent ingestion, exactly-once processing, partitioning by date/event type.

Code/Syntax Justification:

# Kafka producer
producer.send("events", key=event_type, value=event_data)

# Flink aggregation
events.key_by("user_id").window(TumblingEventTimeWindows.of(Time.hours(1)))
.aggregate(new CountAggregator())

🔍 Behind the Question: Event volume, throughput, aggregation, storage. ❌ The Red Flags: No event buffering; synchronous DB writes on hot path.

Q33: How do you design a payment processing system?​

Technical Answer: API accepts payment intent → validates amount/currency → calls payment provider (Stripe/PayPal) → webhook confirms result → idempotency key prevents duplicate charges → DB records transaction → async emits events (order confirmed, email sent). Reconciliation job nightly compares provider statements with internal records.

Code/Syntax Justification:

# Idempotent charge
charge = stripe.Charge.create(
amount=amount,
currency="usd",
source=token,
idempotency_key=idempotency_key
)

🔍 Behind the Question: Idempotency, webhooks, reconciliation, compliance. ❌ The Red Flags: No idempotency; no reconciliation; trusting webhook without verification.

Q34: How do you design a search autocomplete system?​

Technical Answer: As user types, send prefix to API. Trie data structure in memory for instant prefix lookup. Popular queries cached in Redis. Fallback to Elasticsearch for long-tail queries. Rank by: query frequency, recency, personalization. Debounce input (150-300ms) to reduce load.

Code/Syntax Justification:

# Trie node
class TrieNode:
def __init__(self):
self.children = {}
self.is_end = False
self.frequency = 0

# Redis ranked suggestions
ZADD suggestions {query: frequency}
ZRANGEBYLEX suggestions [prefix] [prefix+\xff"

🔍 Behind the Question: Latency, ranking, caching, fallback strategy. ❌ The Red Flags: SQL LIKE queries; no debouncing; no ranking.

Q35: How do you design a multi-tenant SaaS architecture?​

Technical Answer: Three approaches: (1) Shared DB, shared schema (tenant_id column, row-level security). (2) Shared DB, separate schemas per tenant. (3) Separate DB per tenant. Choose by: isolation requirements, tenant count, operational complexity. Shared schema is simplest; separate DB is strongest isolation.

Code/Syntax Justification:

# Shared schema with tenant isolation
def get_tenant_db(tenant_id):
return scoped_session(lambda: engine.connect().execution_options(
schema_translate_map={"tenant": f"tenant_{tenant_id}"}
))

# PostgreSQL RLS
SET app.tenant = 'tenant_123';
SELECT * FROM orders WHERE tenant_id = current_setting('app.tenant');

🔍 Behind the Question: Isolation, cost, scalability, operational complexity. ❌ The Red Flags: No tenant isolation; all tenants share one superuser.

Q36: How do you design a video streaming service?​

Technical Answer: Upload: chunked upload to object store, transcode to multiple resolutions (HLS/DASH) via media server. Storage: CDN for hot content, origin server for cold. Streaming: HLS chunks (.m3u8 manifest + .ts segments). DRM if needed. Scale: edge caching, origin shield, prefetch next segments.

Code/Syntax Justification:

# Transcode with FFmpeg
ffmpeg.input("input.mp4").output(
"output_720p.m3u8",
vf="scale=1280:720",
codec="libx264",
format="hls",
hls_time=4,
hls_playlist_type="vod"
).run()

🔍 Behind the Question: Adaptive bitrate, transcoding, CDN, latency. ❌ The Red Flags: Single resolution; no chunking; client downloads full file.

Q37: How do you design a logging and monitoring system?​

Technical Answer: Structured logging (JSON) from services → collection agent (Fluentd/Vector) → buffer (Kafka) → processing (parse, enrich, PII redact) → storage (Elasticsearch for logs, Prometheus for metrics, Tempo for traces) → visualization (Grafana, Kibana) → alerting.

Code/Syntax Justification:

# Structured log
logger.info("payment_processed",
extra={"user_id": uid, "amount": amount, "provider": "stripe"})

🔍 Behind the Question: Log volume, retention, correlation, alerting. ❌ The Red Flags: Unstructured logs; no correlation IDs; no retention policy.

Q38: How do you design a content delivery network (CDN)?​

Technical Answer: CDN caches static content (images, JS, CSS, video) at edge locations close to users. Request: DNS resolves to nearest PoP, PoP serves cached content or fetches from origin. Cache-control headers determine TTL. Invalidation: purge by URL or tag. Dynamic content: route to origin with geo-routing.

Code/Syntax Justification:

# Cache-control header
location /static/ {
expires 1y;
add_header Cache-Control "public, immutable";
}

🔍 Behind the Question: Cache hit ratio, invalidation, origin shield, geo-routing. ❌ The Red Flags: No cache headers; no invalidation strategy; origin serves all traffic.

Q39: How do you design a secrets management system?​

Technical Answer: Store secrets encrypted at rest in a vault (HashiCorp Vault, AWS Secrets Manager). Access via API with authentication (IAM, mTLS, tokens). Audit every access. Rotate secrets periodically. Inject secrets into apps via environment variables or sidecar — never hardcode. Use separate credentials per service.

Code/Syntax Justification:

# Fetch secret from Vault
client = hvac.Client(url=VAULT_URL)
secret = client.secrets.kv.read_secret_version(path="prod/db")
db_password = secret["data"]["data"]["password"]

🔍 Behind the Question: Encryption at rest, access control, rotation, audit. ❌ The Red Flags: Secrets in code or config files; no rotation; no audit log.

Q40: How do you design a CI/CD pipeline?​

Technical Answer: Code push → trigger CI (lint, test, build) → artifact registry (Docker image) → deploy to staging → automated tests (integration, E2E) → manual approval → deploy to production → smoke tests → monitor. Rollback: keep previous version, automate rollback on failure.

Code/Syntax Justification:

# GitHub Actions
jobs:
deploy:
steps:
- run: docker build -t app:${{ github.sha }} .
- run: kubectl set image deployment/app app=app:${{ github.sha }}
- run: kubectl rollout status deployment/app

🔍 Behind the Question: Automation, testing gates, rollback, observability. ❌ The Red Flags: No automated tests; manual deploys only; no rollback plan.


🏗️ Section 3: Principal-Level Topics​

Q41: How do you design a distributed transaction system?​

Technical Answer: Distributed transactions span multiple services. Patterns: (1) 2PC (Two-Phase Commit) — coordinator asks all participants to prepare, then commit. Strong consistency but blocking. (2) Saga — each service commits locally and emits compensating transactions for rollback. Eventual consistency, no blocking. (3) Outbox pattern — atomic local write + outbox table, relay to other services.

Code/Syntax Justification:

# Saga orchestration
@transactional
def place_order(order_id, items):
payment = charge(order_id, items.total)
if not payment.success:
compensate("order_created", order_id)
raise PaymentFailed()
inventory = reserve(items)
if not inventory.success:
compensate("payment_charged", order_id)
raise InventoryFailed()
confirm(order_id)

🔍 Behind the Question: Consistency models, failure handling, saga vs 2PC. ❌ The Red Flags: 2PC without timeout handling; no compensation logic.

Q42: What is CQRS and when should you use it?​

Technical Answer: Command Query Responsibility Segregation separates write model (commands) from read model (queries). Writes go through domain logic to the write DB; reads hit a denormalized read DB optimized for queries. Benefits: independent scaling, optimized reads, clear separation. Cost: eventual consistency between models, added complexity.

Code/Syntax Justification:

# Command
def create_order(command):
order = Order(**command.data)
order_db.save(order)
event_bus.publish(OrderCreated(order.id, ...))

# Query
def get_order(order_id):
return read_db.query(OrderView).filter_by(id=order_id).first()

🔍 Behind the Question: Read/write separation, event-driven sync, complexity tradeoff. ❌ The Red Flags: CQRS for simple CRUD; no event sync strategy.

Q43: How do you design a data pipeline for real-time analytics?​

Technical Answer: Ingest: Kafka for high-throughput event streaming. Process: Flink/Spark Streaming for windowed aggregations. Store: ClickHouse/Druid for OLAP queries. Serve: REST API or BI tool. Ensure exactly-once with idempotent sinks and transactional writes.

Code/Syntax Justification:

# Flink job
env.add_source(kafka_source) \
.key_by(lambda e: e["user_id"]) \
.window(TumblingEventTimeWindows.of(Time.minutes(5))) \
.aggregate(UserEventCountAggregator()) \
.add_sink(clickhouse_sink)

🔍 Behind the Question: Throughput, latency, exactly-once, storage choices. ❌ The Red Flags: Batch processing for real-time needs; no windowing strategy.

Q44: How do you handle data migration in production with zero downtime?​

Technical Answer: Expand-contract pattern: (1) Add new column, write to both old and new. (2) Backfill existing rows to new column. (3) Switch reads to new column. (4) Drop old column. For complex changes: use feature flags and shadow traffic to validate new logic before cutover.

Code/Syntax Justification:

# Step 1: Add new column
ALTER TABLE users ADD COLUMN email_new VARCHAR(255);

# Step 2: Backfill
UPDATE users SET email_new = email WHERE email_new IS NULL;

# Step 3: Switch reads
SELECT COALESCE(email_new, email) FROM users;

# Step 4: Drop old column
ALTER TABLE users DROP COLUMN email;

🔍 Behind the Question: Zero-downtime migrations, backfill strategy, feature flags. ❌ The Red Flags: Big-bang migration; no backfill; dropping old column immediately.

Q45: How do you design a distributed lock service?​

Technical Answer: Use a consensus system (etcd, Zookeeper) or Redis with Redlock algorithm. Lock acquisition: create ephemeral node or set key with TTL. Lock release: delete node or key. Heartbeat: extend TTL periodically. Fencing: include token in requests to prevent stale lock use.

Code/Syntax Justification:

# Redis distributed lock
lock = redis.set("lock:resource", token, nx=True, ex=30)
if lock:
try:
# critical section
pass
finally:
redis.delete("lock:resource")

🔍 Behind the Question: Consensus, lock granularity, fencing tokens, failure handling. ❌ The Red Flags: No TTL; no fencing; single-node Redis for critical locks.

Q46: What is the Outbox pattern and why is it important?​

Technical Answer: Instead of publishing events directly to a message broker (risking inconsistency if the DB transaction commits but the message fails), write events to an outbox table in the same DB transaction. A separate process reads the outbox and publishes to the broker, ensuring atomicity.

Code/Syntax Justification:

@transactional
def create_order(order):
order_db.save(order)
outbox.save({"event": "order_created", "order_id": order.id})

# Outbox publisher
while True:
events = outbox.fetch_unpublished(limit=100)
for event in events:
kafka.send("orders", event)
outbox.mark_published(event.id)

🔍 Behind the Question: Atomicity between DB and message broker, reliability. ❌ The Red Flags: Publishing events outside the transaction; no retry.

Q47: How do you design a system for end-to-end encryption?​

Technical Answer: Each user has a key pair. Messages encrypted with recipient's public key; only recipient's private key decrypts. Key exchange: secure channel or key exchange protocol (Signal Protocol). Key storage: private keys on device, public keys on server. Forward secrecy: derive per-message keys from master key.

Code/Syntax Justification:

# Signal Protocol (conceptual)
identity_key = generate_key_pair()
prekey = generate_key_pair()
session = X3DH(identity_key, prekey, recipient_prekey)
message_key = KDF(session.shared_secret, counter=0)
ciphertext = AES_GCM(message, message_key)

🔍 Behind the Question: Key management, forward secrecy, key exchange. ❌ The Red Flags: Server holds private keys; no forward secrecy.

Q48: How do you design a machine learning model serving system?​

Technical Answer: Training pipeline: data collection → feature engineering → model training → validation → registry (MLflow). Serving: model server (TorchServe, TF Serving) loads model, exposes REST/gRPC API. Features: feature store for online features, A/B testing, shadow deployments. Monitor: prediction latency, drift, accuracy.

Code/Syntax Justification:

# Model server (FastAPI + TorchServe)
@app.post("/predict")
def predict(features: FeatureVector):
tensor = preprocess(features)
prediction = model(tensor)
return {"class": prediction.argmax().item()}

🔍 Behind the Question: Model versioning, feature consistency, monitoring, A/B testing. ❌ The Red Flags: Model in monolith; no feature store; no drift monitoring.

Q49: How do you design a system to handle 1M concurrent WebSocket connections?​

Technical Answer: Use async server (Node.js, Go, FastAPI with WebSockets) on beefy instances or scale horizontally behind a load balancer with sticky sessions. Each connection holds minimal state. Use connection registry (Redis pub/sub) to broadcast across instances. Heartbeat/ping-pong to detect dead connections. Connection limits per instance, autoscaling on connection count.

Code/Syntax Justification:

# FastAPI WebSocket with Redis pub/sub
@app.websocket("/ws")
async def ws(websocket: WebSocket):
await websocket.accept()
pubsub = redis.pubsub()
await pubsub.subscribe("broadcast")
async for message in pubsub.listen():
await websocket.send_text(message["data"])

🔍 Behind the Question: Connection scalability, state management, cross-instance broadcast. ❌ The Red Flags: Thread-per-connection; storing full state per connection in memory.

Q50: How do you design a data mesh?​

Technical Answer: Decentralized data ownership: each domain team owns their data products (datasets, APIs). Self-serve infrastructure: data platform team provides tools (storage, pipelines, observability). Federated governance: common standards for quality, security, SLA. Consumers discover and use data products via a catalog.

Code/Syntax Justification:

# Data product manifest
name: orders
owner: team-payments
sla: 99.9%
quality_rules:
- column: amount
type: decimal
nullable: false
access: rest /graphql

🔍 Behind the Question: Domain ownership, self-serve, federated governance. ❌ The Red Flags: Centralized data team bottleneck; no domain ownership.

Q51: How do you design a system for real-time collaborative editing (Google Docs)?​

Technical Answer: Operational Transformation (OT) or Conflict-free Replicated Data Type (CRDT). Each client sends operations (insert/delete) to server. Server broadcasts operations to all clients. Clients apply operations in order. CRDT is simpler for distributed environments; OT is older but proven.

Code/Syntax Justification:

# CRDT concept (LWW-Element-Set)
class LWWRegister:
def __init__(self):
self.state = {} # {element: (timestamp, value)}
def add(self, element, value, timestamp):
if timestamp > self.state.get(element, (0,))[0]:
self.state[element] = (timestamp, value)

🔍 Behind the Question: Concurrency, conflict resolution, latency. ❌ The Red Flags: Last-write-wins globally; no conflict resolution.

Q52: How do you design a feature flag system?​

Technical Answer: Flag service stores flags with targeting rules (user segment, percentage rollout, whitelist). SDK evaluates rules client-side for low latency or server-side for consistency. Features: gradual rollout, kill switch, A/B testing, targeting by user attributes. Store flag state and audit log.

Code/Syntax Justification:

def is_enabled(flag_name, user):
flag = flag_service.get(flag_name)
if flag.rollout_percentage:
return hash(f"{user.id}:{flag_name}") % 100 < flag.rollout_percentage
return user.id in flag.whitelist

🔍 Behind the Question: Gradual rollout, targeting, audit, consistency. ❌ The Red Flags: Flags in config files; no targeting; no audit trail.

Q53: How do you design an API gateway?​

Technical Answer: Gateway is single entry point: routes requests to microservices, handles auth (JWT validation), rate limiting, request/response transformation, logging, circuit breaking. Can be built with Nginx/Envoy or frameworks like Kong. Benefit: centralized cross-cutting concerns.

Code/Syntax Justification:

# Envoy route
virtual_hosts:
- name: api
domains: ["api.example.com"]
routes:
- match: { prefix: "/users" }
route: { cluster: user_service }
- match: { prefix: "/orders" }
route: { cluster: order_service }

🔍 Behind the Question: Routing, auth, rate limiting, observability. ❌ The Red Flags: No auth at gateway; no timeout; single point of failure.

Q54: How do you design a pub/sub system?​

Technical Answer: Publisher sends message to topic. Broker (Kafka, RabbitMQ, Redis Pub/Sub) routes to all subscribers. Scaling: partition topics for parallelism. Delivery guarantees: at-most-once, at-least-once, exactly-once. Retention: time-based or size-based. Ordering: per-partition.

Code/Syntax Justification:

# Publisher
producer.send("notifications", key="user:123", value={"type": "email"})

# Subscriber
consumer.subscribe(["notifications"])
for msg in consumer:
process(msg.value)
consumer.commit()

🔍 Behind the Question: Delivery guarantees, scaling, retention, ordering. ❌ The Red Flags: No delivery guarantee; no partitioning; unbounded retention.

Q55: How do you design a distributed cache invalidation strategy?​

Technical Answer: Strategies: (1) TTL — expire after fixed time, simplest. (2) Write-through — update cache on write, stale-free but write penalty. (3) Write-behind — async cache update, fast but risk of loss. (4) Event-driven — publish invalidation event on write, subscribers evict cache. Choose by consistency requirements.

Code/Syntax Justification:

# Event-driven invalidation
@transactional
def update_user(user_id, data):
db.execute("UPDATE users SET ... WHERE id = %s", (user_id,))
cache.delete(f"user:{user_id}")
event_bus.publish("user_updated", {"user_id": user_id})

🔍 Behind the Question: Cache consistency, write patterns, event-driven architecture. ❌ The Red Flags: No invalidation; cache lives forever with stale data.

Q56: How do you design a system to prevent double-click/duplicate submissions?​

Technical Answer: Client: disable submit button after click, show loading state. Server: idempotency key (client-generated UUID) + DB unique constraint. API: check idempotency key before processing, return existing result if duplicate. This handles network retries and user double-clicks.

Code/Syntax Justification:

@app.post("/payments")
def create_payment(request: PaymentRequest):
existing = db.query(Payment).filter_by(idempotency_key=request.idempotency_key).first()
if existing:
return existing
payment = Payment(**request.dict())
db.add(payment)
db.commit()
return payment

🔍 Behind the Question: Idempotency, UX, retry safety. ❌ The Red Flags: No idempotency key; server processes every request blindly.

Q57: How do you design a rate-based autoscaler?​

Technical Answer: Monitor metrics: CPU, memory, request rate, queue depth, p99 latency. Autoscale rules: if CPU > 70% for 2 min, add instance; if < 30% for 5 min, remove instance. Scale-out faster than scale-in to avoid capacity shortage. Cooldown period prevents flapping.

Code/Syntax Justification:

# Kubernetes HPA
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70

🔍 Behind the Question: Metric selection, cooldown, scale direction asymmetry. ❌ The Red Flags: Autoscaling on request count only; no cooldown.

Q58: How do you design a system for A/B testing?​

Technical Answer: Assignment service maps user ID to variant (control/treatment) using consistent hashing or random bucket. Store assignments in DB for consistency. Track metrics per variant (conversion rate, latency). Statistical significance calculator determines winner. Feature flags often used alongside.

Code/Syntax Justification:

def get_variant(user_id, experiment_id):
bucket = hash(f"{user_id}:{experiment_id}") % 100
variant = "treatment" if bucket < 50 else "control"
db.execute("INSERT INTO assignments ...")
return variant

🔍 Behind the Question: Assignment consistency, statistical rigor, metric tracking. ❌ The Red Flags: Random assignment per request; no statistical significance check.

Q59: How do you design a system for handling time-series data?​

Technical Answer: Use a time-series database (InfluxDB, TimescaleDB, Prometheus) optimized for timestamp-indexed writes and queries. Data model: metric name + tags (dimensions) + timestamp + value. Retention policies downsample or expire old data. Queries: range aggregations, downsampling.

Code/Syntax Justification:

-- TimescaleDB hypertable
CREATE TABLE metrics (
time TIMESTAMPTZ NOT NULL,
metric TEXT,
value DOUBLE PRECISION,
tags JSONB
);
SELECT create_hypertable('metrics', 'time');

🔍 Behind the Question: Write throughput, retention, aggregation, storage efficiency. ❌ The Red Flags: Storing time-series in a relational table without partitioning.

Q60: How do you design a disaster recovery (DR) strategy?​

Technical Answer: Define RPO (Recovery Point Objective: max acceptable data loss) and RTO (Recovery Time Objective: max acceptable downtime). Backup strategy: continuous replication (RPO ~0), periodic snapshots (RPO = snapshot interval). DR site: warm standby (async replication, manual failover) or hot standby (sync replication, automatic failover). Test failover regularly.

Code/Syntax Justification:

# PostgreSQL streaming replication
# Primary
wal_level = replica
max_wal_senders = 5

# Replica
primary_conninfo = 'host=primary port=5432'

🔍 Behind the Question: RPO/RTO, replication modes, failover automation. ❌ The Red Flags: No tested restore; backups on same site as primary.

Q61: How do you design a microservices communication pattern?​

Technical Answer: Patterns: (1) Sync REST/HTTP — simple, but couples services. (2) gRPC — typed contract, streaming, high performance. (3) Async messaging (Kafka, RabbitMQ) — decouples services, resilient. (4) Event-driven — services react to events. Choose: sync for request/response, async for workflows and fan-out.

Code/Syntax Justification:

# Sync (HTTP)
response = httpx.get("http://order-service/orders/123")

# Async (Kafka)
producer.send("order_events", key="order:123", value=event)

🔍 Behind the Question: Coupling, latency, reliability, scalability. ❌ The Red Flags: Chatty sync calls forming a distributed monolith.

Q62: How do you design a system for handling large file uploads?​

Technical Answer: Chunked upload: client splits file into chunks, uploads each chunk with chunk number, server assembles. Benefits: resume on failure, parallel uploads, no request size limits. Storage: object store (S3) with multipart upload. Track upload state in DB. Cleanup incomplete uploads with lifecycle policy.

Code/Syntax Justification:

# Multipart upload
s3.create_multipart_upload(Bucket="bucket", Key="file.zip")
parts = []
for i, chunk in enumerate(chunks):
resp = s3.upload_part(Bucket="bucket", Key="file.zip",
PartNumber=i+1, Body=chunk)
parts.append({"PartNumber": i+1, "ETag": resp["ETag"]})
s3.complete_multipart_upload(Bucket="bucket", Key="file.zip",
MultipartUpload={"Parts": parts})

🔍 Behind the Question: Resume, parallelism, storage efficiency, cleanup. ❌ The Red Flags: Single POST with 5GB file; no resume support.

Q63: How do you design a system for audit logging?​

Technical Answer: Immutable append-only log of all actions (who, what, when, result). Store in separate, append-only store (write-only DB, object storage). Include: user ID, action, resource type, resource ID, timestamp, IP, result. Sign entries or use blockchain-style hashing for tamper detection. Retention: immutable for compliance period.

Code/Syntax Justification:

def audit_log(user_id, action, resource, result):
db.execute(
"INSERT INTO audit_log (user_id, action, resource_id, ts, ip) VALUES (?, ?, ?, ?, ?)",
(user_id, action, resource.id, now(), request.ip)
)

🔍 Behind the Question: Tamper resistance, compliance, completeness, retention. ❌ The Red Flags: Mutable log table; no user context; log in same DB as business data.

Q64: How do you design a system for graceful degradation?​

Technical Answer: Identify critical vs non-critical features. Under load: disable non-critical features (recommendations, personalization), serve cached responses, queue non-urgent work. Circuit breakers on downstream services. Feature flags for instant toggle. Monitor system load and auto-degrade.

Code/Syntax Justification:

def get_homepage():
if system_load > 0.9:
return cached_homepage() # degraded mode
recommendations = rec_service.get(user_id) # may fail
return render_homepage(recommendations)

🔍 Behind the Question: Resilience under load, feature criticality, monitoring. ❌ The Red Flags: All-or-nothing failure; no fallback; no load-based switching.

Q65: How do you design a bulk email system?​

Technical Answer: Queue-based architecture. API accepts send request → validates → enqueues to message broker. Workers consume, render template with user data, send via provider (SendGrid, SES). Rate limiting per domain/IP, bounce/complaint handling via webhooks, unsubscribe management. Retry with exponential backoff for transient failures.

Code/Syntax Justification:

# Producer
queue.send("email", {"template": "welcome", "user_id": uid})

# Worker
template = render("welcome", user=user)
provider.send(user.email, subject, template)

🔍 Behind the Question: Throughput, deliverability, bounce handling, compliance. ❌ The Red Flags: Synchronous email sending; no retry; no bounce handling.

Q66: How do you design a configuration management system?​

Technical Answer: Central config store (etcd, Consul, Zookeeper) or config-as-code (git). Features: versioning, access control, change audit, hot reload. Applications pull config at startup or watch for changes. Sensitive values: use secret manager integration. Environment-specific configs.

Code/Syntax Justification:

# etcd client
client = etcd.client(host="etcd.service")
config = client.get("/config/app").value
# Watch for changes
events, _ = client.watch("/config/app")
for event in events:
reload_config(event.value)

🔍 Behind the Question: Centralization, versioning, secrets, hot reload. ❌ The Red Flags: Config in environment variables only; no versioning; no secrets separation.

Q67: How do you design a system for user presence (online/offline)?​

Technical Answer: Heartbeat: client sends heartbeat every N seconds. Server updates last-seen timestamp in Redis with TTL. Presence service aggregates by user. Online if last-seen < threshold. WebSocket connections tracked in connection registry for real-time presence.

Code/Syntax Justification:

# Heartbeat
redis.setex(f"presence:{user_id}", 30, "online")

# Get online users
online = [uid for uid in user_ids if redis.exists(f"presence:{uid}")]

🔍 Behind the Question: Heartbeat interval, false positives, scalability. ❌ The Red Flags: No TTL; heartbeat interval too short; no connection registry.

Q68: How do you design a fraud detection system?​

Technical Answer: Real-time: rules engine (velocity checks, amount thresholds) + ML model score on every transaction. Batch: nightly ML model retraining on labeled fraud data. Features: user behavior, device fingerprint, IP reputation, transaction graph. Alert: high-score transactions held for review.

Code/Syntax Justification:

def evaluate_transaction(txn):
score = ml_model.predict(txn.features)
if score > 0.9:
hold_for_review(txn)
elif score > 0.7:
require_2fa(txn)
else:
approve(txn)

🔍 Behind the Question: Real-time vs batch, feature engineering, false positives. ❌ The Red Flags: Rules only; no ML; high false positive rate.

Q69: How do you design a system for OAuth2 / SSO?​

Technical Answer: Authorization server issues access tokens (JWT or opaque). Resource server validates token (JWKS endpoint for JWT, introspection for opaque). Client stores token securely (HttpOnly cookie or secure storage). Refresh tokens for long-lived sessions. PKCE for public clients (mobile, SPA).

Code/Syntax Justification:

# Token validation
jwks_client = jwt.PyJWKClient("https://auth.example.com/.well-known/jwks.json")
payload = jwt.decode(token, jwks_client, algorithms=["RS256"])

🔍 Behind the Question: Token types, validation, refresh, PKCE, security. ❌ The Red Flags: Storing tokens in localStorage; no refresh tokens; no PKCE.

Q70: How do you design a message queue with exactly-once semantics?​

Technical Answer: Producer: idempotent producer with sequence numbers. Broker: deduplicate by producer ID + sequence number. Consumer: transactional consume + process + commit. Or at-least-once delivery + idempotent consumer (DB unique constraint). Exactly-once is expensive; at-least-once + idempotency is usually sufficient.

Code/Syntax Justification:

# Idempotent consumer
INSERT INTO processed_events (event_id, data) VALUES (?, ?)
ON CONFLICT (event_id) DO NOTHING

🔍 Behind the Question: Delivery guarantees, idempotency, cost. ❌ The Red Flags: Exactly-once for everything; no idempotency fallback.


Senior / Principal — Real World Scenarios​

Q71: Cascading failure — one slow dependency takes down the whole system​

Level: Senior

Scenario: A FastAPI gateway calls three downstream services (payment, inventory, user). The payment service degrades and responds slowly. Soon the gateway itself becomes unhealthy, and even inventory/user calls fail though those services are healthy.

Impact: Total site outage caused by one degraded dependency.

Root Cause: No timeout, no circuit breaker, and no bulkhead isolation. All worker connections get occupied waiting on payment, exhausting the connection pool and leaving no capacity for other calls.

Simple explanation: A narrow hallway gets blocked by one slow person. Everyone behind them is stuck, even people going to different rooms.

Technical Answer:

  1. Per-dependency timeouts so one slow call fails fast.
  2. Circuit breaker per dependency — after N failures, open the circuit and reject fast for a cooldown period.
  3. Bulkhead isolation — separate connection pools / thread pools per downstream service so one cannot starve the others.
  4. Fallback responses (degraded mode) when a dependency is down.

Code/Syntax Justification:

from circuitbreaker import circuit
import httpx

@circuit(failure_threshold=5, recovery_timeout=30)
async def call_payment(data):
async with httpx.AsyncClient(timeout=3.0) as client:
resp = await client.post("https://payment.internal/charge", json=data)
resp.raise_for_status()
return resp.json()

@app.post("/checkout")
async def checkout(order: Order):
try:
payment = await call_payment(order)
except CircuitBreakerError:
return {"status": "degraded", "message": "payment temporarily unavailable"}
# proceed with inventory, user

Workflow Diagram:

Request arrives
|
v
[Per-dependency timeout]
|
v
Call downstream (payment)
|
+-- success within timeout --> proceed
|
+-- timeout/error --> count failure
|
v
[Circuit Breaker]
/ \
closed open
| |
allow call fail fast
| |
v v
continue fallback response

🔍 Behind the Question: Resilience design: timeouts, circuit breakers, bulkheads, graceful degradation, blast radius.

❌ The Red Flags: No timeouts; shared connection pool across all dependencies; retry storms without backoff.


Q72: Race condition — double charging customers on retry​

Level: Principal

Scenario: A client retries a POST /payment after a timeout. The server actually processed the first payment but the response was lost. The retry creates a second charge. The customer is charged twice.

Impact: Financial loss, refund overhead, damaged customer trust.

Root Cause: The payment endpoint is not idempotent. There is no idempotency key, so the server treats the retry as a brand-new payment.

Simple explanation: You pay at a cashier, the receipt printer jams, you do not know if it went through, so you pay again. The cashier charges you twice because they do not recognize it is the same purchase.

Technical Answer:

  1. Require an Idempotency-Key header on write endpoints.
  2. Store processed keys with results in Redis/DB with TTL.
  3. On retry with the same key, return the cached result instead of re-processing.

Code/Syntax Justification:

from fastapi import Header, HTTPException
from fastapi.responses import JSONResponse
import redis, json

r = redis.Redis(host="redis", port=6379, db=0)

@app.post("/payment")
async def payment(request: Request, idempotency_key: str = Header(...)):
cache_key = f"idem:{idempotency_key}"
cached = r.get(cache_key)
if cached:
return JSONResponse(content=json.loads(cached), status_code=200)

result = await process_payment(request)
r.setex(cache_key, 86400, json.dumps(result)) # 24h TTL
return result

Workflow Diagram:

Client POST /payment (Idempotency-Key: abc-123)
|
v
[Check Redis for idem:abc-123]
|
+-- FOUND --> return cached result (no re-charge)
|
+-- NOT FOUND --> process payment
|
v
store result in Redis (TTL 24h)
|
v
return result to client
|
v
Client retries with same Idempotency-Key: abc-123
|
v
[Redis HIT] --> return cached result (safe, no double charge)

🔍 Behind the Question: Idempotency design for distributed systems, exactly-once vs at-least-once, safe retries.

❌ The Red Flags: Tell clients not to retry; no idempotency key; storing keys forever without TTL.


Q73: Cache stampede after cache expiry​

Level: Senior

Scenario: A popular product page is cached in Redis with TTL 60 seconds. When it expires, 2000 concurrent requests all miss the cache and hit the database simultaneously to recompute.

Impact: Database spike. Query latency jumps. Possible DB outage.

Root Cause: No single-flight / locking. All concurrent requests see the missing key and all regenerate it.

Simple explanation: The store runs out of bread. 50 customers all ask the baker to bake more at once instead of one person baking and sharing.

Technical Answer:

  1. Short local lock (SET NX with short TTL) so only one request regenerates; others wait briefly and read the fresh value.
  2. Stale-while-revalidate: serve stale data while background regenerates.
  3. Probabilistic early expiration: recompute slightly before TTL expires.

Code/Syntax Justification:

import redis, time, json

r = redis.Redis()

def get_product(product_id):
key = f"product:{product_id}"
cached = r.get(key)
if cached:
return json.loads(cached)

lock_key = f"lock:{key}"
acquired = r.set(lock_key, "1", nx=True, ex=10)
if acquired:
try:
value = expensive_db_lookup(product_id)
r.setex(key, 60, json.dumps(value))
return value
finally:
r.delete(lock_key)
else:
time.sleep(0.2)
cached = r.get(key)
if cached:
return json.loads(cached)
return expensive_db_lookup(product_id)

Workflow Diagram:

Request A and B arrive (cache expired)
|
v
Both check cache -> MISS
|
v
A: SET NX lock -> OK (regenerates)
B: SET NX lock -> FAIL (lock held)
|
v
A: compute -> write cache (TTL 60s) -> delete lock
B: wait 200ms -> read cache HIT -> return fresh value
|
v
DB hit once instead of twice

🔍 Behind the Question: Cache stampede, single-flight locking, stale-while-revalidate.

❌ The Red Flags: No lock on regeneration; all clients hit DB on expiry; long lock TTL blocking others.


Q74: Database connection pool exhaustion under load​

Level: Senior

Scenario: A FastAPI app using SQLAlchemy works fine normally. During a traffic spike, all requests hang with "could not obtain connection from pool within 30 seconds." Database CPU is low.

Impact: All API calls hang or fail. Users see timeouts. Database is healthy but unreachable.

Root Cause: Connection pool size too small (default 5) while traffic creates many concurrent requests. Each request holds a connection for its entire duration. The pool empties and new requests queue.

Simple explanation: Five checkout counters. Hundreds of customers. Each counter stays occupied because customers pay slowly. New customers cannot check out.

Technical Answer:

  1. Size the pool based on concurrent worker count (e.g., pool_size = 2 × CPU cores × workers).
  2. Set pool_timeout to fail fast instead of hanging forever.
  3. Set pool_recycle and pool_pre_ping for stale connections.
  4. Keep DB transactions short — do not call external services inside a transaction.

Code/Syntax Justification:

from sqlalchemy import create_engine

engine = create_engine(
DATABASE_URL,
pool_size=20,
max_overflow=10,
pool_timeout=30,
pool_recycle=1800,
pool_pre_ping=True,
)

Workflow Diagram:

Request arrives
|
v
[Check connection pool]
|
+-- available --> checkout -> run query -> return to pool
|
+-- none available --> wait up to pool_timeout
|
+-- freed --> proceed
|
+-- timeout --> 503 Service Unavailable

🔍 Behind the Question: Connection pool sizing, fail-fast vs hang, short transactions.

❌ The Red Flags: Default pool_size=5 under high load; external API calls inside DB transactions; no pool timeout.


Q75: Replication lag — read replica returns stale data​

Level: Principal

Scenario: A user updates their profile and immediately refreshes. The update is not visible because the read replica lags behind the primary by 5–10 seconds.

Impact: Users see stale data. Confusing experience. Support tickets.

Root Cause: PostgreSQL streaming replication is asynchronous. WAL records ship to replicas after commit on primary. Under heavy write load, replicas fall behind.

Simple explanation: The primary notebook is updated instantly. The photocopy (replica) is updated minutes later. Readers of the photocopy see old content.

Technical Answer:

  1. Read-after-write consistency: route the user's own reads to the primary for a short window after their write.
  2. Synchronous replication for critical tables (synchronous_standby_names).
  3. Accept eventual consistency for non-critical reads; display "updated" timestamps.
  4. Monitor replication lag with pg_stat_replication.

Code/Syntax Justification:

class DatabaseRouter:
def __init__(self):
self.primary_until = {} # user_id -> timestamp

def get_connection(self, user_id, just_wrote=False):
if just_wrote:
self.primary_until[user_id] = time.time() + 5
return primary_conn
if user_id in self.primary_until and time.time() < self.primary_until[user_id]:
return primary_conn
return replica_conn

Workflow Diagram:

User UPDATE profile (Primary)
|
v
Primary commits
|
v
[Async WAL shipping to Replica]
|
+-- lag < 5s --> replica catches up
|
+-- lag > 5s --> replica stale

User READ profile
|
+-- within 5s of write --> route to Primary (read-your-writes)
|
+-- otherwise --> route to Replica (eventual consistency)

🔍 Behind the Question: Asynchronous replication, read-your-writes consistency, replication lag monitoring.

❌ The Red Flags: Assuming replicas always in sync; no lag monitoring; routing all reads to replicas regardless of freshness needs.


Q76: Deadlock under concurrent updates​

Level: Senior

Scenario: Two workers process orders at the same time. Worker A updates order 1 then order 2. Worker B updates order 2 then order 1. Both wait for each other. PostgreSQL aborts one with a deadlock error.

Impact: One worker fails. Orders need reprocessing. Retry logic must handle it.

Root Cause: Transactions acquire row locks in different orders, forming a cycle.

Simple explanation: Two people each hold one end of a rope and want the other end. Neither lets go. Stuck forever.

Technical Answer:

  1. Always acquire locks in a consistent global order (e.g., by ID ascending).
  2. Use SELECT ... FOR UPDATE to lock rows explicitly before updating.
  3. Keep transactions short to reduce lock hold time.
  4. Catch deadlock errors and retry with backoff.

Code/Syntax Justification:

ids = sorted([order1_id, order2_id])
with db.begin() as transaction:
db.execute("SELECT * FROM orders WHERE id = :id FOR UPDATE", {"id": ids[0]})
db.execute("SELECT * FROM orders WHERE id = :id FOR UPDATE", {"id": ids[1]})

Workflow Diagram:

Without fix: With fix (consistent order):
A locks 1 A locks 1 (lower id first)
B locks 2 B waits for 1
A waits for 2 ---> B gets 1 after A commits
B waits for 1 B locks 2
DEADLOCK No deadlock

🔍 Behind the Question: Lock ordering, deadlock detection, retry handling, SELECT FOR UPDATE.

❌ The Red Flags: No consistent lock order; no retry on deadlock; long-running transactions holding locks.


Q77: Connection storm — too many app instances connecting directly​

Level: Principal

Scenario: After autoscaling, 100 FastAPI instances each open 20 DB connections = 2000 connections. PostgreSQL max_connections is 200. New connections fail.

Impact: Application cannot connect to DB. Widespread errors.

Root Cause: Each app process maintains its own pool. Total connections exceed PostgreSQL max_connections.

Simple explanation: 100 people each calling one receptionist at once. The line is busy. Nobody gets through.

Technical Answer:

  1. Use a connection pooler (PgBouncer) between app and PostgreSQL. PgBouncer multiplexes many app connections into fewer DB connections.
  2. Set PgBouncer default_pool_size to match DB capacity (e.g., 150–200).
  3. Reduce per-instance pool size.
  4. Monitor pg_stat_activity for connection count by state.

Code/Syntax Justification:

[pgbouncer]
pool_mode = transaction
default_pool_size = 150
max_client_conn = 1000
reserve_pool_size = 20

Workflow Diagram:

100 App Instances PgBouncer PostgreSQL
| | |
|--- 20 conns each ->| |
| | multiplexes into |
| | 150 DB conns |
| |--------------------->|
| | |

🔍 Behind the Question: Connection pooling at scale, PgBouncer, max_connections planning.

❌ The Red Flags: Each app instance with large pool directly to DB; max_connections too low; no pooler.


Q78: Deploy causes widespread 500s — breaking change​

Level: Principal

Scenario: After a deploy, all API calls return 500. Rolling back the image fixes it. The change was a new required environment variable and a renamed database column.

Impact: Full site outage during deploy window.

Root Cause: Deploy was not backward-compatible. Old code expected the old column; new code expected the new env var. No feature flag, no canary, no rollback validation.

Simple explanation: You swap the engine in a car while it is driving. The car stops because new parts do not fit the old body yet.

Technical Answer:

  1. Backward-compatible DB changes (additive columns, dual-write during migration).
  2. Feature flags to toggle new behavior per deployment.
  3. Canary deploy (5% traffic), monitor error rate, then promote.
  4. Validate config at startup (fail fast if required env vars missing).

Code/Syntax Justification:

from pydantic import BaseSettings
from fastapi import FastAPI

class Settings(BaseSettings):
DATABASE_URL: str
NEW_FEATURE_FLAG: bool = False

class Config:
env_file = ".env"

settings = Settings()
app = FastAPI()

@app.on_event("startup")
async def validate():
if not settings.DATABASE_URL:
raise RuntimeError("DATABASE_URL is required")

Workflow Diagram:

Deploy new version
|
v
[Startup validation: env vars present?]
|
+-- NO --> fail fast, do not accept traffic
|
+-- YES --> canary rollout (5% traffic)
|
v
[Monitor error rate / latency]
|
+-- healthy --> promote to 100%
|
+-- unhealthy --> auto-rollback

🔍 Behind the Question: Zero-downtime deployment, backward compatibility, canary releases, operational maturity.

❌ The Red Flags: Breaking DB changes without dual-write; no rollback plan; no startup validation.


Q79: Vacuum and bloat — autovacuum not keeping up​

Level: Principal

Scenario: A high-write table grows to 3× expected size. Queries slow down. pg_total_relation_size() shows heavy bloat. Autovacuum runs but cannot keep up with UPDATE/DELETE volume.

Impact: Disk waste. Slower queries. Worse cache hit ratio.

Root Cause: Each UPDATE in PostgreSQL is DELETE + INSERT (MVCC). Dead tuples accumulate until vacuum removes them. If update rate exceeds vacuum rate, bloat grows.

Simple explanation: Every edit leaves a scrap of paper on the desk. The cleaner removes scraps, but if you write faster than the cleaner cleans, the desk gets buried.

Technical Answer:

  1. Tune autovacuum for high-write tables: lower autovacuum_vacuum_scale_factor.
  2. Increase autovacuum_max_workers and maintenance_work_mem.
  3. Use pg_repack or VACUUM FULL (requires lock) for severe bloat.
  4. Use fillfactor to leave space for HOT updates on frequently updated tables.

Code/Syntax Justification:

ALTER TABLE high_write_table SET (
autovacuum_vacuum_scale_factor = 0.05,
autovacuum_vacuum_cost_limit = 2000
);

SELECT schemaname, tablename, n_dead_tup, last_vacuum, last_autovacuum
FROM pg_stat_user_tables
WHERE n_dead_tup > 100000;

Workflow Diagram:

UPDATE row
|
v
Old row version marked dead
|
v
Autovacuum runs periodically
|
+-- keeps up --> dead tuples removed, size stable
|
+-- cannot keep up --> dead tuples accumulate = bloat
|
v
queries slower, disk wasted

🔍 Behind the Question: MVCC, autovacuum tuning, bloat management, fillfactor, HOT updates.

❌ The Red Flags: VACUUM FULL during peak (long exclusive lock); ignoring n_dead_tup; not tuning per high-write table.


Q80: Design a URL shortener for high read, low write traffic​

Level: Senior

Scenario: Design a service like bit.ly. Short URLs are read millions of times but created rarely. Need fast redirects, analytics, and custom aliases.

Impact: Must handle high read QPS with low latency. Writes are infrequent but must be reliable.

Technical Answer:

  1. Encoding: Base62 hash of the long URL + counter, or random 7-char code. Check for collisions.
  2. Storage: Primary key = short code, value = long URL. Redis cache for hot short codes; PostgreSQL as durable store.
  3. Read path: Check Redis cache first; on miss, read PostgreSQL and populate Redis with TTL.
  4. Write path: Generate code, insert into PostgreSQL, invalidate/seed Redis.
  5. Analytics: Async event logging to a stream (Kafka) processed separately, not on the redirect path.

Code/Syntax Justification:

import hashlib, base64
import string

def encode_url(long_url, salt="myapp"):
digest = hashlib.sha256((salt + long_url).encode()).digest()
code = base64.urlsafe_b64encode(digest)[:7].decode()
return code

Workflow Diagram:

GET /abc123
|
v
[Check Redis cache for code]
|
+-- HIT --> return redirect to long URL
|
+-- MISS --> query PostgreSQL
|
v
found --> populate Redis (TTL 24h) -> redirect
|
v
not found --> 404

🔍 Behind the Question: Read-heavy design, cache-aside, encoding/collisions, async analytics.

❌ The Red Flags: No cache; synchronous analytics on redirect path; no collision handling.


Q81: Design a rate limiter for an API gateway​

Level: Senior

Scenario: Protect a public API from abuse. Each API key has a limit of 1000 requests per minute. Must be accurate, distributed across gateway instances, and fail open if Redis is down.

Impact: Prevent abuse and ensure fair use. Legitimate users must not be blocked incorrectly.

Technical Answer:

  1. Algorithm: Fixed window (simple, Redis INCR with expiry) or sliding log (accurate, Redis sorted set) or token bucket (smooth, Redis Lua script).
  2. Distributed: Use Redis as the shared counter store across all gateway instances.
  3. Fail open: If Redis is unreachable, allow requests (or degrade gracefully) rather than blocking all traffic.
  4. Identify clients by API key or IP + user agent.

Code/Syntax Justification:

# Fixed window rate limiter with Redis
import redis, time

r = redis.Redis()
LIMIT = 1000
WINDOW = 60 # seconds

def allow_request(api_key):
key = f"ratelimit:{api_key}"
now = int(time.time())
window = now // WINDOW
field = f"{window}"
count = r.hincrby(key, field, 1)
if count == 1:
r.expire(key, WINDOW * 2)
return count <= LIMIT

Workflow Diagram:

Request arrives with API key
|
v
[Redis HINCRBY counter for key + window]
|
+-- count <= LIMIT --> allow request
|
+-- count > LIMIT --> return 429 Too Many Requests
|
v
[If Redis down] --> fail open (allow) or fail closed (block)

🔍 Behind the Question: Rate limiting algorithms, distributed counters, fail-open vs fail-closed.

❌ The Red Flags: Per-instance in-memory counters (not distributed); blocking all traffic when Redis down; no TTL on counters (memory leak).