RabbitMQ, Celery & ARQ - Interview Questions with Simple Answers
A complete guide for backend developers learning about message queues and task processing
Table of Contents
- Message Queue Fundamentals
- RabbitMQ Questions
- Celery Questions
- ARQ Questions
- Comparison Questions
- Real-World Scenarios
1. Message Queue Fundamentals
Basic Level
Q1: What is a Message Queue and why do we need it?
Answer: A Message Queue (MQ) is a form of asynchronous service-to-service communication used in serverless and microservices architectures. It provides a lightweight buffer that stores messages until they are processed by a receiving application.
Why we need it:
- Decoupling: Services can communicate without knowing where the other service is or how it works. They only need to know how to talk to the queue.
- Scalability: You can add more "Workers" (consumers) to process messages faster during peak traffic without changing the "Producer" code.
- Resilience (Fault Tolerance): If a consumer crashes, the message stays in the queue. It is not lost and will be processed once the consumer recovers or another worker picks it up.
- Smoothing Spikes: Instead of your database crashing when 1,000 users sign up at once, the requests sit in a queue and are processed one-by-one at a steady pace (Load Leveling).
Q2: How does a Message Queue work?
Answer: The lifecycle of a message typically follows these steps:
- Produce: An application (Producer) creates a message (e.g., "Send welcome email to user 123") and sends it to the MQ.
- Store: The Message Broker (e.g., RabbitMQ, Redis) stores the message on disk or in memory.
- Consume: A background worker (Consumer) connects to the queue and asks for the next available message.
- Acknowledge (ACK): Once the worker finishes the task, it sends an "ACK" signal back to the broker.
- Delete: The broker removes the message from the queue only after receiving the ACK.
Q3: What are the core components of a Message Queue system?
Answer:
- Producer (Publisher): The app that creates and sends the data.
- Consumer (Subscriber/Worker): The app that retrieves and processes the data.
- Message: The actual data package (Payload + Headers).
- Queue: The line or buffer where messages wait.
- Broker: The server software that manages the queues (e.g., RabbitMQ, Kafka).
- Exchange (Specific to RabbitMQ): An agent that receives messages from producers and routes them to different queues based on rules (bindings).
Q4: What is the difference between a Queue and a Topic?
Answer: This defines how messages are distributed to multiple consumers.
-
Queue (Point-to-Point):
- Each message is delivered to exactly one consumer.
- If there are multiple workers, the broker usually uses Round Robin to distribute tasks.
- Use Case: Task processing (e.g., "Generate PDF report").
-
Topic (Publish/Subscribe):
- One message can be delivered to multiple different subscribers.
- Each subscriber gets their own copy of the message.
- Use Case: Event notifications (e.g., "User signed up" -> Notify Email Service AND Notify Analytics Service AND Notify Slack).
Q5: What are the different message delivery patterns?
Answer: There are three main message delivery patterns:
1. Point-to-Point (P2P):
- One producer sends to one consumer
- Message is removed after processing
- Used for: Task processing, order fulfillment
# Producer sends to queue
channel.queue_declare(queue='orders')
channel.basic_publish(exchange='', routing_key='orders', body='Order #123')
# Consumer receives from queue
def callback(ch, method, properties, body):
process_order(body)
ch.basic_ack(delivery_tag=method.delivery_tag)
2. Publish-Subscribe (Pub/Sub):
- One producer sends, multiple consumers receive
- Each consumer gets its own copy
- Used for: Notifications, event broadcasting
# Producer publishes to exchange
channel.exchange_declare(exchange='notifications', type='fanout')
channel.basic_publish(exchange='notifications', routing_key='', body='New order!')
# Multiple consumers subscribe
channel.queue_declare(queue='email_queue')
channel.queue_bind(exchange='notifications', queue='email_queue')
3. Request-Reply:
- Producer sends a message and waits for a response
- Like making a function call but asynchronous
- Used for: RPC calls, getting data back
# Send request with reply queue
channel.queue_declare(queue='response_queue')
channel.basic_publish(
exchange='',
routing_key='request',
properties=pika.BasicProperties(
reply_to='response_queue',
correlation_id='123'
),
body='Get user data'
)
Q6: What are message delivery guarantees (semantics)?
Answer: Message queues can provide different delivery guarantees:
At-most-once (Fire and Forget):
- Message is delivered at most one time
- May be lost but never duplicated
- Use when: Missing messages are okay (metrics, logs)
- Risk: Messages might be lost
# No acknowledgment - message may be lost
channel.basic_publish(exchange='', routing_key='queue', body='data')
At-least-once (Default):
- Message is delivered at least one time
- May be duplicated but never lost
- Use when: Duplicates can be handled (idempotency)
- Risk: Messages might be duplicated
# With acknowledgment - message guaranteed but may retry
channel.basic_consume(queue='queue', on_message_callback=callback)
# Must ack after processing
ch.basic_ack(delivery_tag=method.delivery_tag)
Exactly-once (Hardest):
- Message is delivered exactly one time
- Neither lost nor duplicated
- Use when: Critical data (payments, orders)
- Implementation: Idempotent consumers + deduplication
# Use unique message IDs for deduplication
message_id = str(uuid.uuid4())
properties = pika.BasicProperties(message_id=message_id)
# Consumer checks if already processed
if not already_processed(message_id):
process_message()
mark_processed(message_id)
Q7: What are synchronous vs asynchronous messaging?
Answer:
Synchronous Messaging:
- Sender waits for receiver to process the message
- Like making a phone call - you wait for the other person to answer
- Use when: You need an immediate response
# Synchronous - waits for response
result = api_call(data) # Blocks until done
Asynchronous Messaging:
- Sender sends message and continues immediately
- Like sending an email - you don't wait for a reply
- Use when: Response isn't needed immediately
# Asynchronous - returns immediately
queue.send(data) # Returns right away
# Processing happens in background
Q8: What are the benefits of using Message Queues?
Answer: Benefits of using Message Queues:
-
Loose Coupling: Components don't need to know about each other. They just communicate through the queue.
-
Scalability: Easily add more consumers to handle increased load.
-
Reliability: Messages are persisted until processed, so no data is lost if systems crash.
-
Performance: Producers don't wait for consumers, improving overall throughput.
-
Load Balancing: Work is distributed across multiple consumers.
-
Fault Tolerance: If one consumer fails, other consumers can pick up the work.
-
Order Preservation: Messages can be processed in order (FIFO - First In, First Out).
-
Decoupling: You can update or replace producers and consumers independently.
Q9: What are some popular Message Queue systems?
Answer: Popular Message Queue systems include:
RabbitMQ:
- Implementation: AMQP protocol
- Best for: Complex routing, enterprise applications
- Pros: Rich features, flexible routing
- Cons: Slower than some alternatives
Apache Kafka:
- Implementation: Distributed event streaming
- Best for: High-throughput streaming, event sourcing
- Pros: Very fast, persistent logs
- Cons: Complex setup, higher latency for simple use cases
Redis (with streams):
- Implementation: In-memory data store with queue features
- Best for: Simple use cases, caching
- Pros: Fast, simple
- Cons: Not as robust as dedicated message queues
Amazon SQS:
- Implementation: Managed cloud service
- Best for: AWS users, managed infrastructure
- Pros: Fully managed, highly available
- Cons: Vendor lock-in, costs at scale
ActiveMQ:
- Implementation: JMS (Java Message Service)
- Best for: Enterprise Java applications
- Pros: Feature-rich
- Cons: Older, slower than alternatives
Q10: When should you use a Message Queue?
Answer: Use a Message Queue when:
- Background Processing: Tasks that take time (sending emails, processing images)
- Microservices Communication: Services need to talk to each other without direct calls
- Load Leveling: Handle traffic spikes by queuing requests
- Event-Driven Architecture: One event triggers multiple actions
- Task Scheduling: Run tasks at specific times or intervals
- Batch Processing: Process large amounts of data in chunks
Don't use a Message Queue when:
- Simple Cases: Just use direct function calls
- Real-time Needs: When latency matters (sub-second responses)
- Strong Consistency Required: Use synchronous calls instead
- Simple CRUD Operations: Database transactions are simpler
Intermediate Level
Q11: What is Message Queue architecture?
Answer: Message Queue architecture consists of several layers:
1. Application Layer:
- Producers create messages
- Consumers process messages
- Business logic lives here
2. Protocol Layer:
- AMQP (RabbitMQ): Advanced features, complex routing
- STOMP (RabbitMQ): Simple text-based protocol
- MQTT: Lightweight, for IoT
- Kafka Protocol: Binary protocol for high throughput
3. Broker Layer:
- Message routing
- Queue management
- Message persistence
- Connection handling
4. Storage Layer:
- In-memory (fast, but volatile)
- Disk-based (slower, but persistent)
- Hybrid (lazy writing to disk)
Q12: What is message serialization?
Answer: Message serialization converts data structures into a format that can be stored or transmitted. Common formats include:
JSON:
- Human-readable
- Widely supported
- Larger size
import json
message = json.dumps({'data': 'value'})
MessagePack (msgpack):
- Binary format
- Smaller than JSON
- Faster parsing
import msgpack
message = msgpack.packb({'data': 'value'})
Protocol Buffers:
- Schema-based
- Very compact
- Requires schema definition
Pickle (Python only):
- Can serialize any Python object
- Security risks (don't use with untrusted data)
Q13: What is a Dead Letter Queue (DLQ)?
Answer: A Dead Letter Queue is a special queue that stores messages that can't be processed successfully. Messages go to DLQ when:
- They exceed maximum retry attempts
- They expire (TTL)
- Processing fails with non-retryable error
- Queue reaches maximum length
Why use DLQ:
- Prevent losing failed messages
- Analyze why messages failed
- Manual intervention for problematic messages
- Audit trail for debugging
# Set up DLQ
channel.queue_declare(
queue='main_queue',
arguments={
'x-dead-letter-exchange': 'dlx',
'x-dead-letter-routing-key': 'dead_messages'
}
)
# Create DLQ
channel.queue_declare(queue='dead_messages')
Q14: What is Message TTL (Time To Live)?
Answer: TTL determines how long a message lives in the queue before being deleted. It's like an expiration date.
Why use TTL:
- Auto-cleanup of old messages
- Prevent queue from filling up
- Message becomes stale after a certain time
- Implement message expiration patterns
# Queue-level TTL (all messages)
channel.queue_declare(
queue='temp_data',
arguments={'x-message-ttl': 3600000} # 1 hour
)
# Message-level TTL (specific message)
channel.basic_publish(
exchange='',
routing_key='temp_data',
body='expires soon',
properties=pika.BasicProperties(expiration='60000') # 60 seconds
)
Q15: What is Message Priority?
Answer: Priority queues allow important messages to be processed first. Higher priority messages jump ahead of lower priority ones.
# Create priority queue (0-10)
channel.queue_declare(
queue='urgent_queue',
arguments={'x-max-priority': 10}
)
# Send high priority message
channel.basic_publish(
exchange='',
routing_key='urgent_queue',
body='URGENT: Server down!',
properties=pika.BasicProperties(priority=10)
)
# Send low priority message
channel.basic_publish(
exchange='',
routing_key='urgent_queue',
body='Update logs',
properties=pika.BasicProperties(priority=1)
)
Advanced Level
Q16: What is Message Queue Clustering?
Answer: Clustering combines multiple message queue servers to work as one. Benefits include:
- High Availability: If one server fails, others take over
- Scalability: Handle more messages by adding servers
- Load Balancing: Distribute load across nodes
Types of Clustering:
1. Active-Passive (Master-Slave):
- One active server handles all traffic
- Passive servers standby for failover
- Simple but wastes resources
# RabbitMQ clustering
rabbitmqctl join_cluster rabbit@node1
2. Active-Active:
- All servers handle traffic actively
- More complex but better resource utilization
- Requires conflict resolution
3. Distributed Queues:
- Queue data split across nodes (sharding)
- Each node handles subset of messages
- Good for very high throughput
Q17: What is Message Queue Federation?
Answer: Federation connects message queues across different locations or organizations. Unlike clustering (nodes in same network), federation works over wide area networks.
Use cases:
- Connect data centers in different regions
- Share messages between organizations
- Distribute load geographically
# RabbitMQ Federation
rabbitmqctl set_policy federation_exchange \
"^amq\." \
'{"federation-upstream":"upstream-name"}' \
--apply-to exchanges
Q18: What is exactly-once delivery and how to achieve it?
Answer: Exactly-once is the hardest delivery guarantee. It requires:
- Producer idempotency: Same message sent only once
- Consumer idempotency: Same message processed only once
Implementation strategies:
# 1. Producer: Add unique ID
message_id = str(uuid.uuid4())
channel.basic_publish(
exchange='',
routing_key='queue',
body=json.dumps(data),
properties=pika.BasicProperties(message_id=message_id)
)
# 2. Consumer: Check for duplicates
processed = set()
def process_message(ch, method, properties, body):
msg_id = properties.message_id
if msg_id in processed:
ch.basic_ack(delivery_tag=method.delivery_tag)
return
# Process message
do_work(body)
processed.add(msg_id)
ch.basic_ack(delivery_tag=method.delivery_tag)
Q19: What is the difference between RabbitMQ and Kafka?
Answer: RabbitMQ is a Traditional Message Broker, while Kafka is a Distributed Event Streaming Platform. They are built for different architectural goals.
| Feature | RabbitMQ | Kafka |
|---|---|---|
| Model | Push: Broker pushes to consumers. | Pull: Consumers pull at their own pace. |
| Retention | Messages deleted after ACK. | Messages persisted (stored) for days/years. |
| Routing | Complex routing (Exchanges/Bindings). | Simple routing (Topics/Partitions). |
| Ordering | Guaranteed within a queue. | Guaranteed within a partition. |
| Performance | ~20k msgs/sec (High features). | ~1M+ msgs/sec (High throughput). |
| Replay | Not possible (one-time use). | Replayable: Rewind to read old events. |
Practical Analogy:
- RabbitMQ is like a Post Office: It takes your letter, finds the right address, delivers it, and once it's confirmed, the letter is gone.
- Kafka is like a Library: Books (events) are added to shelves (topics). Readers (consumers) can come in, read books at their own speed, and even go back and re-read books that were added last year.
When to choose what?
- RabbitMQ: For standard background tasks (sending emails), complex routing logic, and where low latency for individual messages is key.
- Kafka: For big data processing, logging, activity tracking, and where you need to store and analyze "streams" of events over time.
Q20: How do you secure a Message Queue?
Answer: Security measures for message queues:
1. Authentication:
# RabbitMQ with credentials
credentials = pika.PlainCredentials('user', 'password')
connection = pika.BlockingConnection(
pika.ConnectionParameters('localhost', credentials=credentials)
)
2. Authorization (Permissions):
# RabbitMQ user permissions
rabbitmqctl set_permissions -p /myvhost user ".*" ".*" ".*"
3. SSL/TLS Encryption:
# RabbitMQ SSL
ssl_options = {
'certfile': '/path/to/cert.pem',
'keyfile': '/path/to/key.pem',
'cafile': '/path/to/ca.pem'
}
connection = pika.BlockingConnection(
pika.ConnectionParameters('localhost', ssl_options=ssl_options)
)
4. Virtual Hosts (vhosts):
- Separate different applications
- Each vhost has own queues and permissions
5. Message Encryption:
- Encrypt sensitive data before putting in message
- Use AES encryption for payload
2. RabbitMQ Questions
Basic Level
Q21: What is RabbitMQ and why do we need it?
Answer: RabbitMQ is a message broker - think of it like a postal service for computer programs. When one part of your application needs to send data to another part, RabbitMQ handles the delivery. It's useful because:
- It separates the sender from the receiver, so both can work independently
- If the receiver is busy, messages wait in line until it's ready
- If the receiver crashes, messages are saved and processed when it recovers
Q22: What are the basic parts of RabbitMQ?
Answer: The main parts are:
- Producer: The application that creates and sends a message
- Consumer: The application that receives and processes the message
- Queue: A waiting area where messages stay until someone picks them up
- Exchange: A distribution point that decides which queue should get the message
- Binding: A connection that tells the exchange which queues to use
Q23: What types of exchanges does RabbitMQ have?
Answer: There are four main types:
- Direct: Sends message to one specific queue that matches exactly
- Fanout: Copies the message to ALL connected queues
- Topic: Uses patterns (like wildcards) to send to matching queues
- Headers: Looks at message properties instead of the routing key
Q24: What is the difference between a queue and an exchange?
Answer: Think of it like a post office:
- Exchange is like the sorting desk - it receives mail and decides where to send it
- Queue is like the mailbox - it holds the mail until someone picks it up
The exchange decides WHERE to send, the queue actually STORES the message.
Q25: What happens when you publish a message in RabbitMQ?
Answer: Here's the simple flow:
- Your code sends a message to an exchange
- The exchange looks at the routing rules (bindings)
- Based on the rules, it sends the message to the right queue(s)
- The message waits in the queue
- When a consumer is ready, it takes the message from the queue
- After processing, the consumer confirms the message was handled
- RabbitMQ removes the message from the queue
Q26: What is message acknowledgment?
Answer: When a consumer receives a message, it can tell RabbitMQ "I got this" (acknowledge) or "I couldn't handle this" (reject). This is important because:
- If a consumer crashes after receiving but before processing, the message won't be lost
- The message goes back to the queue for someone else to try
- It ensures every message is processed at least once
Q27: What is the difference between persistent and transient messages?
Answer:
- Persistent: Saved to disk immediately. Survives if RabbitMQ restarts. Takes longer to process because of disk writing.
- Transient: Kept only in memory. Lost if RabbitMQ restarts. Faster to process.
Use persistent when you can't afford to lose messages. Use transient for quick, less important messages.
Q28: What is a virtual host (vhost)?
Answer: A vhost is like having separate post offices within one RabbitMQ server. Each vhost has its own:
- Queues
- Exchanges
- User permissions
This lets different applications share one RabbitMQ while staying completely separate.
Q29: How do you check if RabbitMQ is running?
Answer: On Linux:
sudo systemctl status rabbitmq-server
Or check the management interface at: http://localhost:15672 (needs to be enabled first).
Q30: How do you send a message using Python?
Answer: Using the pika library:
import pika
# Connect to RabbitMQ
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Create a queue
channel.queue_declare(queue='hello')
# Send a message
channel.basic_publish(
exchange='',
routing_key='hello',
body='Hello World!'
)
print("Message sent")
connection.close()
Intermediate Level
Q31: What is prefetch and why does it matter?
Answer: Prefetch limits how many messages a consumer can grab at once. It's like a waiter not bringing all dishes at once - they bring a few, serve them, then bring more.
If prefetch is 10, a consumer can hold up to 10 unprocessed messages. This prevents:
- One busy consumer from getting overwhelmed
- Messages piling up while other consumers are idle
# Only accept 10 messages at a time
channel.basic_qos(prefetch_count=10)
Q32: What is a Dead Letter Queue?
Answer: A Dead Letter Queue (DLQ) is a place for messages that can't be processed. Messages go there when:
- They are rejected without being requeued
- They expire (too old)
- The queue is full and new messages arrive
This lets you inspect failed messages instead of losing them.
Q33: What is TTL (Time To Live)?
Answer: TTL sets how long a message lives before being deleted. It's like a milk expiration date.
# Queue-level: messages expire after 10 seconds
channel.queue_declare(
queue='my_queue',
arguments={'x-message-ttl': 10000}
)
Q34: What are priority queues?
Answer: Priority queues let important messages jump to the front of the line. Higher priority messages get processed first.
# Create a queue that supports 10 priority levels (0-10)
channel.queue_declare(
queue='important_queue',
arguments={'x-max-priority': 10}
)
Q35: How do you make queues highly available?
Answer: Queue mirroring copies each queue to multiple RabbitMQ servers. If one server fails, another has the data.
# Enable mirroring for queues starting with "ha-"
rabbitmqctl set_policy ha-all "^ha\." '{"ha-mode":"all"}'
Q36: What is lazy queue mode?
Answer: Lazy queues store all messages on disk immediately, keeping minimal data in memory. This is good when:
- You have millions of messages in a queue
- You want to reduce memory usage
- You're okay with slightly slower performance
Q37: What is publisher confirm?
Answer: Publisher confirm is like getting a delivery receipt. When you publish a message, RabbitMQ confirms it was received. Without this, you don't know if your message actually arrived.
Q38: How do you handle connection problems?
Answer: Use automatic recovery:
connection = pika.BlockingConnection(
pika.ConnectionParameters(
'localhost',
heartbeat=60,
blocked_connection_timeout=300,
connection_attempts=3,
retry_delay=2
)
)
Q39: What is the difference between clustering and federation?
Answer:
-
Clustering: Multiple RabbitMQ servers working as one. All share the same queues. Good for high availability within a data center.
-
Federation: Servers in different locations share messages. Good for connecting data centers or different organizations.
Q40: How do you monitor RabbitMQ?
Answer: Use the management plugin:
# Enable
rabbitmq-plugins enable rabbitmq_management
# Access at
http://localhost:15672
Advanced Level
Q41: How do you optimize RabbitMQ for high speed?
Answer: Speed optimization tips:
- Use faster disks: SSD instead of HDD
- Increase memory: More RAM = more messages in cache
- Tune prefetch: Balance between speed and fairness
- Use lazy queues: For large queues
- Batch publishing: Send multiple messages together
- Connection pooling: Reuse connections instead of creating new ones
- Enable TCP optimizations: Use nodelay and keepalive
Q42: What are quorum queues?
Answer: Quorum queues are a newer way to keep messages safe. They use a voting system (like Raft) to ensure messages are saved to multiple servers before confirming delivery. They're more reliable than regular mirrored queues.
Q43: How do you handle memory problems?
Answer: If RabbitMQ uses too much memory:
- Check what's using memory:
rabbitmqctl status | grep memory - Lower memory limit:
rabbitmqctl set_vm_memory_high_watermark 0.6 - Use lazy queues to keep less in memory
- Set queue limits to prevent unbounded growth
Q44: What is Stream in RabbitMQ?
Answer: Streams are a new feature for handling huge amounts of data. Unlike regular queues (where messages disappear after reading), streams let multiple consumers read the same messages. They're great for:
- Event logging
- Audit trails
- Processing data by multiple systems
Q45: How do you handle exactly-once delivery?
Answer: True exactly-once is hard to achieve. Here's a practical approach:
- Give each message a unique ID
- Track which IDs you've processed
- Ignore duplicates
Q46: How does RabbitMQ handle network problems?
Answer: When network fails, RabbitMQ can:
- Pause minority: Stop the smaller group of servers to prevent confusion
- Auto-merge: Automatically reconnect when network recovers
- Manual fix: Let administrator decide
The goal is preventing "split-brain" where two groups think they're the main server.
Q47: How do you migrate messages between servers?
Answer: Use the Shovel plugin:
rabbitmqctl set_shovel migration_name \
--rabbitmq-amqp-uri="amqp://source-server" \
--rabbitmq-queue="source_queue" \
--rabbitmq-delete-after="never"
Q48: What is the difference between RabbitMQ and Kafka?
Answer:
| Feature | RabbitMQ | Kafka |
|---|---|---|
| How it works | Message queue | Event log |
| Message deletion | After reading | Configurable retention |
| Speed | Good | Very fast |
| Complexity | Simpler | More complex |
| Best for | Task queues | Event streams |
Q49: How do you secure RabbitMQ?
Answer: Security steps:
- Use credentials: Require username/password
- Use SSL: Encrypt connections
- Use vhosts: Separate different applications
- Limit permissions: Give users minimal access needed
Q50: How do you design for millions of messages?
Answer: When scaling to millions:
- Use lazy queues - keeps memory low
- Add more consumers - parallel processing
- Use quorum queues - better reliability
- Monitor closely - watch memory and disk
- Consider partitioning - split across servers
- Use Streams - for read-heavy workloads
3. Celery Questions
Basic Level
Q51: What is Celery and why is it used?
Answer: Celery is an asynchronous task queue/job queue based on distributed message passing. It is used to execute work outside the HTTP request-response cycle.
Common Use Cases:
- Heavy Computations: Image/video processing.
- Third-party APIs: Sending emails, triggering webhooks.
- Scheduled Jobs: Cleaning up a database every midnight.
- Retrying: Automatically retrying a failed task with exponential backoff.
Q52: What are the main components of a Celery architecture?
Answer:
- Producer: Your web app (e.g., Django/FastAPI) that creates a task.
- Broker: The transport mechanism (RabbitMQ or Redis) that stores the task.
- Worker: A separate process that constantly monitors the broker and executes tasks.
- Result Backend: Where the result of the task is stored (e.g., Postgres, Redis) so the web app can check it later.
- Beat: A scheduler that kicks off tasks at regular intervals.
Q53: How do you define and call a Celery task?
Answer:
from celery import Celery
app = Celery('tasks', broker='redis://localhost:6379/0')
@app.task
def add(x, y):
return x + y
# 1. Asynchronous call (Non-blocking)
# This returns a result object immediately, while work happens in background
result = add.delay(4, 4)
# 2. Getting the result (Blocking)
print(result.get(timeout=1)) # 8
Q54: Difference between .delay() and .apply_async()?
Answer:
.delay(): A shortcut for.apply_async(). It is simpler but doesn't allow many options.- Usage:
add.delay(10, 20)
- Usage:
.apply_async(): The more powerful method. It allows you to specify execution options like:countdown: Wait X seconds before running.eta: Run at a specific date/time.queue: Send to a specific worker group.retry: Custom retry logic for this specific call.- Usage:
add.apply_async((10, 20), countdown=60)(runs in 1 minute)
Q55: What are task states?
Answer: Tasks go through different states:
- PENDING - Created but not started yet
- STARTED - Worker picked it up
- SUCCESS - Finished without errors
- FAILURE - Something went wrong
- RETRY - Trying again after failure
- REVOKED - Cancelled
Q56: How do you get the result of a task?
Answer:
task = process_data.delay(data)
result = task.get(timeout=30)
Q57: How do you start a Celery worker?
Answer:
celery -A myapp worker --loglevel=info
Q58: What is the difference between Celery and a simple thread?
Answer:
| Aspect | Celery | Thread |
|---|---|---|
| Runs where | Any computer | Same computer |
| Survives crash | Yes | No |
| Scales | Easy | Hard |
| Memory | Separate per worker | Shared |
Q59: How do you configure Celery?
Answer:
broker_url = 'redis://localhost:6379/0'
result_backend = 'redis://localhost:6379/1'
task_serializer = 'json'
task_time_limit = 300
Q60: What does bind=True mean in a task?
Answer: It gives the task access to itself (self). This lets you retry the task, update state/progress, and access request information.
Intermediate Level
Q61: How do you make a task retry on failure?
Answer:
@app.task(bind=True, max_retries=3)
def process_payment(self, payment_id):
try:
return call_payment_gateway(payment_id)
except PaymentError as exc:
raise self.retry(exc=exc, countdown=2 ** self.request.retries)
Q62: What are task routes?
Answer: Routes let you send different tasks to different queues:
task_routes = {
'tasks.send_email': {'queue': 'emails'},
'tasks.generate_report': {'queue': 'reports'},
}
Q63: What is Celery Canvas?
Answer: Canvas lets you chain tasks together:
- Chain: Run tasks one after another
- Group: Run tasks in parallel
- Chord: Run tasks, then a callback
Q64: How do you schedule periodic tasks?
Answer: Use Celery Beat:
beat_schedule = {
'cleanup-every-hour': {
'task': 'tasks.cleanup',
'schedule': 3600,
},
}
Q65: What is serialization in Celery?
Answer: Serialization converts Python objects to bytes. Options: JSON (recommended), Pickle (less secure), msgpack (smaller and faster).
Q66: How do you handle timeouts?
Answer:
@task(soft_time_limit=60, time_limit=120)
def quick_task():
pass
Q67: What are prefork, eventlet, and solo pools?
Answer:
- prefork: Multiple processes. Good for CPU-heavy tasks. Default.
- eventlet/gevent: Lightweight threads. Good for I/O-heavy tasks.
- solo: Single process. Simple but no parallelism.
Q68: How do you monitor Celery?
Answer: Use Flower:
pip install flower
celery -A myapp flower --port=5555
Q69: What are Celery signals?
Answer: Signals let you run code when things happen:
@signals.task_postrun.connect
def after_task(task_id, task, *args, **kwargs):
print(f"Finished {task.name}")
Q70: What is max_tasks_per_child?
Answer: It restarts a worker after processing a certain number of tasks. Prevents memory leaks.
worker_max_tasks_per_child = 1000
Advanced Level
Q71: How do you implement exactly-once processing?
Answer: Make tasks idempotent - safe to run multiple times by checking if already processed.
Q72: How do you handle distributed transactions?
Answer: Use the Saga pattern - do work, and if something fails, undo in reverse.
Q73: How do you secure Celery?
Answer:
- Use password for Redis
- Don't store sensitive results
- Validate inputs
- Rate limit tasks
Q74: How do you prevent task overlaps?
Answer: Use distributed locks with Redis.
Q75: How do you scale Celery?
Answer: Run multiple workers with different queues and pool types based on workload.
Q76: What is Celery's error handling?
Answer: Use auto-retry for specific exceptions, manual recovery with checkpoints.
Q77: How do you upgrade Celery tasks?
Answer: Version your tasks so old and new versions can coexist.
Q78: How do you debug Celery tasks?
Answer: Enable verbose logging and use remote debugger (ipdb).
Q79: What are common Celery issues?
Answer: Common problems:
- Task never executes - Check broker connection
- Tasks pile up - Add more workers
- Memory grows - Use max_tasks_per_child
- Lost tasks - Use result backend
- Tasks execute twice - Make them idempotent
Q80: When would you NOT use Celery?
Answer: Don't use Celery when:
- Simple - just use threading
- Need sub-second latency
- Single server, simple needs
- Not Python
4. ARQ Questions
Basic Level
Q81: What is ARQ?
Answer: ARQ (Async Redis Queue) is a lightweight, high-performance job queue for Python that is built specifically for asyncio and Redis. It is designed to be a modern, async-first alternative to Celery.
Q82: How is ARQ different from Celery?
Answer:
- Native Async: ARQ is built on top of
asyncio. You don't needgeventoreventletto run many I/O-bound tasks concurrently. - Simplicity: ARQ has a much smaller codebase and fewer dependencies.
- Performance: For many small async tasks, ARQ is significantly faster and more memory-efficient than Celery.
- Persistence: ARQ uses Redis exclusively, whereas Celery supports multiple brokers.
Q83: How do you set up an ARQ worker?
Answer: You define a settings class that specifies the tasks and Redis connection.
Example:
# tasks.py
async def say_hello(ctx, name):
return f"Hello {name}"
class WorkerSettings:
functions = [say_hello]
redis_settings = RedisSettings()
# Command to run:
# arq tasks.WorkerSettings
Q84: How do you define an ARQ task?
Answer:
from arq import app
@app.task
async def process_data(data: dict):
result = await async_operation(data)
return result
Q85: How do you enqueue a task?
Answer:
job = await process_data.delay(redis, {'key': 'value'})
result = await job.result(timeout=30)
Q86: What is the difference between delay() and enqueue_job()?
Answer: delay() is quick way on the task function itself. enqueue_job() is more flexible, specify task by name.
Q87: How do you get task results?
Answer:
job = await my_task.delay(redis, data)
result = await job.result(timeout=30)
Q88: How do you start an ARQ worker?
Answer:
python -m arq worker.tasks:worker
Q89: What does concurrent_tasks do?
Answer: Controls how many tasks run at the same time.
Q90: What is ARQ health check?
Answer: Verifies the worker is running.
Intermediate Level
Q91: How do you retry failed tasks?
Answer: Use Retry class:
from arq import Retry
raise Retry(countdown=10, args=[attempt + 1])
Q92: How does ARQ scheduling (cron) work?
Answer:
@app.cron(cron(hour=8, minute=0))
async def daily_task(ctx):
await generate_report()
Q93: How do you set task timeouts?
Answer:
@app.task(timeout=30)
async def timed_task(data):
pass
Q94: How do you use queues for priority?
Answer:
@app.task(queue='high_priority')
async def urgent_task():
pass
Q95: How do you use ARQ with FastAPI?
Answer: Integrate ARQ with FastAPI using lifespan events.
Q96: How do you chain tasks?
Answer: Use chaining or sequential execution with job.wait().
Q97: What are ARQ worker settings?
Answer: Settings include redis_settings, concurrent_tasks, max_jobs, job_timeout, keep_job_result.
Q98: How do you monitor ARQ?
Answer: Use basic stats from Redis or add Prometheus metrics.
Q99: What serialization does ARQ use?
Answer: ARQ uses msgpack by default.
Q100: How do you use multiple queues?
Answer:
await urgent_task.delay(redis, data, queue='high')
Advanced Level
Q101: How do you implement exactly-once in ARQ?
Answer: Use idempotency keys with Redis.
Q102: How do you implement distributed locks?
Answer: Use Redis-based distributed locks.
Q103: What is a circuit breaker?
Answer: Prevents cascading failures by stopping calls to failing services.
Q104: How do you do bulk processing?
Answer: Process in chunks using batched operations.
Q105: How do you optimize ARQ performance?
Answer: Tune concurrent_tasks, sync_thread_pool_size, keep_job_result.
Q106: How do you scale ARQ?
Answer: Run multiple workers on different queues.
Q107: How do you handle backpressure?
Answer: Check queue size before adding new tasks.
Q108: How do you prioritize tasks?
Answer: ARQ handles this internally with queue_name ordering.
Q109: What are ARQ best practices?
Answer: Use appropriate timeouts, concurrent tasks, health checks.
Q110: How do you migrate from Celery to ARQ?
Answer: Convert tasks to async, update enqueue calls.
5. Comparison Questions
Q111: When should you use RabbitMQ directly vs Celery?
Answer:
Use RabbitMQ directly when:
- You need complex routing
- Working with multiple languages
- Need fine-grained control
Use Celery when:
- Python-only
- Need scheduling (Beat)
- Want simpler API
Q112: When should you choose Celery vs ARQ?
Answer:
Choose Celery when:
- Complex workflows (chains, chords)
- Need Django integration
- Scheduling features needed
Choose ARQ when:
- Using FastAPI or asyncio
- Simpler needs
- Performance critical
Q113: Which is fastest?
Answer: Generally: ARQ > RabbitMQ > Celery
Q114: Which is easiest to set up?
Answer: ARQ > Celery > RabbitMQ
6. Real-World Scenarios
Q115: Design a notification system
Answer: Use task queue for async notification delivery with priority queues.
Q116: Design an image processing pipeline
Answer: Use Celery chains for sequential processing steps.
Q117: Design a payment system
Answer: Use RabbitMQ with persistent messages and exactly-once processing.
Q118: Design a rate limiter
Answer: Use Redis-based sliding window rate limiting.
Quick Reference
RabbitMQ
# Start
sudo systemctl start rabbitmq-server
# Management UI
rabbitmq-plugins enable rabbitmq_management
# Check status
rabbitmqctl status
Celery
# Worker
celery -A project worker -l info
# Beat (scheduler)
celery -A project beat
# Flower (monitoring)
celery -A project flower
ARQ
# Worker
python -m arq worker.tasks:worker
# Watch mode
python -m arq worker.tasks:worker --watch
End of Guide