Temporal.io Workflow Engine - Interview Preparation (2026 Edition)
Total Questions: 61 Curriculum: Core Fundamentals (100), Intermediate (120), Advanced (80)
Core Fundamentals
Q1: What is Temporal.io and how does it work?
Technical Answer: Temporal is a workflow orchestration platform that executes long-running business logic reliably. It uses event-sourced state machines where each workflow step is recorded as an event, ensuring exactly-once execution even during failures.
Code Example:
// Go example
package main
import (
"go.temporal.io/sdk/workflow"
)
func UserRegistrationWorkflow(ctx workflow.Context, userData UserData) error {
// Each step is durably executed
err := workflow.ExecuteActivity(ctx, ValidateUserData, userData).Get(ctx, nil)
if err != nil {
return err
}
err = workflow.ExecuteActivity(ctx, SendWelcomeEmail, userData.Email).Get(ctx, nil)
if err != nil {
return err
}
return nil
}
# Python example
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id: str) -> str:
# Retry policy built-in
await workflow.execute_activity(
process_payment,
order_id,
start_to_close_timeout=timedelta(seconds=30)
)
await workflow.execute_activity(
ship_order,
order_id,
start_to_close_timeout=timedelta(minutes=5)
)
return "completed"
🔍 Interviewer Notes: Testing understanding of durable execution, event sourcing, and reliability patterns.
❌ Common Mistakes:
- Confusing with traditional message queues
- Not understanding exactly-once semantics
- Thinking it's just a scheduler
Q2: What are Activities in Temporal workflows?
Technical Answer: Activities are the basic units of work in Temporal. They are stateless, retryable functions that perform the actual business logic. Activities can be short (milliseconds) or long-running (hours), and Temporal tracks their state durably.
Code Example:
// Java example with retry policy
@ActivityMethod
@Retryable(maximumAttempts = 5, backoff = @Backoff(delay = 2000))
public String chargeCreditCard(PaymentInfo info) {
// Business logic here
return paymentService.charge(info);
}
# Python with timeout and retry config
@activity.defn
async def fetch_external_api(data: dict) -> dict:
async with aiohttp.ClientSession() as session:
async with session.post("https://api.example.com", json=data) as resp:
return await resp.json()
🔍 Interviewer Notes: Testing knowledge of retry policies, idempotency, and activity failure handling.
❌ Common Mistakes:
- Not making activities idempotent
- Setting wrong timeouts
- Ignoring version evolution
Q3: How does Temporal handle failures and retries?
Technical Answer: Temporal uses event sourcing to record state changes. When an activity fails, Temporal automatically retries based on the configured policy. If the worker dies, another worker can pick up the workflow from the last recorded event, ensuring exactly-once execution semantics.
Code Example:
# Custom retry policy
@activity.defn
def risky_operation(data: dict) -> dict:
# This will be retried up to 3 times with exponential backoff
if random.random() < 0.7:
raise Exception("Transient failure")
return {"status": "success"}
# Workflow with per-activity retry config
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id: str):
await workflow.execute_activity(
process_payment,
order_id,
retry_policy=RetryPolicy(
maximum_attempts=3,
initial_interval=timedelta(seconds=1),
maximum_interval=timedelta(seconds=30)
)
)
🔍 Interviewer Notes: Testing understanding of failure recovery and exactly-once semantics.
❌ Common Mistakes:
- Not configuring retry policies
- Activities not being idempotent
- Long retry intervals causing delays
Q4: What are Temporal Workflows vs traditional state machines?
Technical Answer: Traditional state machines require you to persist state manually and handle failures yourself. Temporal workflows are code-first - you write normal functions, and Temporal handles persistence, versioning, and recovery automatically through event sourcing.
Comparison Table:
| Aspect | Traditional State Machine | Temporal Workflow |
|---|---|---|
| State Persistence | Manual implementation | Automatic via event logs |
| Failure Recovery | Custom logic | Built-in durability |
| Scaling | Manual partitioning | Automatic load balancing |
| Versioning | Breaking changes | Backward compatible |
| Monitoring | Custom dashboards | Built-in observability |
🔍 Interviewer Notes: Testing architectural understanding and migration patterns.
❌ Common Mistakes:
- Treating Temporal like a simple queue
- Not leveraging versioning capabilities
- Underestimating operational simplicity
Q5: How do you version Temporal workflows safely?
Technical Answer:
Temporal supports workflow versioning through the @WorkerVersioning decorator or manual version checks. You can run multiple versions of a workflow simultaneously, allowing safe rollouts and rollbacks without data loss.
Code Example:
@workflow.defn
class PaymentWorkflow:
@workflow.run
async def run(self, payment_data: dict):
# Check current workflow version
if workflow.info().version == 1:
result = await workflow.execute_activity(
old_payment_processor, payment_data
)
else:
result = await workflow.execute_activity(
new_payment_processor, payment_data
)
return result
# Worker configuration for versioning
worker = Worker(
client,
task_queue="payment-tasks",
workflows=[PaymentWorkflow],
activities=[old_payment_processor, new_payment_processor],
build_id="v2.0.0"
)
🔍 Interviewer Notes: Testing understanding of production deployment strategies.
❌ Common Mistakes:
- Not planning for version compatibility
- Breaking changes without migration paths
- Ignoring in-flight workflow handling
Q6: What are Signals in Temporal?
Technical Answer: Signals are asynchronous messages that can update workflow state without restarting the workflow. They allow external systems to communicate with running workflows, enabling real-time updates and cancellations.
Code Example:
@workflow.defn
class OrderWorkflow:
def __init__(self):
self.status = "pending"
@workflow.run
async def run(self, order_id: str):
# Main workflow logic
result = await workflow.execute_activity(process_order, order_id)
return result
@workflow.signal
async def cancel_order(self):
self.status = "cancelled"
@workflow.signal
async def update_priority(self, priority: int):
self.priority = priority
// Sending a signal from client
signal := client.SignalWorkflow(cmd, workflowID, "", "cancel_order")
🔍 Interviewer Notes: Testing knowledge of external workflow interaction patterns.
❌ Common Mistakes:
- Treating signals as activities
- Not handling race conditions
- Forgetting signal idempotency
Q7: What is the Temporal Query API used for?
Technical Answer: The Query API allows you to inspect workflow state without modifying it. This is useful for monitoring, debugging, and building administrative interfaces that need to check workflow progress.
Code Example:
@workflow.defn
class OrderWorkflow:
def __init__(self):
self._progress = 0
@workflow.query
def get_progress(self) -> int:
return self._progress
@workflow.run
async def run(self, order_id: str):
# Update progress during execution
self._progress = 50
await workflow.execute_activity(step_one, order_id)
self._progress = 100
return "complete"
# CLI query
temporal workflow query --workflow-id my-workflow --query-type get_progress
🔍 Interviewer Notes: Testing understanding of monitoring and observability patterns.
❌ Common Mistakes:
- Using queries for control flow
- Not making queries deterministic
- Blocking queries for long periods
Q8: How does Temporal handle long-running workflows?
Technical Answer: Temporal uses event sourcing and checkpointing. A workflow can run for months or years, with each step recorded as an event. The state is reconstructed by replaying events, ensuring reliability regardless of duration.
Code Example:
@workflow.defn
class AnnualSubscriptionWorkflow:
@workflow.run
async def run(self, subscription_id: str):
while True:
# Wait for next billing cycle (could be days/weeks)
await workflow.sleep(timedelta(days=30))
await workflow.execute_activity(
bill_customer,
subscription_id,
start_to_close_timeout=timedelta(hours=1)
)
await workflow.execute_activity(
send_receipt,
subscription_id
)
🔍 Interviewer Notes: Testing knowledge of temporal patterns and resource management.
❌ Common Mistakes:
- Not accounting for versioned code changes during long runs
- Missing heartbeats for long activities
- Inefficient state storage
Q9: What are the different task queue types?
Technical Answer: Task queues in Temporal route work to appropriate workers. Each queue can have multiple worker types. Build IDs (introduced in Temporal 1.21+) allow version-aware task routing, ensuring workflows run on compatible code versions.
Configuration Example:
# temporal.yaml worker config
worker:
task_queues:
- name: "critical-payments"
build_id: "v2.1.0"
max_concurrent_workflow_tasks: 100
- name: "background-jobs"
build_id: "v2.1.0"
max_concurrent_activity_tasks: 50
🔍 Interviewer Notes: Testing operational knowledge and version management.
❌ Common Mistakes:
- Overloaded task queues
- Not using build IDs for version routing
- Shared queues across unrelated workflows
Q10: What is a Continue-As-New pattern?
Technical Answer: Continue-As-New prevents workflow histories from growing indefinitely by completing the current workflow and starting a new one with the same workflow ID. This is essential for workflows expected to run for very long periods.
Code Example:
@workflow.defn
class LongRunningWorkflow:
@workflow.run
async def run(self, batch_number: int):
# Process current batch
await process_batch(batch_number)
# After 10,000 events, continue as new
if workflow.get_version("continue-as-new", 0, 10000) > 0:
return await workflow.continue_as_new(batch_number + 1)
return "complete"
🔍 Interviewer Notes: Testing understanding of workflow lifecycle management.
❌ Common Mistakes:
- Not checking history size
- Breaking state continuity
- Not handling in-flight activities
Intermediate Topics
Q11: How do you handle timeouts in Temporal activities?
Technical Answer: Temporal provides start-to-close timeout (execution time), schedule-to-start timeout (queue wait time), and schedule-to-close timeout (total time). These prevent activities from running indefinitely and ensure timely workflow progress.
Code Example:
@activity.defn
async def data_processing(data: dict) -> dict:
# Activity logic
return result
@workflow.defn
class ProcessingWorkflow:
@workflow.run
async def run(self, data: dict):
result = await workflow.execute_activity(
data_processing,
data,
start_to_close_timeout=timedelta(minutes=5),
schedule_to_start_timeout=timedelta(minutes=1),
schedule_to_close_timeout=timedelta(minutes=10) # Total cap
)
return result
🔍 Interviewer Notes: Testing timeout configuration understanding.
❌ Common Mistakes:
- Not setting reasonable timeouts
- Confusing timeout types
- Too aggressive timeout values
Q12: What are Temporal timers and child workflows?
Technical Answer: Timers delay workflow execution without blocking workers. Child workflows are workflows started from within another workflow, enabling composition and modularity. Both support durable execution.
Code Example:
@workflow.defn
class ReminderWorkflow:
@workflow.run
async def run(self, user_id: str):
# Wait 30 days before sending reminder
await workflow.sleep(timedelta(days=30))
await workflow.execute_activity(send_reminder, user_id)
@workflow.defn
class ParentWorkflow:
@workflow.run
async def run(self, request: dict):
# Start child workflow
child_handle = await workflow.execute_child_workflow(
ReminderWorkflow,
request["user_id"]
)
# Do other work in parallel
result = await workflow.execute_activity(process_data, request)
# Wait for child (optional)
await child_handle
🔍 Interviewer Notes: Testing workflow composition patterns.
❌ Common Mistakes:
- Not using timers efficiently
- Child workflow versioning issues
- Missing cancellation propagation
Q13: How do you implement cancellation handling in Temporal?
Technical Answer: Temporal propagates cancellation from parent to child workflows and activities. Activities can check for cancellation via context, allowing graceful shutdown. Workflows should handle cancellation signals.
Code Example:
@activity.defn
async def long_running_task(ctx: activity.ActivityContext, data: dict) -> dict:
for item in data["items"]:
# Check for cancellation
if ctx.is_cancelled():
# Cleanup and exit gracefully
await cleanup()
raise asyncio.CancelledError()
await process_item(item)
return {"status": "complete"}
@workflow.defn
class CancellableWorkflow:
@workflow.run
async def run(self, data: dict):
activity_handle = workflow.execute_activity(
long_running_task,
data
)
# External signal can trigger cancellation
try:
await activity_handle
except asyncio.CancelledError:
# Handle cancellation
pass
🔍 Interviewer Notes: Testing cancellation and cleanup patterns.
❌ Common Mistakes:
- Not handling cancellation in long operations
- Forgetting to clean up resources
- Not propagating cancellation properly
Q14: What is Temporal cron scheduling?
Technical Answer:
Temporal supports cron-style scheduling for workflows using cron_schedule. This enables periodic execution like scheduled jobs. The schedule is stored durably and survives worker restarts.
Code Example:
@workflow.defn
class DailyReportWorkflow:
@workflow.run
async def run(self):
# Generate daily report
await workflow.execute_activity(generate_report)
# Start with cron schedule
handle = client.start_workflow(
DailyReportWorkflow.run,
id="daily-report",
task_queue="reports",
cron_schedule="0 9 * * *" # Every day at 9 AM
)
🔍 Interviewer Notes: Testing job scheduling patterns.
❌ Common Mistakes:
- Not understanding cron overlap behavior
- Missing timezone considerations
- Not handling backfill scenarios
Q15: How do you debug Temporal workflows?
Technical Answer: Temporal provides CLI tools, web UI, and tctl for debugging. You can query workflow state, replay history, and inspect task queues. History events show exactly what happened.
Debugging Commands:
# Describe workflow
tctl wf describe --wfid my-workflow
# Query workflow state
tctl wf query --wfid my-workflow --qt get_progress
# List task queues
tctl task-queue list
# Count workflow executions
tctl wf count --tq my-queue
🔍 Interviewer Notes: Testing debugging and operational skills.
❌ Common Mistakes:
- Not using history for root cause analysis
- Missing structured logging
- Not understanding task queue backlogs
Advanced Topics
Q16: How do you implement Temporal interceptors and middleware?
Technical Answer: Temporal interceptors intercept workflow and activity calls for logging, metrics, authentication, and other cross-cutting concerns. They provide hooks before and after execution.
Code Example:
class LoggingInterceptor:
async def execute_activity(self, input, invocation):
logger.info(f"Starting activity: {input.activity_type}")
start = time.time()
try:
result = await invocation()
logger.info(f"Completed in {time.time() - start}s")
return result
except Exception as e:
logger.error(f"Activity failed: {e}")
raise
# Register interceptor
worker = Worker(
client,
task_queue="my-queue",
interceptors=[LoggingInterceptor()]
)
🔍 Interviewer Notes: Testing middleware patterns.
❌ Common Mistakes:
- Not handling exceptions in interceptors
- Performance overhead from logging
Q17: What is the Temporal visibility API?
Technical Answer: The visibility API allows listing, filtering, and searching workflows. It's optimized for operational queries, not programmatic workflow management.
Usage:
# List running workflows
handles = client.list_workflows(
query="WorkflowType='OrderWorkflow' AND ExecutionStatus='Running'"
)
for handle in handles:
print(handle.workflow_id)
🔍 Interviewer Notes: Testing operational tooling knowledge.
❌ Common Mistakes:
- Not indexing searchable fields
- Expensive queries on large datasets
Q18: How do you handle Temporal schema evolution?
Technical Answer:
Use get_version() for conditional logic based on execution history. This enables backward-compatible workflow changes without breaking running executions.
Pattern:
@workflow.defn
class EvolvingWorkflow:
@workflow.run
async def run(self, data: dict):
if workflow.get_version("new_field", 0, 1) == 1:
result = await workflow.execute_activity(
new_activity,
data["new_field"]
)
else:
result = await workflow.execute_activity(
old_activity,
data.get("old_field")
)
🔍 Interviewer Notes: Testing versioning strategies.
❌ Common Mistakes:
- Breaking changes without version checks
- Not testing old workflow paths
Q19: What are Temporal side effects?
Technical Answer: Side effects execute non-deterministic code (like now(), random()) in workflows. They're cached and only execute once per workflow run, ensuring determinism.
Example:
@workflow.defn
class SideEffectWorkflow:
@workflow.run
async def run(self):
# Get current time once, cached for replay
timestamp = workflow.side_effect(time.time)
# Generate random ID once
order_id = workflow.side_effect(uuid.uuid4)
🔍 Interviewer Notes: Testing determinism understanding.
❌ Common Mistakes:
- Using side effects for core business logic
- Not caching expensive computations
Q20: How do you implement Temporal mutex for resource locking?
Technical Answer:
Use workflow.Lock to ensure exclusive access to critical sections within workflows, preventing concurrent execution of sensitive operations.
Code:
@workflow.defn
class LockingWorkflow:
_lock = workflow.Lock()
@workflow.run
async def run(self, resource_id: str):
async with self._lock:
# Critical section
await update_shared_resource(resource_id)
🔍 Interviewer Notes: Testing concurrency control.
❌ Common Mistakes:
- Not releasing locks properly
- Deadlock scenarios
Advanced Topics
Q21: How do you tune Temporal worker performance?
Technical Answer: Configure max concurrent activities, task queue prefetch, and worker shard settings. Monitor queue depths and adjust for optimal throughput.
Configuration:
worker:
max_concurrent_activities: 50
max_concurrent_workflow_tasks: 100
task_queue_prefetch: 10
🔍 Interviewer Notes: Testing performance tuning skills.
Q22: How do you perform Temporal capacity planning?
Technical Answer: Analyze workflow complexity, concurrent executions, and retention policies. Size databases (Cassandra/SQL) appropriately and plan for growth.
Metrics to track:
- Workflow starts per second
- History events per second
- Task queue depths
- Database write throughput
Q23: What are Temporal best practices for production?
Technical Answer: Use structured logging, implement circuit breakers, version workflows carefully, monitor task queues, and have runbooks for common issues.
Key practices:
- Enable metrics collection
- Set up alerting on queue backlogs
- Use appropriate retry policies
- Document workflow contracts
Q24: What is the Temporal sleep behavior and how does it differ from blocking a thread?
Technical Answer:
Temporal workflow.sleep() suspends the workflow — the worker frees the thread/goroutine, and the workflow resumes via event history when the timer fires. It is not a blocking sleep. This lets thousands of workflows run on few workers. Use workflow.Sleep in Python/Go or TimerFired for cancellation-sensitive waits.
Code/Syntax Justification:
from temporalio import workflow
@workflow.defn
class ReminderWorkflow:
@workflow.run
async def run(self, minutes: int) -> str:
await workflow.sleep(timedelta(minutes=minutes)) # non-blocking
return "Reminder fired"
🔍 Behind the Question: Event-sourced execution model.
❌ The Red Flags: Using time.sleep() inside workflows (blocks the worker).
Q25: How do you send and receive Signals in Python Temporal?
Technical Answer:
Signals are asynchronous messages into a running workflow. Define signal handlers with @workflow.signal; workflows call workflow.wait_condition or workflow.any on signal-set flags. To send, use client.get_workflow_handle(wf_id).signal(name, payload) from outside.
Code/Syntax Justification:
@workflow.defn
class OrderWorkflow:
def __init__(self):
self.cancelled = False
@workflow.signal
async def cancel_signal(self) -> None:
self.cancelled = True
@workflow.run
async def run(self) -> None:
await workflow.wait_condition(lambda: self.cancelled)
await workflow.execute_activity(cancel_order)
🔍 Behind the Question: Dynamic workflow interaction. ❌ The Red Flags: Polling DB inside workflow instead of signals.
Q26: What is a Workflow ID reuse policy and why does it matter?
Technical Answer:
Workflow IDs must be unique among open workflows; reuse policy controls what happens when starting with an existing ID: AllowDuplicate (starts new, may break previous), AllowDuplicateFailedOnly (reuse if previous failed), RejectDuplicate (raise on any open workflow with same ID). Default AllowDuplicateFailedOnly prevents accidental duplicate business processes (e.g., double-charging a customer).
Code/Syntax Justification:
await client.start_workflow(
OrderWorkflow.run,
args=[order_id],
id=f"order-{order_id}",
id_reuse_policy=IDReusePolicy.REJECT_DUPLICATE,
)
🔍 Behind the Question: Idempotency and business-process protection. ❌ The Red Flags: Allowing duplicates for payment/order workflows.
Q27: How do you handle side effects (non-deterministic operations) in workflows?
Technical Answer:
Workflows must be deterministic — the same history must always replay identically. Wrap non-deterministic code (random, UUID, current time, API calls) in workflow.side_effect (records the value in history once) or move it into Activities. workflow.now() gives deterministic time; workflow.random() is seeded deterministically.
Code/Syntax Justification:
request_id = await workflow.side_effect(lambda: str(uuid.uuid4()))
# or, better, put nondeterminism in an Activity
result = await workflow.execute_activity(generate_request_id)
🔍 Behind the Question: Determinism enforcement.
❌ The Red Flags: Calling random()/datetime.now() directly in workflow code.
Q28: How do you unit test Temporal workflows?
Technical Answer:
Use WorkflowEnvironment.start_time_skipping() (Python) — an in-memory env that auto-advances time so sleep-based workflows run fast. Assert on final result, signals sent, and activities called (via workflow.state or a fake). Test determinism by running the workflow twice and comparing history.
Code/Syntax Justification:
async def test_reminder():
async with await WorkflowEnvironment.start_time_skipping() as env:
async with env.start_client() as client:
handle = await client.start_workflow(ReminderWorkflow.run, id="w1")
result = await handle.result() # time-skipped, fast
assert result == "Reminder fired"
🔍 Behind the Question: Workflow correctness without real sleeps.
❌ The Red Flags: Testing with real time.sleep (minutes-long tests).
Q29: What are Temporal Search Attributes and when do you use them?
Technical Answer:
Search Attributes are indexed custom fields on workflows enabling rich queries: by user, order, status, custom business keys. Useful for building dashboards ("find all workflows for user X with status=blocked"), visibility, and operator UIs. They must be registered in the cluster; set with workflow.upsert_search_attributes.
Code/Syntax Justification:
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, user_id: str):
workflow.upsert_search_attributes(
{"UserID": user_id, "Status": "processing"}
)
# Query by search attribute
workflows = await client.list_workflows(
query='UserID = "u-123" AND Status = "processing"'
)
🔍 Behind the Question: Visibility and operational queries. ❌ The Red Flags: Using workflow IDs to encode business queries instead of search attributes.
Q30: What is Temporal's "Continue-As-New" and when is it triggered?
Technical Answer:
Long workflows accumulate history events; after limits (default ~50k events) execution continues as a new workflow with the same ID, resetting history while carrying over state/args. Trigger explicitly via workflow.continue_as_new or automatically on history size. Patterns: infinite loops, per-day aggregation jobs, recurring cleanup.
Code/Syntax Justification:
@workflow.defn
class DailyAggregator:
@workflow.run
async def run(self, day: int = 1):
await workflow.execute_activity(aggregate_day, args=[day])
if day < 365:
await workflow.continue_as_new(args=[day + 1]) # new history
🔍 Behind the Question: Managing unbounded histories. ❌ The Red Flags: Infinite workflows without continue-as-new (history bloat).
Q31: How do you run Temporal with a message queue pattern (SQS/Kafka)?
Technical Answer:
Temporal IS the durable queue for workflows — but for fan-in/out and external event ingestion, pair it with SQS/Kafka: consumers enqueue start_workflow/signal calls; workflows produce side-effect outputs via Activities that publish to topics. Use workflow.execute_activity for publish (deterministic), never publish from workflow code directly.
Code/Syntax Justification:
# Worker consumes SQS -> starts workflow
async def sqs_handler(msg):
order = json.loads(msg.body)
await client.start_workflow(OrderWorkflow.run, args=[order],
id=f"order-{order['id']}")
# Workflow publishes via Activity
@activity.defn
async def publish_event(event: dict):
await kafka_producer.send("order.events", event)
🔍 Behind the Question: Integration patterns between Temporal and event buses. ❌ The Red Flags: Making HTTP/Kafka calls directly inside workflow code.
Q32: What is the difference between Temporal Activities and Workflows in error handling?
Technical Answer:
Activities are where I/O happens (they can fail, retry, have timeouts); Workflows orchestrate and must not fail nondeterministically. Activity failures surface as ActivityError in workflow code — you catch, compensate, or let Temporal retry per retry policy. Workflow failures with pending state can be retried (by policy) but cause state to re-run from start if not idempotent.
Code/Syntax Justification:
try:
await workflow.execute_activity(charge_customer, retry_policy=RetryPolicy(maximum_attempts=3))
except ActivityError:
await workflow.execute_activity(refund_customer) # compensate
🔍 Behind the Question: Failure boundaries and retry design. ❌ The Red Flags: Catching all errors and swallowing retries; no compensation.
Q33: How do you version workflows (compatibility mode)?
Technical Answer:
Use workflow.patched / version markers: mark a point in history with a version ID; when replaying old histories, old code takes the old branch, new code takes the new branch for new executions. Never change a workflow's logic without versioning — it breaks determinism on replay.
Code/Syntax Justification:
if workflow.patched("v2-charge-flow"):
await workflow.execute_activity(charge_v2)
else:
await workflow.execute_activity(charge_v1)
🔍 Behind the Question: Safe workflow evolution. ❌ The Red Flags: Editing workflow logic without version markers.
Q34: How do you implement idempotent Activities?
Technical Answer: Activities can be retried and re-executed — make them idempotent by keying side effects: use a deterministic business key (payment_intent_id, order_id) and store results in the DB with unique constraints so a duplicate execution no-ops or returns the stored result.
Code/Syntax Justification:
@activity.defn
async def charge_customer(card: dict, amount: int) -> str:
key = f"charge:{card['id']}:{amount}"
existing = await redis.get(key)
if existing:
return existing # idempotent replay
result = await payment_provider.charge(card, amount)
await redis.setex(key, 86400, result)
return result
🔍 Behind the Question: At-least-once semantics. ❌ The Red Flags: Double-charging customers on retries.
Q35: What is a Temporal Worker and how do you deploy it at scale?
Technical Answer:
Workers poll task queues and execute workflows/activities. Deploy them as services (ECS/Fargate, EKS, Lambda for activities) with the SDK; scale workers independently per queue. Use max_concurrent_workflow_tasks/max_concurrent_activity_tasks tuning, and worker sticky execution. Monitor task queue backlog and worker health. Separate workflow-heavy vs activity-heavy worker pools for throughput isolation.
Code/Syntax Justification:
worker = Worker(
client,
task_queue="orders",
workflows=[OrderWorkflow],
activities=[charge_customer, refund_customer],
max_concurrent_workflow_tasks=50,
max_concurrent_activity_tasks=100,
)
await worker.run()
🔍 Behind the Question: Deployment topology and throughput tuning. ❌ The Red Flags: One worker handling everything with no backlog monitoring.
Senior / Principal — Real World Scenarios
Q36: Long-running workflow blocks a worker slot — workflow starvation
Level: Senior
Scenario: A Temporal workflow waits on an external approval (e.g., await asyncio.sleep(86400 * 7) or a signal). The worker slot stays occupied for days. Many such workflows exhaust worker concurrency, blocking new workflows.
Impact: New workflows cannot start. Workers saturated with idle waiting workflows.
Root Cause: Temporal workflow code runs on the worker. Blocking/sleeping in workflow code consumes a workflow task slot for the entire wait duration.
Simple explanation: A cashier stands at the register waiting for a customer to decide, blocking the line for hours while other customers wait.
Technical Answer:
- Never use
time.sleepor longawaitin workflow code. Use Temporal timers (workflow.wait_condition,workflow.sleepviatimer) which park the workflow state without blocking a worker slot. - Use signals for external events (approval) — the workflow parks and resumes when the signal arrives.
- Size
max_concurrent_workflow_tasksbased on active (not waiting) workflows. - Separate long-running workflows to a dedicated task queue with its own worker pool.
Code/Syntax Justification:
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id):
# Park workflow without blocking worker slot; resumes on signal or timer
await workflow.wait_condition(lambda: self.approved, timeout=timedelta(days=7))
if not self.approved:
raise TimeoutError("Approval expired")
await workflow.execute_activity(charge_customer, order_id)
@workflow.signal
def approve(self):
self.approved = True
Workflow Diagram:
Workflow waiting for approval
|
v
Wrong: time.sleep(7 days) Right: workflow.wait_condition (timer)
| |
v v
Worker slot blocked for 7 days Worker slot freed immediately
(workflow state persisted)
|
v
Resumes when signal/timer fires
🔍 Behind the Question: Workflow state parking, timers vs sleep, worker slot management.
❌ The Red Flags: time.sleep in workflow; long-running loops blocking workers; no signal/timer pattern.
Q37: Activity retry storm — downstream service overwhelmed
Level: Principal
Scenario: A payment activity calls a downstream payment gateway. The gateway is degraded (slow/errors). Temporal retries the activity with default backoff. Hundreds of workflows retry simultaneously, hammering the already-degraded gateway and making it worse.
Impact: Retry storm. Complete outage of the payment gateway. Cascading failure.
Root Cause: Uncontrolled retries without exponential backoff, jitter, or rate limiting. All workflows retry at the same interval.
Simple explanation: A store's phone line is busy. 100 people all redial every 5 seconds. The line never frees up.
Technical Answer:
- Use exponential backoff with jitter (
retry_policywithbackoff_coefficient,maximum_interval). - Set
maximum_attemptsso retries stop after N tries. - Use
retry_policyper activity; distinguish retryable vs non-retryable errors. - Add circuit breaker in the activity itself (stop calling if gateway is down).
- Monitor activity failure rate and task queue backlog.
Code/Syntax Justification:
@activity.defn
async def charge_customer(order_id, amount):
return await payment_gateway.charge(order_id, amount)
# In workflow
await workflow.execute_activity(
charge_customer,
args=[order_id, amount],
retry_policy=RetryPolicy(
initial_interval=timedelta(seconds=1),
backoff_coefficient=2.0,
maximum_interval=timedelta(minutes=5),
maximum_attempts=5,
),
)
Workflow Diagram:
Gateway degraded
|
v
Without backoff: With exponential backoff + jitter:
retry every 1s retry 1s, 2s, 4s, 8s... + random jitter
| |
v v
100 workflows retry together workflows spread out over time
gateway overwhelmed gateway recovers
🔍 Behind the Question: Retry policy, exponential backoff, jitter, circuit breaker, retry storms.
❌ The Red Flags: No retry limits; fixed short interval retries; retrying non-retryable errors (auth failures).
Q38: Non-deterministic workflow code — replay failure
Level: Principal
Scenario: A Temporal workflow uses datetime.now(), random.uuid4(), or an external API call directly in workflow code. On replay (after a worker crash or new code version), Temporal re-executes the workflow from history and gets a different result. The workflow fails with "non-deterministic error."
Impact: Workflows stuck, cannot replay. Deployments break existing in-flight workflows.
Root Cause: Temporal replays workflow code deterministically from event history. Any non-deterministic call (time, random, IO, current date) produces different results on replay, violating the replay contract.
Simple explanation: You tell a story, then retell it. But the second time, the details changed because you looked at a clock. The audience is confused.
Technical Answer:
- All non-deterministic work must be in activities (which are not replayed — executed once).
- Workflow code must be pure/deterministic: use
workflow.now()instead ofdatetime.now(),workflow.uuid4()instead ofrandom.uuid4(). - Side effects (DB writes, API calls) only in activities.
- Version workflows using
workflow.versionwhen changing logic, so old history replays with old code.
Code/Syntax Justification:
# Wrong (non-deterministic, replay fails)
import datetime
start = datetime.datetime.now() # different on replay!
# Right (deterministic, Temporal-managed)
from temporalio import workflow
start = workflow.now() # replay-safe
uid = workflow.uuid4() # replay-safe
# Side effect in activity (not replayed)
await workflow.execute_activity(charge_customer, order_id)
Workflow Diagram:
Non-deterministic in workflow: Deterministic + activity:
workflow.now() differs on replay workflow.now() replay-safe
random() differs on replay activity runs ONCE (not replayed)
| |
v v
"Non-deterministic workflow error" Successful replay & recovery
🔍 Behind the Question: Workflow determinism, replay contract, activities for side effects, versioning.
❌ The Red Flags: datetime.now()/random()/time.time() in workflow; DB calls in workflow; no versioning on code changes.
Q39: Workflow versioning — deploying new code without breaking in-flight workflows
Level: Principal
Scenario: You deploy a new version of a workflow that adds a step. In-flight workflows (started with old code) replay with new code and fail because the new step expects data that doesn't exist in their history.
Impact: In-flight workflows fail after deploy. Cannot roll forward safely.
Root Cause: Temporal replays workflow code from history. Changing workflow logic changes what events are expected. Old history doesn't match new code paths.
Simple explanation: You change the rules of a game mid-play. Players who started under old rules can't follow new rules.
Technical Answer:
- Use
workflow.version()to mark code changes; old history replays with old code path, new workflows use new path. - Prefer additive changes only (new signals, new activities) that don't break old history.
- Avoid changing the order or existence of workflow-deterministic calls.
- Test replay explicitly: start a workflow with old code, upgrade, and verify it still completes.
Code/Syntax Justification:
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id):
if workflow.version("add-refund-step", "1.0", "2.0"):
await workflow.execute_activity(refund_customer, order_id)
await workflow.execute_activity(charge_customer, order_id)
Workflow Diagram:
Old code deployed --> workflows A, B in-flight (history recorded)
|
v
New code deployed (changed step order)
|
v
Without version(): With workflow.version():
A replays with new A replays with old code path (v1.0)
code -> fails B (new) uses new code path (v2.0)
🔍 Behind the Question: Workflow versioning, replay compatibility, safe deployments.
❌ The Red Flags: Changing workflow step order without versioning; no replay testing; breaking in-flight workflows on deploy.
Senior / Principal — Temporal Workflow Real World Scenarios (Continued)
Q40: Workflow task queue saturation — workers can't keep up
Level: Senior
Scenario: A Temporal workflow processes customer onboarding. Task queue backlog grows to 50,000 pending workflow tasks. Workers are at 100% CPU but can't drain the queue. New workflows wait hours before starting.
Impact: Workflow latency. Customer onboarding delayed. SLA breached.
Root Cause: Worker concurrency too low for the workflow start rate. Each worker slot handles one workflow task at a time. More workflows arrive than workers can process.
Simple explanation: A factory conveyor brings 1000 boxes/hour. 10 workers can pack 100/hour. Boxes pile up. Need more workers or faster packing.
Technical Answer:
- Increase
max_concurrent_workflow_tasksper worker (up to CPU/memory limits). - Scale workers horizontally (more worker instances/pods).
- Use separate task queues for different workflow types to isolate backlog.
- Rate-limit workflow starts if downstream can't keep up.
- Monitor
temporal_workflow_tasks_backlogandtemporal_worker_task_slot_usage.
Code/Syntax Justification:
worker = Worker(
client,
task_queue="onboarding",
workflows=[OnboardingWorkflow],
max_concurrent_workflow_tasks=100, # tune based on CPU/memory
max_concurrent_activity_tasks=200,
)
Workflow Diagram:
Workflow start rate > Worker capacity
|
v
Task queue backlog grows
|
v
Options:
+-- Increase max_concurrent_workflow_tasks
+-- Add worker instances (horizontal scale)
+-- Separate task queues per workflow type
+-- Rate-limit workflow starts
|
v
Backlog drains
🔍 Behind the Question: Worker concurrency, task queue backlog, horizontal scaling, rate limiting.
❌ The Red Flags: One worker for all workflows; no backlog monitoring; not rate-limiting starts.
Q41: Activity heartbeats — long-running activity appears stuck
Level: Senior
Scenario: An activity processes a large file (10 minutes). Temporal marks it as timed out after schedule_to_close_timeout (default 5 minutes) because no heartbeat. The workflow retries the activity from the beginning. Work never completes.
Impact: Long activities never complete. Infinite retry loop. Wasted compute.
Root Cause: Activity doesn't send heartbeats. Temporal assumes it's dead after heartbeat_timeout. Workflow retries from scratch.
Simple explanation: You're cooking a 10-minute dish. The timer beeps at 5 minutes. The chef assumes you died and starts over. You never finish cooking.
Technical Answer:
- Set
heartbeat_timeoutshorter than activity duration (e.g., 60 seconds). - Call
activity.heartbeat()periodically in the activity code. - Set
schedule_to_close_timeoutto total expected duration + retries. - On retry, use
last_heartbeatdetails to resume from checkpoint (activity is idempotent). - Record progress details in heartbeat so retries can resume.
Code/Syntax Justification:
from temporalio import activity
@activity.defn
async def process_large_file(file_path: str):
total = get_file_size(file_path)
processed = 0
with open(file_path, "rb") as f:
while chunk := f.read(8192):
process_chunk(chunk)
processed += len(chunk)
# Heartbeat with progress
activity.heartbeat(
details={"bytes_processed": processed, "total": total}
)
Workflow Diagram:
Long activity (10 min): With heartbeats:
Start Start
| |
v v
5 min timeout (no heartbeat) 1 min heartbeat (progress: 20%)
| |
v v
Workflow retries from start Worker continues, heartbeats every 1 min
| |
v v
Never completes Completes at 10 min
🔍 Behind the Question: Activity heartbeats, timeout configuration, retry from checkpoint, activity details.
❌ The Red Flags: No heartbeat in long activity; heartbeat_timeout too short; not recording progress in heartbeat details.
Q42: Signal handling — workflow misses signal sent before wait
Level: Principal
Scenario: A workflow waits for an approval signal with workflow.wait_condition. The signal is sent before the workflow reaches the wait. The workflow misses it and waits forever (or until timeout).
Impact: Workflow stuck. Approval never processed. Manual intervention needed.
Root Cause: Signal sent before workflow calls wait_condition. Temporal buffers signals but the workflow must be in a waiting state to consume them, or use a signal handler that sets a flag checked by the wait condition.
Simple explanation: You knock on the door before the person sits down to wait for the knock. They sit down and wait for a knock that already happened.
Technical Answer:
- Use a signal handler that sets a boolean flag (e.g.,
self.approved = True). wait_conditionchecks the flag:await workflow.wait_condition(lambda: self.approved).- If signal already arrived, flag is True, wait returns immediately.
- Combine with
workflow.wait_conditiontimeout for safety. - Signals are buffered by Temporal even if workflow isn't waiting.
Code/Syntax Justification:
@workflow.defn
class ApprovalWorkflow:
def __init__(self):
self.approved = False
@workflow.signal
def approve(self):
self.approved = True
@workflow.run
async def run(self):
# If signal already sent, this returns immediately
await workflow.wait_condition(lambda: self.approved, timeout=timedelta(days=7))
if not self.approved:
raise TimeoutError("Approval expired")
await workflow.execute_activity(process_approval)
Workflow Diagram:
Signal before wait: With flag + wait_condition:
Signal sent Signal sets self.approved = True
| |
v v
Workflow reaches wait Workflow reaches wait_condition
| |
v v
Waits forever (missed signal) wait_condition sees flag=True, returns immediately
🔍 Behind the Question: Signal handling, wait_condition, signal buffering, race condition.
❌ The Red Flags: Waiting for signal without flag; signal sent before workflow starts waiting; no timeout on wait_condition.
Q43: Child workflow failures — parent doesn't handle child failure
Level: Senior
Scenario: A parent workflow starts 5 child workflows (one per region). One child fails (e.g., region-specific payment gateway down). Parent doesn't handle the failure. Parent completes as if all succeeded. Data inconsistency.
Impact: Silent data inconsistency. Missing region processing. No alerting.
Root Cause: Parent workflow doesn't catch child workflow exceptions. execute_child_workflow raises on failure; if not caught, parent fails. But if parent catches and continues, it may not reflect partial failure.
Simple explanation: You send 5 assistants on errands. One fails. You don't check. You report "all done" when one task is undone.
Technical Answer:
- Wrap
execute_child_workflowin try/except per child. - Use
workflow.execute_child_workflowwithretry_policyfor transient failures. - Collect results/failures; decide whether to continue or fail parent based on business logic.
- Use
workflow.get_external_workflow_handlefor long-running external workflows. - Consider
workflow.wait_for_child_workflow_closefor explicit failure handling.
Code/Syntax Justification:
@workflow.defn
class ParentWorkflow:
@workflow.run
async def run(self, regions: list[str]):
results = []
for region in regions:
try:
result = await workflow.execute_child_workflow(
RegionWorkflow.run,
region,
id=f"region-{region}",
retry_policy=RetryPolicy(maximum_attempts=3),
)
results.append(result)
except Exception as exc:
results.append({"region": region, "error": str(exc)})
# Decide: fail if any region failed, or continue with partial success
if any("error" in r for r in results):
raise ApplicationError("Some regions failed", results)
return results
Workflow Diagram:
Parent starts 5 child workflows
|
v
Child 3 fails (payment gateway down)
|
v
Without handling: With try/except per child:
Parent reports success Catch exception, record failure
| |
v v
Data inconsistency Decide: fail parent or continue partial
🔍 Behind the Question: Child workflow failure handling, retry policy, partial failure, error aggregation.
❌ The Red Flags: Not catching child workflow exceptions; assuming all children succeed; no partial failure strategy.
Q44: Workflow timeout — workflow_execution_timeout too short
Level: Senior
Scenario: A workflow processes a monthly billing cycle. It takes 3 days (waiting for payment confirmations). workflow_execution_timeout is 24 hours (default). Workflow times out mid-execution. History is lost. Billing incomplete.
Impact: Workflow terminated. Business process incomplete. Data loss.
Root Cause: workflow_execution_timeout (max time from start to close) is shorter than actual business process duration.
Simple explanation: You give a chef 1 hour to cook a 3-day slow-roast. The chef is stopped at 1 hour. Meal incomplete.
Technical Answer:
- Set
workflow_execution_timeoutto expected max duration + buffer (e.g., 7 days for monthly billing). - Use
workflow_run_timeoutfor actual execution time (excluding wait time). - Use
workflow_task_timeoutfor individual task completion (default 10 seconds). - For very long processes, use
continue_as_newto reset execution time while preserving state. - Monitor
temporal_workflow_execution_durationto set appropriate timeouts.
Code/Syntax Justification:
# Start workflow with appropriate timeouts
handle = await client.start_workflow(
BillingWorkflow.run,
args=[month],
id=f"billing-{month}",
workflow_execution_timeout=timedelta(days=7),
workflow_run_timeout=timedelta(days=3),
workflow_task_timeout=timedelta(seconds=10),
)
Workflow Diagram:
Monthly billing (3 days): With proper timeouts:
Start workflow_execution_timeout = 7 days
| |
v v
24 hour timeout -> terminated Runs 3 days, completes successfully
| |
v v
Billing incomplete Workflow closed, history preserved
🔍 Behind the Question: Workflow timeouts, execution vs run vs task timeout, continue_as_new, timeout tuning.
❌ The Red Flags: Default 24-hour execution timeout for long processes; not distinguishing run vs execution timeout; no continue_as_new for very long workflows.
Q45: continue_as_new — resetting long-running workflow state
Level: Principal
Scenario: A workflow runs for months (e.g., subscription renewal cycle). Event history grows large (thousands of events). Replay becomes slow and expensive. Workflow history hits size limits.
Impact: Slow replay. High history storage cost. Workflow history limit reached.
Root Cause: Workflow never calls continue_as_new. History accumulates forever. Replaying full history on every task is inefficient.
Simple explanation: You keep adding pages to a single notebook. After 1000 pages, finding anything takes forever. You should start a new notebook and carry forward the summary.
Technical Answer:
- Use
workflow.continue_as_new()to start a new execution with updated input, carrying forward state. - Call
continue_as_newperiodically (e.g., every N events or every N days). - Pass relevant state (not full history) as new input arguments.
- Old execution closes; new execution starts fresh with compact history.
- External handles/signal names remain valid across
continue_as_new.
Code/Syntax Justification:
@workflow.defn
class SubscriptionWorkflow:
def __init__(self):
self.renewal_count = 0
@workflow.run
async def run(self, customer_id: str):
# ... do work ...
self.renewal_count += 1
# Continue as new every 12 renewals to keep history small
if self.renewal_count % 12 == 0:
await workflow.continue_as_new(customer_id)
Workflow Diagram:
Single workflow execution (months): With continue_as_new:
History grows: 1000s of events Execution 1: events 1-500
| |
v v
Slow replay, high cost continue_as_new -> Execution 2
High storage cost History: events 1-500 (carried as input)
Execution 2: events 501-1000
🔍 Behind the Question: continue_as_new, history size, replay performance, state preservation across executions.
❌ The Red Flags: Never calling continue_as_new for long-running workflows; history hitting size limits; slow replay.
Q46: Deterministic activity — non-deterministic code in activity
Level: Senior
Scenario: An activity uses datetime.now() to timestamp results. On retry, the activity re-executes and gets a different timestamp. Temporal doesn't detect this (activities aren't replayed like workflows), but downstream logic depending on the timestamp breaks.
Impact: Inconsistent timestamps. Retry produces different results. Hard to debug.
Root Cause: Activities are executed once (not replayed), but non-deterministic calls (time, random, UUID) create inconsistent results on retry.
Simple explanation: You write the time on a receipt. If you rewrite the receipt, the time changes. Different copies show different times.
Technical Answer:
- Activities can use
datetime.now()(they run once), but be aware retries re-execute fully. - Pass timestamps from workflow (deterministic via
workflow.now()) to activities if consistency needed. - Use
activity.heartbeatwith progress details for resumable activities. - Make activities idempotent so retries are safe.
- For truly deterministic needs, generate IDs/timestamps in workflow and pass to activity.
Code/Syntax Justification:
# Workflow passes deterministic timestamp to activity
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id: str):
order_time = workflow.now() # deterministic, replay-safe
result = await workflow.execute_activity(
process_order, order_id, order_time
)
return result
@activity.defn
async def process_order(order_id: str, order_time: datetime):
# Use order_time, not datetime.now()
...
Workflow Diagram:
Activity uses datetime.now(): Workflow passes workflow.now():
Run 1: timestamp = 10:00:00 Workflow: order_time = 10:00:00
Retry: timestamp = 10:05:00 Activity: uses order_time = 10:00:00
| |
v v
Inconsistent results Consistent across retries
🔍 Behind the Question: Activity determinism, retry behavior, workflow.now() vs datetime.now(), idempotent activities.
❌ The Red Flags: Relying on activity timestamps for consistency; non-idempotent activities; mixing workflow and activity time sources.
Q47: Search attributes — querying workflows by business fields
Level: Senior
Scenario: Operations team needs to find all workflows for customer_id = "C-123" and status = "pending". Temporal list API only filters by workflow ID, type, and time. Can't query by business data.
Impact: Can't find workflows for support/ops. Manual scanning. Slow incident response.
Root Cause: No search attributes set on workflow. Temporal can't index business data.
Simple explanation: You need to find all red folders. The filing cabinet only sorts by date. You must open every folder.
Technical Answer:
- Define search attributes (custom text, keyword, datetime, int, double, bool, keyword list).
- Set search attributes when starting workflow:
search_attributes={"CustomerId": "C-123", "Status": "pending"}. - Update search attributes during workflow with
workflow.upsert_search_attributes. - Query with Temporal CLI/SDK:
--query="CustomerId = 'C-123' AND Status = 'pending'". - Index commonly queried fields (customer ID, status, order ID).
Code/Syntax Justification:
# Start workflow with search attributes
from temporalio.common import SearchAttributeKey, SearchAttributeValue
handle = await client.start_workflow(
OrderWorkflow.run,
args=[order_id],
id=f"order-{order_id}",
search_attributes={
SearchAttributeKey.custom_text_fields["CustomerId"]: order_id,
SearchAttributeKey.custom_keyword_list_fields["Status"]: ["pending"],
},
)
# Update during workflow
workflow.upsert_search_attributes({
SearchAttributeKey.custom_keyword_list_fields["Status"]: ["completed"],
})
Workflow Diagram:
Without search attributes: With search attributes:
List all workflows, filter in code Query: CustomerId = 'C-123' AND Status = 'pending'
| |
v v
Slow, manual Instant, server-side filtering
🔍 Behind the Question: Search attributes, Temporal query language, operational visibility, indexing business data.
❌ The Red Flags: No search attributes; scanning all workflows in application code; not updating status during workflow.
Q48: Workflow cancellation — cleanup on cancel
Level: Senior
Scenario: A user cancels a long-running export workflow. Temporal cancels the workflow. Activities already started continue running. No cleanup. Partial files left in S3. Orphaned resources.
Impact: Resource leakage. Partial data. Storage cost. No cleanup.
Root Cause: Workflow doesn't handle cancellation. No try/finally or cancellation handler to clean up resources.
Simple explanation: You cancel a print job. The printer keeps printing the current page. Paper wasted. You must manually remove it.
Technical Answer:
- Use
workflow.cancel()to request cancellation. - Wrap critical sections in
try/finallyfor cleanup. - Check
workflow.cancelled()or catchCancelledError. - Use
workflow.execute_activitywith cancellation support. - Clean up external resources (S3 files, DB rows) in the
finallyblock.
Code/Syntax Justification:
@workflow.defn
class ExportWorkflow:
@workflow.run
async def run(self, export_id: str):
try:
result = await workflow.execute_activity(
start_export, export_id
)
await workflow.execute_activity(upload_result, result)
finally:
# Cleanup on cancel or completion
await workflow.execute_activity(cleanup_export, export_id)
Workflow Diagram:
Workflow cancelled: With finally cleanup:
Activities continue Cancel requested
| |
v v
Orphaned S3 files finally block: cleanup_export runs
| |
v v
Resource leak Resources cleaned
🔍 Behind the Question: Workflow cancellation, cleanup/finally, cancelled error handling, resource management.
❌ The Red Flags: No cancellation handling; orphaned external resources; not using try/finally.
Q49: Workflow replay — code changes break replay
Level: Principal
Scenario: A developer renames an activity function. In-flight workflows (started with old code) replay with new code. execute_activity("old_name") fails because function no longer exists. Workflow fails with "activity not registered".
Impact: In-flight workflows fail after deploy. Can't roll forward safely.
Root Cause: Workflow replays history deterministically. Changing activity names or workflow logic breaks replay compatibility with old history.
Simple explanation: You change the rules of a game mid-play. Players who started under old rules can't follow new rules. Game breaks.
Technical Answer:
- Never rename activities used by in-flight workflows. Keep old activity registrations.
- Use
workflow.version()for code changes that affect replay. - Make changes additive only (new activities, new signals) that don't break old history.
- Test replay: start workflow with old code, upgrade, verify completion.
- Use
workflow.memofor versioning metadata if needed.
Code/Syntax Justification:
# Old activity kept for replay compatibility
@activity.defn
async def send_email_v1(to: str, message: str): # keep name
await email_service.send(to, message)
# New activity with better signature
@activity.defn
async def send_email_v2(to: str, message: str, template: str):
await email_service.send_template(to, template)
# Workflow uses version to choose
@workflow.defn
class NotifyWorkflow:
@workflow.run
async def run(self, to: str, message: str):
if workflow.version("email-v2", "1.0", "2.0"):
await workflow.execute_activity(send_email_v2, to, message, "welcome")
else:
await workflow.execute_activity(send_email_v1, to, message)
Workflow Diagram:
Code change (rename activity): With version():
Old workflows replay with new code Old workflows use v1.0 path (old activity)
| |
v v
Activity not found -> workflow fails New workflows use v2.0 path
🔍 Behind the Question: Workflow replay, activity naming, version(), backward compatibility, safe deployments.
❌ The Red Flags: Renaming activities; changing activity signatures; not testing replay; breaking in-flight workflows.
Q50: Temporal cluster — namespace isolation and throttling
Level: Principal
Scenario: Multiple teams share one Temporal cluster. Team A's workflow flood (10,000 starts/min) throttles Team B's critical workflows. No isolation. One team's outage affects all.
Impact: Cross-team blast radius. Critical workflows delayed. No SLA isolation.
Root Cause: No namespace isolation or rate limiting. All teams share the same Temporal namespace and task queues.
Simple explanation: 10 teams share one kitchen. Team A orders 1000 meals. Team B's meal waits 2 hours.
Technical Answer:
- Use separate Temporal namespaces per team (isolated workflows, task queues, configs).
- Set namespace-level rate limits (workflow starts per second).
- Use separate task queues per team/workflow type.
- Monitor per-namespace metrics (
temporal_namespace_workflow_start_count). - Consider separate Temporal clusters for critical vs non-critical teams.
Code/Syntax Justification:
# Client per namespace
client_a = await Client.connect("temporal:7233", namespace="team-a")
client_b = await Client.connect("temporal:7233", namespace="team-b")
# Namespace rate limit (Temporal config)
# namespace_rate_limit:
# workflow_start_rate: 100 # per second
Workflow Diagram:
Shared namespace: Separate namespaces:
Team A: 10,000 starts/min Team A namespace: rate-limited to 100/s
Team B: 50 starts/min Team B namespace: unaffected
| |
v v
Team B throttled by Team A Team B SLAs met
Critical workflows delayed Isolation
🔍 Behind the Question: Namespace isolation, rate limiting, multi-tenancy, blast radius, SLA isolation.
❌ The Red Flags: One namespace for all teams; no rate limits; no task queue separation; no per-namespace monitoring.
Senior / Principal — Temporal Workflow Real World Scenarios (Continued)
Q51: Workflow replay — code changes break replay
Level: Principal
Scenario: A developer renames an activity function. In-flight workflows (started with old code) replay with new code. execute_activity("old_name") fails because function no longer exists. Workflow fails with "activity not registered".
Impact: In-flight workflows fail after deploy. Can't roll forward safely.
Root Cause: Temporal replays workflow code deterministically. Changing activity names or workflow logic breaks replay compatibility with old history.
Simple explanation: You change the rules of a game mid-play. Players who started under old rules can't follow new rules. Game breaks.
Technical Answer:
- Never rename activities used by in-flight workflows. Keep old activity registrations.
- Use
workflow.version()for code changes that affect replay. - Make changes additive only (new activities, new signals) that don't break old history.
- Test replay: start workflow with old code, upgrade, verify completion.
- Use
workflow.memofor versioning metadata if needed.
Code/Syntax Justification:
# Old activity kept for replay compatibility
@activity.defn
async def send_email_v1(to: str, message: str): # keep name
await email_service.send(to, message)
# New activity with better signature
@activity.defn
async def send_email_v2(to: str, message: str, template: str):
await email_service.send_template(to, template)
# Workflow uses version to choose
@workflow.defn
class NotifyWorkflow:
@workflow.run
async def run(self, to: str, message: str):
if workflow.version("email-v2", "1.0", "2.0"):
await workflow.execute_activity(send_email_v2, to, message, "welcome")
else:
await workflow.execute_activity(send_email_v1, to, message)
Workflow Diagram:
Code change (rename activity): With version():
Old workflows replay with new code Old workflows use v1.0 path (old activity)
| |
v v
Activity not found -> workflow fails New workflows use v2.0 path
🔍 Behind the Question: Workflow replay, activity naming, version(), backward compatibility, safe deployments.
❌ The Red Flags: Renaming activities; changing activity signatures; not testing replay; breaking in-flight workflows on deploy.
Q52: Workflow cancellation — cleanup on cancel
Level: Senior
Scenario: A user cancels a long-running export workflow. Temporal cancels the workflow. Activities already started continue running. No cleanup. Partial files left in S3. Orphaned resources.
Impact: Resource leakage. Partial data. Storage cost. No cleanup.
Root Cause: Workflow doesn't handle cancellation. No try/finally or cancellation handler to clean up resources.
Simple explanation: You cancel a print job. The printer keeps printing the current page. Paper wasted. You must manually remove it.
Technical Answer:
- Use
workflow.cancel()to request cancellation. - Wrap critical sections in
try/finallyfor cleanup. - Check
workflow.cancelled()or catchCancelledError. - Use
workflow.execute_activitywith cancellation support. - Clean up external resources (S3 files, DB rows) in the
finallyblock.
Code/Syntax Justification:
@workflow.defn
class ExportWorkflow:
@workflow.run
async def run(self, export_id: str):
try:
result = await workflow.execute_activity(
start_export, export_id
)
await workflow.execute_activity(upload_result, result)
finally:
# Cleanup on cancel or completion
await workflow.execute_activity(cleanup_export, export_id)
Workflow Diagram:
Workflow cancelled: With finally cleanup:
Activities continue Cancel requested
| |
v v
Orphaned S3 files finally block: cleanup_export runs
| |
v v
Resource leak Resources cleaned
🔍 Behind the Question: Workflow cancellation, cleanup/finally, cancelled error handling, resource management.
❌ The Red Flags: No cancellation handling; orphaned external resources; not using try/finally.
Q53: Non-deterministic workflow code — replay failure
Level: Principal
Scenario: A Temporal workflow uses datetime.now(), random.uuid4(), or an external API call directly in workflow code. On replay (after a worker crash or new code version), Temporal re-executes the workflow from history and gets a different result. The workflow fails with "non-deterministic error."
Impact: Workflows stuck, cannot replay. Deployments break existing in-flight workflows.
Root Cause: Temporal replays workflow code deterministically from event history. Any non-deterministic call (time, random, IO, current date) produces different results on replay, violating the replay contract.
Simple explanation: You tell a story, then retell it. But the second time, the details changed because you looked at a clock. The audience is confused.
Technical Answer:
- All non-deterministic work must be in activities (which are not replayed — executed once).
- Workflow code must be pure/deterministic: use
workflow.now()instead ofdatetime.now(),workflow.uuid4()instead ofrandom.uuid4(). - Side effects (DB writes, API calls) only in activities.
- Version workflows using
workflow.versionwhen changing logic, so old history replays with old code.
Code/Syntax Justification:
# Wrong (non-deterministic, replay fails)
import datetime
start = datetime.datetime.now() # different on replay!
# Right (deterministic, Temporal-managed)
from temporalio import workflow
start = workflow.now() # replay-safe
uid = workflow.uuid4() # replay-safe
# Side effect in activity (not replayed)
await workflow.execute_activity(charge_customer, order_id)
Workflow Diagram:
Non-deterministic in workflow: Deterministic + activity:
workflow.now() differs on replay workflow.now() replay-safe
random() differs on replay activity runs ONCE (not replayed)
| |
v v
"Non-deterministic workflow error" Successful replay & recovery
🔍 Behind the Question: Workflow determinism, replay contract, activities for side effects, versioning.
❌ The Red Flags: datetime.now()/random()/time.time() in workflow; DB calls in workflow; no versioning on code changes.
Q54: Workflow versioning — deploying new code without breaking in-flight workflows
Level: Principal
Scenario: You deploy a new version of a workflow that adds a step. In-flight workflows (started with old code) replay with new code and fail because the new step expects data that doesn't exist in their history.
Impact: In-flight workflows fail after deploy. Cannot roll forward safely.
Root Cause: Temporal replays workflow code from history. Changing workflow logic changes what events are expected. Old history doesn't match new code paths.
Simple explanation: You change the rules of a game mid-play. Players who started under old rules can't follow new rules.
Technical Answer:
- Use
workflow.version()to mark code changes; old history replays with old code path, new workflows use new path. - Prefer additive changes only (new signals, new activities) that don't break old history.
- Avoid changing the order or existence of workflow-deterministic calls.
- Test replay explicitly: start a workflow with old code, upgrade, and verify it still completes.
Code/Syntax Justification:
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order_id):
if workflow.version("add-refund-step", "1.0", "2.0"):
await workflow.execute_activity(refund_customer, order_id)
await workflow.execute_activity(charge_customer, order_id)
Workflow Diagram:
Old code deployed --> workflows A, B in-flight (history recorded)
|
v
New code deployed (changed step order)
|
v
Without version(): With workflow.version():
A replays with new A replays with old code path (v1.0)
code -> fails B (new) uses new code path (v2.0)
🔍 Behind the Question: Workflow versioning, replay compatibility, safe deployments.
❌ The Red Flags: Changing workflow step order without versioning; no replay testing; breaking in-flight workflows on deploy.
Q55: Activity heartbeats — long-running activity appears stuck
Level: Senior
Scenario: An activity processes a large file (10 minutes). Temporal marks it as timed out after schedule_to_close_timeout (default 5 minutes) because no heartbeat. The workflow retries the activity from the beginning. Work never completes.
Impact: Long activities never complete. Infinite retry loop. Wasted compute.
Root Cause: Activity doesn't send heartbeats. Temporal assumes it's dead after heartbeat_timeout. Workflow retries from scratch.
Simple explanation: You're cooking a 10-minute dish. The timer beeps at 5 minutes. The chef assumes you died and starts over. You never finish cooking.
Technical Answer:
- Set
heartbeat_timeoutshorter than activity duration (e.g., 60 seconds). - Call
activity.heartbeat()periodically in the activity code. - Set
schedule_to_close_timeoutto total expected duration + retries. - On retry, use
last_heartbeatdetails to resume from checkpoint (activity is idempotent). - Record progress details in heartbeat so retries can resume.
Code/Syntax Justification:
from temporalio import activity
@activity.defn
async def process_large_file(file_path: str):
total = get_file_size(file_path)
processed = 0
with open(file_path, "rb") as f:
while chunk := f.read(8192):
process_chunk(chunk)
processed += len(chunk)
# Heartbeat with progress
activity.heartbeat(
details={"bytes_processed": processed, "total": total}
)
Workflow Diagram:
Long activity (10 min): With heartbeats:
Start Start
| |
v v
5 min timeout (no heartbeat) 1 min heartbeat (progress: 20%)
| |
v v
Workflow retries from start Worker continues, heartbeats every 1 min
| |
v v
Never completes Completes at 10 min
🔍 Behind the Question: Activity heartbeats, timeout configuration, retry from checkpoint, activity details.
❌ The Red Flags: No heartbeat in long activity; heartbeat_timeout too short; not recording progress in heartbeat details.
Q56: Activity retry storm — downstream service overwhelmed
Level: Principal
Scenario: A payment activity calls a downstream payment gateway. The gateway is degraded (slow/errors). Temporal retries the activity with default backoff. Hundreds of workflows retry simultaneously, hammering the already-degraded gateway and making it worse.
Impact: Retry storm. Complete outage of the payment gateway. Cascading failure.
Root Cause: Uncontrolled retries without exponential backoff, jitter, or rate limiting. All workflows retry at the same interval.
Simple explanation: A store's phone line is busy. 100 people all redial every 5 seconds. The line never frees up.
Technical Answer:
- Use exponential backoff with jitter (
retry_policywithbackoff_coefficient,maximum_interval). - Set
maximum_attemptsso retries stop after N tries. - Use
retry_policyper activity; distinguish retryable vs non-retryable errors. - Add circuit breaker in the activity itself (stop calling if gateway is down).
- Monitor activity failure rate and task queue backlog.
Code/Syntax Justification:
@activity.defn
async def charge_customer(order_id, amount):
return await payment_gateway.charge(order_id, amount)
# In workflow
await workflow.execute_activity(
charge_customer,
args=[order_id, amount],
retry_policy=RetryPolicy(
initial_interval=timedelta(seconds=1),
backoff_coefficient=2.0,
maximum_interval=timedelta(minutes=5),
maximum_attempts=5,
),
)
Workflow Diagram:
Gateway degraded
|
v
Without backoff: With exponential backoff + jitter:
retry every 1s retry 1s, 2s, 4s, 8s... + random jitter
| |
v v
100 workflows retry together workflows spread out over time
gateway overwhelmed gateway recovers
🔍 Behind the Question: Retry policy, exponential backoff, jitter, circuit breaker, retry storms.
❌ The Red Flags: No retry limits; fixed short interval retries; retrying non-retryable errors (auth failures).
Q57: Workflow task queue saturation — worker starvation
Level: Senior
Scenario: A Temporal workflow processes customer onboarding. Task queue backlog grows to 50,000 pending workflow tasks. Workers are at 100% CPU but can't drain the queue. New workflows wait hours before starting.
Impact: Workflow latency. Customer onboarding delayed. SLA breached.
Root Cause: Worker concurrency too low for the workflow start rate. Each worker slot handles one workflow task at a time. More workflows arrive than workers can process.
Simple explanation: A factory conveyor brings 1000 boxes/hour. 10 workers can pack 100/hour. Boxes pile up. Need more workers or faster packing.
Technical Answer:
- Increase
max_concurrent_workflow_tasksper worker (up to CPU/memory limits). - Scale workers horizontally (more worker instances/pods).
- Use separate task queues for different workflow types to isolate backlog.
- Rate-limit workflow starts if downstream can't keep up.
- Monitor
temporal_workflow_tasks_backlogandtemporal_worker_task_slot_usage.
Code/Syntax Justification:
worker = Worker(
client,
task_queue="onboarding",
workflows=[OnboardingWorkflow],
max_concurrent_workflow_tasks=100, # tune based on CPU/memory
max_concurrent_activity_tasks=200,
)
Workflow Diagram:
Workflow start rate > Worker capacity
|
v
Task queue backlog grows
|
v
Options:
+-- Increase max_concurrent_workflow_tasks
+-- Add worker instances (horizontal scale)
+-- Separate task queues per workflow type
+-- Rate-limit workflow starts
|
v
Backlog drains
🔍 Behind the Question: Worker concurrency, task queue backlog, horizontal scaling, rate limiting.
❌ The Red Flags: One worker for all workflows; no backlog monitoring; not rate-limiting starts.
Q58: Signal handling — workflow misses signal sent before wait
Level: Senior
Scenario: A workflow waits for an approval signal with workflow.wait_condition. The signal is sent before the workflow reaches the wait. The workflow misses it and waits forever (or until timeout).
Impact: Workflow stuck. Approval never processed. Manual intervention needed.
Root Cause: Signal sent before workflow calls wait_condition. Temporal buffers signals but the workflow must be in a waiting state to consume them, or use a signal handler that sets a flag checked by the wait condition.
Simple explanation: You knock on the door before the person sits down to wait for the knock. They sit down and wait for a knock that already happened.
Technical Answer:
- Use a signal handler that sets a boolean flag (e.g.,
self.approved = True). wait_conditionchecks the flag:await workflow.wait_condition(lambda: self.approved).- If signal already arrived, flag is True, wait returns immediately.
- Combine with
workflow.wait_conditiontimeout for safety. - Signals are buffered by Temporal even if workflow isn't waiting.
Code/Syntax Justification:
@workflow.defn
class ApprovalWorkflow:
def __init__(self):
self.approved = False
@workflow.signal
def approve(self):
self.approved = True
@workflow.run
async def run(self):
# If signal already sent, this returns immediately
await workflow.wait_condition(lambda: self.approved, timeout=timedelta(days=7))
if not self.approved:
raise TimeoutError("Approval expired")
await workflow.execute_activity(process_approval)
Workflow Diagram:
Signal before wait: With flag + wait_condition:
Signal sent Signal sets self.approved = True
| |
v v
Workflow reaches wait Workflow reaches wait_condition
| |
v v
Waits forever (missed signal) wait_condition sees flag=True, returns immediately
🔍 Behind the Question: Signal handling, wait_condition, signal buffering, race condition.
❌ The Red Flags: Waiting for signal without flag; signal sent before workflow starts waiting; no timeout on wait_condition.
Q59: Child workflow failures — parent doesn't handle child failure
Level: Senior
Scenario: A parent workflow starts 5 child workflows (one per region). One child fails (e.g., region-specific payment gateway down). Parent doesn't handle the failure. Parent completes as if all succeeded. Data inconsistency.
Impact: Silent data inconsistency. Missing region processing. No alerting.
Root Cause: Parent workflow doesn't catch child workflow exceptions. execute_child_workflow raises on failure; if not caught, parent fails. But if parent catches and continues, it may not reflect partial failure.
Simple explanation: You send 5 assistants on errands. One fails. You don't check. You report "all done" when one task is undone.
Technical Answer:
- Wrap
execute_child_workflowin try/except per child. - Use
workflow.execute_child_workflowwithretry_policyfor transient failures. - Collect results/failures; decide whether to continue or fail parent based on business logic.
- Use
workflow.get_external_workflow_handlefor long-running external workflows. - Consider
workflow.wait_for_child_workflow_closefor explicit failure handling.
Code/Syntax Justification:
@workflow.defn
class ParentWorkflow:
@workflow.run
async def run(self, regions: list[str]):
results = []
for region in regions:
try:
result = await workflow.execute_child_workflow(
RegionWorkflow.run,
region,
id=f"region-{region}",
retry_policy=RetryPolicy(maximum_attempts=3),
)
results.append(result)
except Exception as exc:
results.append({"region": region, "error": str(exc)})
# Decide: fail if any region failed, or continue with partial success
if any("error" in r for r in results):
raise ApplicationError("Some regions failed", results)
return results
Workflow Diagram:
Parent starts 5 child workflows
|
v
Child 3 fails (payment gateway down)
|
v
Without handling: With try/except per child:
Parent reports success Catch exception, record failure
| |
v v
Data inconsistency Decide: fail parent or continue partial
🔍 Behind the Question: Child workflow failure handling, retry policy, partial failure, error aggregation.
❌ The Red Flags: Not catching child workflow exceptions; assuming all children succeed; no partial failure strategy.
Q60: Workflow timeout — workflow_execution_timeout too short
Level: Senior
Scenario: A workflow processes a monthly billing cycle. It takes 3 days (waiting for payment confirmations). workflow_execution_timeout is 24 hours (default). Workflow times out mid-execution. History is lost. Billing incomplete.
Impact: Workflow terminated. Business process incomplete. Data loss.
Root Cause: workflow_execution_timeout (max time from start to close) is shorter than actual business process duration.
Simple explanation: You give a chef 1 hour to cook a 3-day slow-roast. The chef is stopped at 1 hour. Meal incomplete.
Technical Answer:
- Set
workflow_execution_timeoutto expected max duration + buffer (e.g., 7 days for monthly billing). - Use
workflow_run_timeoutfor actual execution time (excluding wait time). - Use
workflow_task_timeoutfor individual task completion (default 10 seconds). - For very long processes, use
continue_as_newto reset execution time while preserving state. - Monitor
temporal_workflow_execution_durationto set appropriate timeouts.
Code/Syntax Justification:
# Start workflow with appropriate timeouts
handle = await client.start_workflow(
BillingWorkflow.run,
args=[month],
id=f"billing-{month}",
workflow_execution_timeout=timedelta(days=7),
workflow_run_timeout=timedelta(days=3),
workflow_task_timeout=timedelta(seconds=10),
)
Workflow Diagram:
Monthly billing (3 days): With proper timeouts:
Start workflow_execution_timeout = 7 days
| |
v v
24 hour timeout -> terminated Runs 3 days, completes successfully
| |
v v
Billing incomplete Workflow closed, history preserved
🔍 Behind the Question: Workflow timeouts, execution vs run vs task timeout, continue_as_new, timeout tuning.
❌ The Red Flags: Default 24-hour execution timeout for long processes; not distinguishing run vs execution timeout; no continue_as_new for very long workflows.
Q61: continue_as_new — resetting long-running workflow state
Level: Principal
Scenario: A workflow runs for months (e.g., subscription renewal cycle). Event history grows large (thousands of events). Replay becomes slow and expensive. Workflow history hits size limits.
Impact: Slow replay. High history storage cost. Workflow history limit reached.
Root Cause: Workflow never calls continue_as_new. History accumulates forever. Replaying full history on every task is inefficient.
Simple explanation: You keep adding pages to a single notebook. After 1000 pages, finding anything takes forever. You should start a new notebook and carry forward the summary.
Technical Answer:
- Use
workflow.continue_as_new()to start a new execution with updated input, carrying forward state. - Call
continue_as_newperiodically (e.g., every N events or every N days). - Pass relevant state (not full history) as new input arguments.
- Old execution closes; new execution starts fresh with compact history.
- External handles/signal names remain valid across
continue_as_new.
Code/Syntax Justification:
@workflow.defn
class SubscriptionWorkflow:
def __init__(self):
self.renewal_count = 0
@workflow.run
async def run(self, customer_id: str):
# ... do work ...
self.renewal_count += 1
# Continue as new every 12 renewals to keep history small
if self.renewal_count % 12 == 0:
await workflow.continue_as_new(customer_id)
Workflow Diagram:
Single workflow execution (months): With continue_as_new:
History grows: 1000s of events Execution 1: events 1-500
| |
v v
Slow replay, high cost continue_as_new -> Execution 2
High storage cost History: events 1-500 (carried as input)
Execution 2: events 501-1000
🔍 Behind the Question: continue_as_new, history size, replay performance, state preservation across executions.
❌ The Red Flags: Never calling continue_as_new for long-running workflows; history hitting size limits; slow replay.