Skip to main content

Python Testing - TDD, Unit Testing, Integration, E2E, Hypothesis (2026 Edition)

Total Questions: 52 Curriculum: Core Fundamentals (100), Intermediate (120), Advanced (80)


Core Fundamentals​

Q1: What is Test-Driven Development (TDD) and the Red-Green-Refactor cycle?​

Technical Answer: TDD is a development approach where you write tests before writing the implementation code. The Red-Green-Refactor cycle consists of: Red (write a failing test), Green (make the test pass with minimal code), Refactor (improve code quality while keeping tests green).

Code Example:

# Step 1: RED - Write failing test
import pytest

def test_add_numbers():
from calculator import add_numbers
result = add_numbers(2, 3)
assert result == 5 # This will fail initially

# Step 2: GREEN - Write minimal code to pass
# calculator.py
def add_numbers(a, b):
return a + b

# Step 3: REFACTOR - Improve implementation
def add_numbers(a, b):
"""Add two numbers with validation."""
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
raise TypeError("Arguments must be numbers")
return a + b
# TDD with pytest fixtures
@pytest.fixture
def sample_data():
return {"name": "test", "value": 42}

def test_process_data(sample_data):
result = process_data(sample_data)
assert result["processed"] == True

🔍 Interviewer Notes: Testing understanding of TDD workflow and benefits.

❌ Common Mistakes:

  • Writing too much code in Green phase
  • Skipping the Refactor phase
  • Not understanding the value of small increments

Q2: How to write effective unit tests with pytest?​

Technical Answer: Effective unit tests are isolated, fast, and test one thing. Use pytest fixtures for setup, parametrize for multiple inputs, and proper assertions. Follow AAA pattern: Arrange, Act, Assert.

Code Example:

import pytest
from unittest.mock import Mock, patch
from myapp.services import UserService

# AAA Pattern - Arrange, Act, Assert
class TestUserService:

@pytest.fixture
def mock_db(self):
with patch('myapp.services.Database') as mock:
yield mock

@pytest.fixture
def service(self, mock_db):
return UserService(mock_db)

def test_create_user_success(self, service, mock_db):
# Arrange
mock_db.insert.return_value = {"id": 1, "name": "John"}
user_data = {"name": "John", "email": "john@example.com"}

# Act
result = service.create_user(user_data)

# Assert
assert result["id"] == 1
mock_db.insert.assert_called_once_with(user_data)

@pytest.mark.parametrize("age,expected", [
(0, "infant"),
(5, "child"),
(18, "adult"),
(65, "senior")
])
def test_age_category(self, service, age, expected):
assert service.get_age_category(age) == expected

@pytest.mark.parametrize("input_data,error_msg", [
({"name": ""}, "Name cannot be empty"),
({"email": "invalid"}, "Invalid email"),
], ids=["empty_name", "invalid_email"])
def test_create_user_validation(self, service, input_data, error_msg):
with pytest.raises(ValueError, match=error_msg):
service.create_user(input_data)

Custom Fixtures:

# conftest.py - shared fixtures
import pytest
from myapp.models import User

@pytest.fixture(scope="session")
def db_connection():
"""Session-scoped database connection."""
conn = connect_test_db()
yield conn
conn.close()

@pytest.fixture
def test_user(db_connection):
"""Create test user with automatic cleanup."""
user = User(name="Test User", email="test@example.com")
user.save(db_connection)
yield user
user.delete(db_connection)

🔍 Interviewer Notes: Testing knowledge of pytest patterns and test organization.

❌ Common Mistakes:

  • Tests depending on each other (no isolation)
  • Not using fixtures for common setup
  • Testing implementation details instead of behavior

Q3: How to use mocking effectively in Python tests?​

Technical Answer: Mocking replaces dependencies with test doubles to isolate units under test. Use unittest.mock.patch decorators/context managers. Mock at the right level (usually external services, database calls, HTTP requests).

Code Example:

from unittest.mock import Mock, patch, MagicMock, call
import pytest

# Method 1: Patch decorator
@patch('requests.get')
def test_fetch_user(mock_get):
mock_get.return_value.json.return_value = {"id": 1, "name": "John"}

result = fetch_user(1)

mock_get.assert_called_once_with("https://api.example.com/users/1")
assert result["name"] == "John"

# Method 2: Context manager for precise control
def test_fetch_with_error():
with patch('requests.get') as mock_get:
mock_get.return_value.status_code = 404
mock_get.return_value.json.return_value = {}

with pytest.raises(UserNotFoundError):
fetch_user(999)

# Method 3: Mock side effects for multiple calls
def test_retry_logic():
with patch('requests.post') as mock_post:
# First call fails, second succeeds
mock_post.side_effect = [
Mock(status_code=500),
Mock(status_code=200, json=lambda: {"status": "ok"})
]

result = make_request_with_retry()
assert result["status"] == "ok"
assert mock_post.call_count == 2

# MagicMock - auto-creates attributes on access
def test_magic_mock():
service = MagicMock()
service.process.return_value = "result"

# Can chain calls without setup
service.nested.method.call()

# Verify interactions
service.process.assert_called_once()

Mocking Best Practices:

# Mock at the system boundary, not internal implementation
# ❌ Bad - mocks internal method
@patch('myapp.internal.UserService._validate_email')

# ✅ Good - mocks external dependency
@patch('myapp.external.EmailService.send')

# Using spec to match real object interface
from myapp.models import User
mock_user = Mock(spec=User) # Only allows real User attributes/methods

🔍 Interviewer Notes: Testing understanding of isolation and mock boundaries.

❌ Common Mistakes:

  • Over-mocking (mocking everything)
  • Mocking implementation details
  • Not verifying mock interactions

Q4: What are pytest fixtures and how to use them effectively?​

Technical Answer: Pytest fixtures define reusable setup code that tests can depend on. They support different scopes (function, class, module, session), parameterization, and can depend on other fixtures. Use autouse for automatic setup.

Code Example:

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

# Function scope (default) - created for each test
@pytest.fixture
def temp_file(tmp_path):
file_path = tmp_path / "test_data.json"
file_path.write_text('{"test": "data"}')
yield file_path
# Cleanup (automatic with tmp_path)

# Session scope - created once per test session
@pytest.fixture(scope="session")
def database():
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
yield engine
engine.dispose()

# Module scope - created once per test module
@pytest.fixture(scope="module")
def api_client():
client = APIClient(base_url="http://testserver")
yield client
client.close()

# Autouse fixture - runs automatically for all tests
@pytest.fixture(autouse=True)
def reset_state():
# Reset global state before each test
State.reset()
yield
# Cleanup after test

# Fixture with parameters
@pytest.fixture
def user_factory():
def create_user(name="Test", email="test@example.com"):
return User(name=name, email=email)
return create_user

# Using fixtures in tests
class TestUserOperations:
def test_create_user(self, user_factory, database):
user = user_factory(name="John")
saved = save_user(user, database)
assert saved.id is not None

Parametrized Fixtures:

# conftest.py
@pytest.fixture(params=["sqlite", "postgresql"])
def database_type(request):
return request.param

# In test file
def test_connection(database_type):
# Runs twice - once for each database type
conn = connect(database_type)
assert conn.is_connected()

🔍 Interviewer Notes: Testing understanding of fixture lifecycle and scope.

❌ Common Mistakes:

  • Heavy fixtures slowing down tests
  • Not understanding fixture scope
  • Circular fixture dependencies

Q5: How to write integration tests for Python applications?​

Technical Answer: Integration tests verify that multiple components work together. They use real dependencies (database, external APIs) but isolate test data. Use test containers or in-memory databases for consistency.

Code Example:

import pytest
from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

# Use test database
@pytest.fixture
def test_db():
engine = create_engine("postgresql://test:test@localhost/test_db")
Session = sessionmaker(bind=engine)
session = Session()
yield session
session.rollback() # Rollback after each test
session.close()

# Test with real HTTP calls but mocked external services
@pytest.fixture
def client(test_db):
app = create_app(test_db)
return TestClient(app)

class TestUserEndpoints:
def test_create_and_get_user(self, client, test_db):
# Create user
response = client.post(
"/users/",
json={"name": "John", "email": "john@example.com"}
)
assert response.status_code == 201
user_id = response.json()["id"]

# Verify in database
user = test_db.query(User).filter_by(id=user_id).first()
assert user.name == "John"

# Get user
response = client.get(f"/users/{user_id}")
assert response.status_code == 200
assert response.json()["name"] == "John"

Testcontainers for Dependencies:

from testcontainers.postgres import PostgresContainer
import pytest

@pytest.fixture(scope="session")
def postgres_container():
with PostgresContainer("postgres:15") as postgres:
yield postgres

@pytest.fixture
def db_url(postgres_container):
return postgres_container.get_connection_url()

🔍 Interviewer Notes: Testing ability to structure integration tests.

❌ Common Mistakes:

  • Tests depending on execution order
  • Not cleaning up test data
  • Too slow (mocking everything instead)

Q6: What is property-based testing with Hypothesis?​

Technical Answer: Hypothesis generates random test inputs to find edge cases you might not think of. Define properties that should always be true, and Hypothesis finds minimal failing examples. Much more powerful than example-based testing.

Code Example:

from hypothesis import given, strategies as st, assume, settings
import pytest

# Basic property testing
@given(st.integers(), st.integers())
def test_addition_commutative(a, b):
assert add(a, b) == add(b, a)

@given(st.lists(st.integers()))
def test_reverse_twice_unchanged(lst):
assert reverse(reverse(lst)) == lst

# More complex strategies
@given(st.dictionaries(
keys=st.text(min_size=1),
values=st.integers(),
min_size=1
))
def test_dict_key_preservation(d):
result = process_dict(d)
for key in d:
assert key in result

# Custom strategies
@st.composite
def user_strategy(draw):
return User(
name=draw(st.text(min_size=1, max_size=50)),
age=draw(st.integers(min_value=0, max_value=150)),
email=draw(st.emails())
)

@given(user_strategy())
def test_user_serialization(user):
json_str = user.to_json()
restored = User.from_json(json_str)
assert restored.name == user.name

# Filtering with assume
@given(st.integers())
def test_division_safe(x):
assume(x != 0) # Skip invalid inputs
assume(x % 2 == 0) # Only even numbers
assert divide_by_2(x) == x // 2

# Settings for more examples
@settings(max_examples=1000)
@given(st.text())
def test_string_function_robust(text):
result = process_string(text)
assert result is not None

Finding Minimal Examples:

# When test fails, Hypothesis shrinks to minimal example
@given(st.lists(st.integers(), min_size=1))
def test_sort_never_longer(lst):
sorted_lst = sort_function(lst)
assert len(sorted_lst) == len(lst) # Catches sorting that drops elements

🔍 Interviewer Notes: Testing understanding of advanced testing techniques.

❌ Common Mistakes:

  • Properties that don't actually test behavior
  • Not understanding shrinking
  • Flaky tests with Hypothesis

Q7: How to structure end-to-end (E2E) tests for web applications?​

Technical Answer: E2E tests verify complete user workflows through the UI or API. They're slower but catch integration issues. Use Page Object Model for UI tests, focus on critical paths, and ensure proper test data isolation.

Playwright Example:

import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture
def page():
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
yield page
browser.close()

class TestUserJourney:
def test_signup_to_dashboard(self, page):
# Navigate to signup
page.goto("https://example.com/signup")

# Fill form
page.fill('input[name="email"]', "test@example.com")
page.fill('input[name="password"]', "SecurePass123!")
page.click('button[type="submit"]')

# Wait for dashboard
page.wait_for_url("**/dashboard")

# Verify logged in
assert page.is_visible("text=Welcome, test@example.com")

# Complete workflow
page.click("text=Create Project")
page.fill('input[name="project_name"]', "My Project")
page.click('button:has-text("Create")')

# Verify project created
page.wait_for_selector("text=My Project")

Pytest with API Testing:

@pytest.fixture
def authenticated_client(client):
response = client.post("/auth/login", json={
"email": "test@example.com",
"password": "password"
})
token = response.json()["token"]
client.headers = {"Authorization": f"Bearer {token}"}
return client

class TestOrderFlow:
def test_complete_order_journey(self, authenticated_client):
# Add to cart
response = authenticated_client.post("/cart", json={"product_id": 1})
cart_id = response.json()["cart_id"]

# Checkout
response = authenticated_client.post(f"/checkout/{cart_id}", json={
"payment_method": "card",
"address": "123 Test St"
})
order_id = response.json()["order_id"]

# Verify order status
response = authenticated_client.get(f"/orders/{order_id}")
assert response.json()["status"] == "confirmed"

🔍 Interviewer Notes: Testing understanding of test scope and critical paths.

❌ Common Mistakes:

  • Too many E2E tests (slow feedback)
  • Not isolating test data
  • Testing too many edge cases

Q8: How to measure and improve test coverage?​

Technical Answer: Use coverage.py with pytest-cov to measure line, branch, and condition coverage. Aim for meaningful coverage on business logic, not just hitting lines. Use coverage reports to identify untested critical paths.

Setup and Usage:

# Run tests with coverage
pytest --cov=myapp --cov-report=html --cov-report=term-missing

# Coverage thresholds
pytest --cov=myapp --cov-fail-under=80
# pyproject.toml coverage settings
[tool.coverage.run]
source = ["myapp"]
omit = ["*/tests/*", "*/__init__.py"]
branch = true

[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise NotImplementedError",
]
# Exclude only code that genuinely can't run in tests
def complex_function(x):
if x > 0:
return process_positive(x)
elif x < 0:
return process_negative(x)
else:
return 0


if __name__ == "__main__": # pragma: no cover - script entry point, not run by tests
complex_function(1)

Interpreting Coverage:

# Focus on missing critical paths
Name Stmts Miss Cover Missing
-----------------------------------------------------
myapp/core.py 50 5 90% 23, 45-48, 67
myapp/utils.py 30 2 93% 15, 22

🔍 Interviewer Notes: Testing understanding of coverage quality vs quantity.

❌ Common Mistakes:

  • 100% coverage without meaningful tests
  • Missing branch coverage
  • Not reviewing coverage reports

Q9: What are test best practices for maintainable test suites?​

Technical Answer: Organize tests by feature, use descriptive names, keep tests fast and independent, review test code quality, and maintain test documentation. Follow testing pyramid: many unit, fewer integration, fewest E2E.

Conventions:

# Test file structure
tests/
├── unit/ # Fast isolated tests
│ ├── test_models.py
│ ├── test_services.py
│ └── test_utils.py
├── integration/ # Component interaction
│ ├── test_api.py
│ └── test_db.py
└── e2e/ # Full system tests
└── test_flows.py

# Test naming convention: test_<module>_<method>_<expected_behavior>
def test_user_service_create_user_with_valid_data():
"""Valid users should be created successfully."""
# Test implementation
pass

def test_user_service_create_user_with_empty_name_raises_error():
"""Empty name should raise ValueError."""
with pytest.raises(ValueError, match="Name cannot be empty"):
create_user("")

Test Patterns:

# Given-When-Then for behavior-driven clarity
def test_user_login_flow():
# Given: A registered user
user = UserFactory(email="test@example.com")

# When: user attempts login
result = login(user.email, "correct_password")

# Then: login succeeds
assert result.success is True
assert result.token is not None

# Single assertion per test for clear failures
def test_user_creation_sets_correct_defaults():
user = create_user("John")
assert user.status == "active" # One clear assertion

🔍 Interviewer Notes: Testing test maintenance and organization skills.

❌ Common Mistakes:

  • Poor test organization
  • Tests that are hard to understand
  • Not maintaining test code quality

Q10: How to debug failing tests effectively?​

Technical Answer: Use pytest's built-in debugging features: -v for verbose output, --tb=short/long/line for traceback control, --pdb for interactive debugging, and custom assertion helpers. Add logging to understand test state.

Debugging Commands:

# Verbose output with custom messages
pytest -v --tb=short

# Drop into debugger on failure
pytest --pdb --pdbcls=IPython.terminal.debugger:TerminalPdb

# Stop after first failure
pytest -x

# Re-run last failed tests
pytest --lf

# Run specific test with full traceback
pytest test_file.py::test_specific -v --tb=long
# Custom assertion messages
def test_api_response_format(response):
assert response.status_code == 200, f"Expected 200, got {response.status_code}"
assert "id" in response.json(), f"Missing 'id' in response: {response.json()}"

# Using pytest logging
def test_with_logging(caplog):
with caplog.at_level("DEBUG"):
result = complex_operation()

assert result.success
assert "Processing complete" in caplog.text

IDE Debugging:

import pytest
import pdb

def test_complex_logic():
result = complex_function()

# Conditional breakpoint
if result.value > threshold:
pdb.set_trace() # Only break when condition met

assert result.valid

🔍 Interviewer Notes: Testing troubleshooting methodology.

❌ Common Mistakes:

  • Not understanding pytest's debugging tools
  • Adding print statements instead of proper debugging
  • Ignoring fixture interaction in failures

Q11: What is the difference between a unit test, integration test, and E2E test?​

Unit: tests one function/class in isolation with mocks (fast, run everywhere). Integration: tests real interactions — DB, API, services (slower, needs fixtures). E2E: tests the whole system through the user interface (slowest, flaky, most realistic). Pyramid: many unit, fewer integration, few E2E.

Q12: What is pytest parametrize and when do you use it?​

@pytest.mark.parametrize runs the same test with multiple input/output sets. Use for boundary values, error cases, and data-driven tests — one test, many cases, no duplication. Naming each case via ids improves failure readability.

Q13: What is monkeypatch and how is it different from mock?​

monkeypatch dynamically replaces attributes/functions during a test (fixture-authentic, no library import needed). Mock (unittest.mock) creates fake objects and records calls/return values. Use monkeypatch for simple dependency swaps, Mock for verifying call patterns and return values.

Q14: What are pytest conftest.py files?​

conftest.py holds shared fixtures, hooks, and plugins for tests in that directory tree. Fixtures defined there are auto-available without imports. Use for DB sessions, API clients, and test data builders shared across test modules.

Q15: What is coverage and when is "100% coverage" a trap?​

Coverage measures executed lines/branches divided by total. 100% doesn't mean correct — you can execute code without asserting meaningful behavior. Target meaningful coverage on critical paths (services, business logic) and prioritize assertions over line-count chasing.

Q16: How do you test time-dependent code?​

Freeze time with freezegun.freeze_time or time-machine (pytest-timeout is different: it aborts slow tests), inject a clock (callable returning now) instead of calling datetime.now() directly. Testing expiry, TTLs, and scheduled behavior requires determinism — a fake clock makes tests repeatable.

Q17: What is snapshot testing?​

Capture expected output once, then compare future runs against the stored snapshot (pytest-snapshot, syrupy). Catches unintended changes in output (serializers, HTML, configs). Careful: snapshots become stale if reviewed carelessly — treat snapshot diffs as code reviews.

Q18: How do you test async code with pytest?​

Use pytest-asyncio and @pytest.mark.asyncio for async tests, or anyio fixtures for multi-backend support. Mock async functions with AsyncMock; await within tests; avoid mixing sync and async incorrectly in fixtures.

Q19: What are test fixtures vs test factories vs builders?​

Fixture: reusable setup (session/function scope) with teardown. Factory: function that creates test data on demand (e.g., make_user(role="admin")). Builder: fluent object construction for complex entities. Use factories for data variation, fixtures for environment (DB, client).

Q20: How do you test that an exception is raised?​

Use pytest.raises(ExceptionType) as a context manager — also capture the exception message (match=), or with pytest.raises(...) as exc_info to assert on attributes. Never rely on try/except + assert in the except block to pass a test.


Senior / Principal — Real World Testing Scenarios​

Q21: Flaky tests — intermittent failures on CI​

Level: Senior

Scenario: A test suite passes locally but fails randomly on CI (1 in 10 runs). Developers re-run until green and merge. Over time, trust in the suite erodes.

Impact: CI waste. Merged bugs. "It passed on CI" becomes meaningless.

Root Cause: Test ordering dependence, shared mutable state, async timing, random data, network calls, or datetime sensitivity.

Simple explanation: A test passes when the stars align but fails when a race condition or leftover state from a previous test interferes.

Technical Answer:

  1. Run tests in isolation to reproduce: rerun the failing order with pytest --randomly-seed=<seed> (pytest-randomly) or repeat a test with pytest --count 50 (pytest-repeat).
  2. Eliminate shared state: use function-scoped fixtures, clean DB between tests, unique test data per run.
  3. Mock time (freezegun) and external services (responses/httpx mock).
  4. Quarantine flaky tests (a quarantine marker excluded from the blocking job; pytest --reruns 2 from pytest-rerunfailures only as a temporary measure) and fix before re-enabling.
  5. Never skip or ignore flaky tests — they hide real regressions.

Code/Syntax Justification:

from datetime import datetime

import freezegun
from myapp.services import create_order

@freezegun.freeze_time("2026-01-15 12:00:00")
def test_order_expiry(db_session):
order = create_order(db_session, expires_in_minutes=30)
assert order.expires_at == datetime(2026, 1, 15, 12, 30, 0)

Workflow Diagram:

Test fails on CI
|
v
Run locally 10x: passes
|
v
Run with --count 50 and the CI seed: reproduces
|
v
Root cause: shared DB state / datetime.now() / network
|
v
Fix: function-scoped fixtures, freezegun, mocks
|
v
Test deterministic: pass every run

🔍 Behind the Question: Test isolation, determinism, flaky test quarantine, CI reproducibility.

❌ The Red Flags: Re-running until green; @pytest.mark.flaky without fixing; tests using datetime.now() or real network.


Q22: Test coverage is 100% but bugs still reach production​

Level: Principal

Scenario: The team proudly reports 100% line coverage. Yet a critical bug (race condition in payment) reaches production and causes double charges.

Impact: False confidence. Coverage metrics gamed. Real bugs undetected.

Root Cause: Line coverage measures executed lines, not logical branches or business scenarios. It doesn't test edge cases, concurrency, or integration points.

Simple explanation: You tested every line of the parachute manual but never jumped from a plane.

Technical Answer:

  1. Measure branch coverage (not just lines) — target >90%.
  2. Add property-based testing (Hypothesis) for edge cases.
  3. Add integration/E2E tests for critical user journeys (checkout, payment).
  4. Add concurrency/load tests for race conditions.
  5. Use mutation testing (mutmut) to verify tests actually catch faults.

Code/Syntax Justification:

from hypothesis import given, strategies as st

@given(st.integers(min_value=0, max_value=1_000_000))
def test_discount_never_negative(amount):
result = apply_discount(amount, 0.1)
assert result >= 0

Workflow Diagram:

100% line coverage
|
v
Bug in branch not tested (race condition, edge case)
|
v
Production incident
|
v
Fix: branch coverage + property-based + integration + mutation testing
|
v
Real bugs caught before production

🔍 Behind the Question: Coverage limits, branch coverage, property-based testing, mutation testing, integration testing.

❌ The Red Flags: 100% line coverage as goal; no branch coverage; no integration tests for critical paths.


pytest Mechanics in Depth​

Q23: Test pyramid vs testing trophy — which shape should a backend test suite have?​

Technical Answer: The test pyramid (Mike Cohn) says: many fast unit tests at the bottom, fewer integration tests in the middle, very few end-to-end tests on top. The testing trophy (Kent C. Dodds) adds static analysis (type checker, linter) as the base and makes integration tests the widest layer, arguing they give the most confidence per unit of cost. Neither is a law. The shape follows where your bugs actually come from:

  • Logic-heavy code (pricing engines, parsers, state machines) → pyramid. Bugs live inside functions, and unit tests find them cheaply.
  • Glue-heavy services (CRUD APIs, ORM queries, serializers, message handlers) → trophy. Bugs live at the seams (SQL, serialization, auth wiring), so unit tests with mocks prove very little.
  • Either way, keep E2E tests for a handful of revenue-critical journeys only.

The real goal is fast feedback plus confidence. Track each layer's runtime and how many escaped bugs each layer would have caught, then rebalance.

Code Example:

# pyproject.toml - make the layers explicit so CI can run them separately
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"integration: talks to a real database or broker",
"e2e: drives the deployed system end to end",
]
# Pre-push / PR job: static checks + unit tests (seconds)
ruff check . && mypy src && pytest -m "not integration and not e2e"

# PR job, in parallel: integration layer against real services
pytest -m integration

# Post-deploy / nightly: the few critical journeys
pytest -m e2e

🔍 Interviewer Notes: Strong candidates say the shape depends on the kind of code and back it up with data on escaped bugs and runtimes. They don't just recite the pyramid.

❌ Common Mistakes:

  • Treating a mock-heavy unit suite as proof that the database layer works
  • Building an "ice-cream cone" (mostly E2E/manual), which gives slow and flaky feedback
  • Not counting static typing as a testing layer

Q24: How does pytest resolve fixtures — scope ordering, teardown order, and ScopeMismatch?​

Technical Answer: For each test, pytest builds a graph of the fixtures it requests and instantiates them higher scope first (session → package → module → class → function). Within the same scope, it goes in dependency order and then in the order they were requested. Each fixture instance is cached for its scope. Teardown (code after yield, or callbacks registered with request.addfinalizer) runs in reverse (LIFO) order, so a fixture is always torn down before the fixtures it depends on.

A fixture cannot depend on a narrower-scoped fixture. If a session fixture requests a function-scoped one, pytest raises ScopeMismatch, because the session fixture would outlive the function fixture's value. Use the factory variants (tmp_path_factory instead of tmp_path) or widen the scope of the dependency.

Code Example:

import pytest

events: list[str] = []


@pytest.fixture(scope="module")
def db():
events.append("db setup")
yield "db-handle"
events.append("db teardown")


@pytest.fixture
def user(db):
events.append("user setup")
yield {"name": "ada", "db": db}
events.append("user teardown") # runs before db teardown


@pytest.fixture
def audit(request):
log: list[str] = []
# Equivalent to code after `yield`, but can be registered conditionally
request.addfinalizer(lambda: events.append(f"audit flushed {len(log)} entries"))
return log


def test_user_is_bound_to_db(user, audit):
audit.append("viewed profile")
assert user["db"] == "db-handle"


# Session-scoped config file: use tmp_path_factory, not tmp_path
@pytest.fixture(scope="session")
def config_file(tmp_path_factory):
path = tmp_path_factory.mktemp("config") / "app.toml"
path.write_text('env = "test"\n')
return path


def test_config_file_exists(config_file):
assert config_file.read_text() == 'env = "test"\n'
# See the exact setup/teardown order pytest chose
pytest --setup-show -q tests/test_fixture_order.py

🔍 Interviewer Notes: Look for an understanding of LIFO teardown, per-scope caching, and why ScopeMismatch exists. --setup-show is a good sign of hands-on debugging.

❌ Common Mistakes:

  • Putting expensive setup in function scope "to be safe" and making the suite slow
  • Widening scope to session for a fixture that holds mutable state, which leaks state between tests
  • Doing cleanup before yield or forgetting teardown when setup partially fails

Q25: How do you use pytest markers — custom markers, skip, skipif, and strict xfail?​

Technical Answer: Markers attach metadata to tests. Built-ins: skip (never run), skipif(condition, reason=...) (conditional skip), xfail (expected to fail), parametrize, usefixtures, filterwarnings. Custom markers (slow, integration) let you select subsets with -m "slow and not integration".

Best practices:

  • Register every custom marker and run with --strict-markers so a typo like @pytest.mark.intergration fails loudly instead of silently selecting nothing.
  • Use xfail(strict=True) for known bugs. If the bug gets fixed, the test XPASSes and fails the run, which reminds you to remove the marker.
  • Use pytest.importorskip("module") for optional dependencies instead of try/except around imports.

Code Example:

# pyproject.toml
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = [
"slow: takes more than a second",
"integration: needs real services",
]
import os
import sys

import pytest


@pytest.mark.slow
def test_bulk_checksum():
data = os.urandom(5_000_000)
assert len(data) == 5_000_000


@pytest.mark.skipif(sys.platform == "win32", reason="relies on os.fork")
def test_fork_available():
assert hasattr(os, "fork")


# round(2.675, 2) is 2.67 because 2.675 cannot be represented exactly in binary.
# strict=True turns an unexpected pass into a failure.
@pytest.mark.xfail(reason="float rounding; switch to Decimal", strict=True)
def test_rounding_half_up():
assert round(2.675, 2) == 2.68


def test_optional_numpy_feature():
np = pytest.importorskip("numpy")
assert np.array([1, 2, 3]).sum() == 6
pytest -m "not slow" # fast inner loop
pytest -m "integration and not slow"
pytest -rxXs # report reasons for skipped / xfailed / xpassed

🔍 Interviewer Notes: --strict-markers and xfail(strict=True) are signs of someone who has seen a suite quietly rot.

❌ Common Mistakes:

  • Using skip for broken tests with no ticket or owner, so they stay skipped forever
  • Non-strict xfail that keeps "passing" after the bug is fixed
  • Unregistered markers that silently select zero tests in CI

Q26: What are advanced parametrize techniques — pytest.param, ids, stacking, and indirect?​

Technical Answer:

  • pytest.param(..., id="...", marks=...) gives a single case a readable ID and per-case marks (for example, one case xfail).
  • Stacking @pytest.mark.parametrize decorators produces the Cartesian product of the arguments.
  • indirect=True routes the parameter value to a fixture (as request.param) instead of the test, so the fixture can build something expensive or stateful from it.
  • ids= can be a list or a callable when you can't use pytest.param (for example, data loaded from a file).

Code Example:

import pytest


@pytest.mark.parametrize(
("raw", "expected"),
[
pytest.param("42", 42, id="plain"),
pytest.param(" 7 ", 7, id="surrounding-whitespace"),
pytest.param("-3", -3, id="negative"),
pytest.param("1_000", 1000, id="underscore-separator"),
pytest.param(
"0x1F", 31, id="hex-not-supported",
marks=pytest.mark.xfail(raises=ValueError, strict=True),
),
],
)
def test_int_parsing(raw, expected):
assert int(raw) == expected


PUBLIC_ROUTES = {("GET", "/health"), ("HEAD", "/health"), ("GET", "/ready"), ("HEAD", "/ready")}


def is_public(method: str, path: str) -> bool:
return (method, path) in PUBLIC_ROUTES


# Stacked: 2 methods x 2 paths = 4 generated tests
@pytest.mark.parametrize("method", ["GET", "HEAD"])
@pytest.mark.parametrize("path", ["/health", "/ready"])
def test_probe_routes_are_public(method, path):
assert is_public(method, path)


# Indirect: the value goes to the fixture, which builds the object
@pytest.fixture
def account(request):
role = request.param
return {"name": "ada", "role": role, "can_export": role == "admin"}


@pytest.mark.parametrize(
("account", "allowed"),
[("admin", True), ("viewer", False)],
indirect=["account"],
)
def test_export_permission(account, allowed):
assert account["can_export"] is allowed

🔍 Interviewer Notes: Candidates should know that stacking multiplies test counts. That cost matters when a slow fixture is involved.

❌ Common Mistakes:

  • Default IDs like test_x[raw3-expected3] that make CI failures unreadable
  • Stacking parametrize on slow integration tests and causing a combinatorial explosion
  • Putting branching logic inside one parametrized test instead of splitting cases

Q27: How do you configure pytest for a professional project (pyproject.toml)?​

Technical Answer: Put configuration in [tool.pytest.ini_options] so local runs and CI behave the same. Important settings:

  • testpaths: only collect from tests/, which speeds up collection and avoids picking up vendored code.
  • addopts: -ra (summary of non-passing tests), --strict-markers, --strict-config (fail on unknown config keys).
  • --import-mode=importlib: avoids sys.path hacks and lets you use duplicate test file basenames. This is the mode the pytest docs recommend for new projects.
  • xfail_strict = true: makes every xfail strict by default.
  • filterwarnings = ["error", ...]: turns warnings into errors so you catch deprecations before an upgrade breaks you, with targeted ignores for third-party noise.
  • minversion: fails fast on an old pytest.

Code Example:

[tool.pytest.ini_options]
minversion = "8.0"
testpaths = ["tests"]
addopts = "-ra --strict-markers --strict-config --import-mode=importlib"
xfail_strict = true
filterwarnings = [
"error",
# Narrow, documented exception for a dependency we cannot fix yet
"ignore::DeprecationWarning:some_third_party_lib",
]
markers = [
"slow: takes more than a second",
"integration: needs real services",
]
# Confirm which config file and rootdir pytest picked up
pytest --co -q | head
pytest --version

🔍 Interviewer Notes: filterwarnings = error and --strict-config show that the candidate thinks about upgrades and suite hygiene, not just writing tests.

❌ Common Mistakes:

  • Options living only in a developer's shell alias, so CI behaves differently
  • Global ignore::DeprecationWarning, which hides upcoming breakage
  • No testpaths, so pytest crawls node_modules or virtualenvs

Mocking and Test Doubles​

Q28: Why must you "patch where it is looked up, not where it is defined"?​

Technical Answer: unittest.mock.patch("a.b.name") replaces the attribute name on the module object a.b, and nothing else. from app.email import send_email copies a reference into the importing module's namespace when the import runs. If you patch app.email.send_email, the name already bound in app.notifications still points at the original function. Patch the name in the module that uses it (app.notifications.send_email).

If the code calls app.email.send_email(...) through the module attribute (import app.email), patching app.email.send_email works, because the lookup happens at call time.

A related trap is that built-in C types can't be patched: patch("datetime.datetime.now") raises TypeError because you can't set attributes on the built-in datetime type. Patch the name your module imported, inject a clock, or use a time library.

Code Example:

# app/email.py
def send_email(to: str, subject: str) -> None:
raise RuntimeError("would hit SMTP")
# app/notifications.py
from app.email import send_email


def notify_signup(email: str) -> None:
send_email(email, "Welcome!")
# tests/test_notifications.py
from unittest.mock import patch

import pytest

from app.notifications import notify_signup


def test_patching_definition_site_does_not_work():
with patch("app.email.send_email"):
with pytest.raises(RuntimeError): # original function still called
notify_signup("ada@example.com")


def test_patch_where_looked_up():
with patch("app.notifications.send_email") as fake_send:
notify_signup("ada@example.com")
fake_send.assert_called_once_with("ada@example.com", "Welcome!")

🔍 Interviewer Notes: This is one of the most common real-world mocking bugs. A candidate who explains name binding at import time, rather than reciting the rule, understands Python's module model.

❌ Common Mistakes:

  • Patching the library's module when the code under test did from lib import thing
  • Concluding "mocking doesn't work here" and adding sleeps or real network calls
  • Trying to patch methods on built-in types such as datetime.datetime

Q29: What is autospec and why should mocks use it?​

Technical Answer: A plain Mock/MagicMock accepts any attribute and any call signature. If the real API changes (an argument is renamed, a parameter becomes required, a method is removed), mock-based tests keep passing while production breaks. create_autospec(obj) or patch(..., autospec=True) builds a mock from the real object's signature:

  • Calling with the wrong arguments raises TypeError.
  • Accessing attributes that don't exist raises AttributeError.
  • spec_set=True additionally blocks setting attributes that don't exist.

Limits: autospec checks call signatures, not return values. A mock can still return a shape the real method never returns. Autospec also introspects lazily, so attributes created in __init__ aren't visible to it. For patched methods with autospec=True, the mock receives self as its first argument.

Code Example:

from unittest.mock import Mock, create_autospec, patch

import pytest


class PaymentGateway:
def charge(self, amount_cents: int, currency: str, *, idempotency_key: str) -> str:
raise NotImplementedError("calls the real provider")


def checkout(gateway: PaymentGateway, amount_cents: int) -> str:
return gateway.charge(amount_cents, "usd", idempotency_key="order-1")


def test_plain_mock_hides_signature_errors():
gateway = Mock()
gateway.charge(100) # wrong arity, no error
gateway.refund_everything() # method does not exist, no error


def test_autospec_enforces_the_real_interface():
gateway = create_autospec(PaymentGateway, instance=True)
with pytest.raises(TypeError):
gateway.charge(100) # missing currency and idempotency_key
with pytest.raises(AttributeError):
gateway.refund_everything


def test_checkout_with_autospec():
gateway = create_autospec(PaymentGateway, instance=True)
gateway.charge.return_value = "ch_123"
assert checkout(gateway, 500) == "ch_123"
gateway.charge.assert_called_once_with(500, "usd", idempotency_key="order-1")


def test_patch_object_with_autospec_receives_self():
with patch.object(PaymentGateway, "charge", autospec=True, return_value="ch_9") as charge:
gw = PaymentGateway()
assert checkout(gw, 100) == "ch_9"
charge.assert_called_once_with(gw, 100, "usd", idempotency_key="order-1")

🔍 Interviewer Notes: A senior answer mentions that autospec does not validate return values, and pairs it with fakes, type checking, or contract tests.

❌ Common Mistakes:

  • Bare MagicMock() for every dependency
  • Believing autospec makes a mock "safe" with respect to return shapes
  • Forgetting the self argument when asserting on an autospecced method patch

Q30: Dummy, stub, spy, mock, fake — what's the difference and when do you prefer a fake?​

Technical Answer: These are Gerard Meszaros's test-double categories:

  • Dummy: passed only to satisfy a signature, never used.
  • Stub: returns canned answers. You assert on the result of the code under test.
  • Spy: records how it was called, often while delegating to the real thing (Mock(wraps=real)).
  • Mock: pre-programmed with expectations; the test asserts on interactions.
  • Fake: a working, simplified implementation (in-memory repository, fake SMTP outbox, SQLite-backed store).

Prefer fakes for your own ports (repositories, queues, clocks). They let you assert on state ("the user exists") instead of interactions ("save was called once with..."), so tests survive refactoring. Reserve interaction assertions for cases where the call is the behavior: an email is sent, a payment is charged exactly once.

Code Example:

from dataclasses import dataclass
from typing import Protocol
from unittest.mock import Mock

import pytest


@dataclass
class User:
id: int
email: str


class UserRepo(Protocol):
def get_by_email(self, email: str) -> User | None: ...
def add(self, email: str) -> User: ...


class InMemoryUserRepo:
"""Fake: a real, tiny implementation of the port."""

def __init__(self) -> None:
self._by_email: dict[str, User] = {}

def get_by_email(self, email: str) -> User | None:
return self._by_email.get(email)

def add(self, email: str) -> User:
user = User(id=len(self._by_email) + 1, email=email)
self._by_email[email] = user
return user


def register(repo: UserRepo, email: str) -> User:
email = email.strip().lower()
if repo.get_by_email(email) is not None:
raise ValueError("email already registered")
return repo.add(email)


def test_register_with_fake_asserts_state():
repo = InMemoryUserRepo()
user = register(repo, " Ada@Example.com ")
assert repo.get_by_email("ada@example.com") == user
with pytest.raises(ValueError):
register(repo, "ADA@example.com")


def test_spy_records_calls_but_keeps_real_behavior():
real = InMemoryUserRepo()
spy = Mock(wraps=real)
register(spy, "grace@example.com")
spy.add.assert_called_once_with("grace@example.com")
assert real.get_by_email("grace@example.com") is not None

🔍 Interviewer Notes: Look for "state over interaction" reasoning and awareness that fakes need their own contract tests so they don't drift from the real implementation.

❌ Common Mistakes:

  • Calling every double a "mock"
  • Asserting on every internal call, which couples tests to implementation
  • Fakes that silently diverge from production behavior (for example, no uniqueness checks)

Q31: How do you mock outbound HTTP calls (requests / httpx) correctly?​

Technical Answer: Mock at the transport layer, not by patching your own client methods. Then the real request building, URL formatting, headers, JSON decoding and raise_for_status() logic all run.

  • httpx → respx (routes are matched against real httpx.Request objects; supports sync and async clients).
  • requests → responses (@responses.activate; by default it raises ConnectionError for unregistered URLs, so nothing leaks to the network).

Test the unhappy paths too: 4xx/5xx, timeouts, malformed JSON. Assert that the route was called. For contract drift, add a small set of integration or contract tests against the provider's sandbox.

Code Example:

import httpx
import pytest
import respx


def get_exchange_rate(client: httpx.Client, base: str, quote: str) -> float:
resp = client.get(f"https://rates.example.com/v1/{base}/{quote}", timeout=5)
resp.raise_for_status()
return float(resp.json()["rate"])


@respx.mock
def test_get_exchange_rate_success():
route = respx.get("https://rates.example.com/v1/USD/EUR").mock(
return_value=httpx.Response(200, json={"rate": "0.92"})
)
with httpx.Client() as client:
assert get_exchange_rate(client, "USD", "EUR") == 0.92
assert route.call_count == 1


@respx.mock
def test_get_exchange_rate_raises_on_server_error():
respx.get("https://rates.example.com/v1/USD/EUR").mock(return_value=httpx.Response(503))
with httpx.Client() as client, pytest.raises(httpx.HTTPStatusError):
get_exchange_rate(client, "USD", "EUR")


@respx.mock
def test_get_exchange_rate_timeout():
respx.get("https://rates.example.com/v1/USD/EUR").mock(side_effect=httpx.ConnectTimeout)
with httpx.Client() as client, pytest.raises(httpx.ConnectTimeout):
get_exchange_rate(client, "USD", "EUR")
import requests
import responses


@responses.activate
def test_requests_based_client():
responses.get("https://api.example.com/health", json={"ok": True}, status=200)
resp = requests.get("https://api.example.com/health", timeout=5)
assert resp.json() == {"ok": True}
assert len(responses.calls) == 1

🔍 Interviewer Notes: Strong answers test failure modes (timeouts, 5xx) and explain why transport-level mocking beats patching client.get.

❌ Common Mistakes:

  • patch("requests.get") returning a MagicMock whose .json() returns anything
  • Only testing the 200 path
  • Tests silently hitting the real internet because a URL wasn't registered

Q32: How do you test async code properly — pytest-asyncio modes, loop scope, AsyncMock, and anyio?​

Technical Answer:

  • pytest-asyncio runs async def tests. In strict mode (the default) tests need @pytest.mark.asyncio and async fixtures need @pytest_asyncio.fixture. auto mode treats every async def test and fixture as asyncio.
  • Loop scope: by default each test gets a fresh event loop. Async resources created in a session-scoped fixture are bound to a different loop than the test, which causes "attached to a different loop" errors. Current pytest-asyncio controls this with the loop_scope argument on the marker and on fixtures, and the asyncio_default_fixture_loop_scope setting. The older pattern of overriding the event_loop fixture is no longer supported in current releases.
  • AsyncMock returns an awaitable when called. Assert with assert_awaited_once_with, await_count, and so on.
  • anyio has its own pytest plugin (@pytest.mark.anyio plus an anyio_backend fixture), which is useful for libraries that must run on both asyncio and trio.
  • Always bound waits with asyncio.timeout() (Python 3.11+) so a missing message fails fast instead of hanging CI.

Code Example:

# pyproject.toml
[tool.pytest.ini_options]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
import asyncio
from unittest.mock import AsyncMock

import pytest
import pytest_asyncio


async def fetch_all(client, ids: list[int]) -> list[dict]:
return await asyncio.gather(*(client.get_user(i) for i in ids))


@pytest.mark.asyncio
async def test_fetch_all_awaits_each_user():
client = AsyncMock()
client.get_user.side_effect = lambda i: {"id": i} # result becomes the awaited value
result = await fetch_all(client, [1, 2, 3])
assert result == [{"id": 1}, {"id": 2}, {"id": 3}]
assert client.get_user.await_count == 3


@pytest_asyncio.fixture
async def job_queue():
queue: asyncio.Queue[int] = asyncio.Queue()
yield queue
assert queue.empty(), "test left unprocessed jobs behind"


@pytest.mark.asyncio
async def test_worker_consumes_job(job_queue):
await job_queue.put(42)
async with asyncio.timeout(1): # fail fast instead of hanging
assert await job_queue.get() == 42


# anyio-based test (requires the anyio package; its pytest plugin is auto-registered)
@pytest.fixture
def anyio_backend():
return "asyncio"


@pytest.mark.anyio
async def test_with_anyio():
await asyncio.sleep(0)

🔍 Interviewer Notes: Event-loop scope mismatches and unbounded awaits are the two most common real failures. Candidates who mention both have debugged async suites.

❌ Common Mistakes:

  • Calling an async function without awaiting it in a sync test; the coroutine never runs and the test passes
  • Using Mock instead of AsyncMock for awaited dependencies (TypeError: object Mock can't be used in 'await' expression)
  • Sharing a session-scoped async client across per-test event loops

Testing APIs, Databases and Time​

Q33: How do you test a FastAPI app with httpx and dependency overrides?​

Technical Answer:

  • For async tests, use httpx.AsyncClient with httpx.ASGITransport(app=app). Requests go straight into the ASGI app in-process, with no server and no network. (fastapi.testclient.TestClient is the sync equivalent.)
  • Replace infrastructure with app.dependency_overrides[real_dep] = fake_dep. This swaps the dependency at the framework's injection point, so you don't need patch at all. Always clear overrides in teardown, or they leak into other tests.
  • ASGITransport does not run lifespan (startup/shutdown) events. If the app depends on them, use TestClient as a context manager or a lifespan helper such as asgi-lifespan.

Code Example:

from typing import Annotated

import httpx
import pytest
import pytest_asyncio
from fastapi import Depends, FastAPI, HTTPException

app = FastAPI()


class UserStore:
def __init__(self) -> None:
self.users: dict[int, dict] = {}

async def get(self, user_id: int) -> dict | None:
return self.users.get(user_id)


def get_store() -> UserStore:
raise RuntimeError("production store is wired in main.py")


@app.get("/users/{user_id}")
async def read_user(user_id: int, store: Annotated[UserStore, Depends(get_store)]) -> dict:
user = await store.get(user_id)
if user is None:
raise HTTPException(status_code=404, detail="user not found")
return user


@pytest.fixture
def store() -> UserStore:
s = UserStore()
s.users[1] = {"id": 1, "name": "Ada"}
return s


@pytest_asyncio.fixture
async def client(store):
app.dependency_overrides[get_store] = lambda: store
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as ac:
yield ac
app.dependency_overrides.clear()


@pytest.mark.asyncio
async def test_read_user(client):
resp = await client.get("/users/1")
assert resp.status_code == 200
assert resp.json() == {"id": 1, "name": "Ada"}


@pytest.mark.asyncio
async def test_read_user_not_found(client):
resp = await client.get("/users/999")
assert resp.status_code == 404
assert resp.json() == {"detail": "user not found"}


@pytest.mark.asyncio
async def test_path_validation(client):
resp = await client.get("/users/not-a-number")
assert resp.status_code == 422

🔍 Interviewer Notes: dependency_overrides plus ASGITransport is the idiomatic answer. Mentioning the lifespan caveat shows real experience.

❌ Common Mistakes:

  • Starting uvicorn in a subprocess just to run API tests
  • Patching module globals instead of overriding dependencies
  • Forgetting to clear dependency_overrides, which causes order-dependent failures

Q34: How do you isolate database tests with a transaction rollback per test?​

Technical Answer: Creating the schema is expensive; rolling back a transaction is cheap. The standard pattern:

  1. Session scope: create the engine and schema once (or run migrations once).
  2. Function scope: open a connection, begin an outer transaction, bind a Session to that connection, run the test, then roll back the outer transaction.
  3. With SQLAlchemy 2.x, pass join_transaction_mode="create_savepoint". When code under test calls session.commit(), it only releases a SAVEPOINT, so the outer rollback still discards everything.

Django's TestCase does the same thing automatically. Use TransactionTestCase (or @pytest.mark.django_db(transaction=True)) only when you need real commits, for example to test on_commit hooks across connections or code that uses multiple threads or processes. Those threads open their own connections and can't see uncommitted test data.

Code Example:

import os

import pytest
from sqlalchemy import String, create_engine, func, select
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column

DB_URL = os.environ.get("TEST_DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")


class Base(DeclarativeBase):
pass


class Customer(Base):
__tablename__ = "customers"
id: Mapped[int] = mapped_column(primary_key=True)
email: Mapped[str] = mapped_column(String(255), unique=True)


def create_customer(session: Session, email: str) -> Customer:
customer = Customer(email=email)
session.add(customer)
session.commit() # real code commits; the test must still be isolated
return customer


@pytest.fixture(scope="session")
def engine():
engine = create_engine(DB_URL)
Base.metadata.create_all(engine)
yield engine
Base.metadata.drop_all(engine)
engine.dispose()


@pytest.fixture
def session(engine):
with engine.connect() as connection:
outer = connection.begin()
session = Session(bind=connection, join_transaction_mode="create_savepoint")
try:
yield session
finally:
session.close()
outer.rollback()


def test_create_customer_commits(session):
create_customer(session, "ada@example.com")
assert session.scalar(select(func.count()).select_from(Customer)) == 1


def test_table_is_empty_again(session):
# Proves the previous test's "commit" was rolled back
assert session.scalar(select(func.count()).select_from(Customer)) == 0

🔍 Interviewer Notes: A candidate should explain why commit() inside the code doesn't break isolation (savepoints), and when rollback isolation is not enough (multiple connections, on_commit).

❌ Common Mistakes:

  • Running create_all/drop_all per test, which makes the suite slow
  • Truncating tables manually with ordering bugs between foreign keys
  • Expecting a background thread to see data from the test's uncommitted transaction

Q35: How do you build test data with factory_boy?​

Technical Answer: Factories generate valid objects with sensible defaults, so each test only spells out the fields it cares about. Key features:

  • factory.Sequence for unique values; factory.Faker for realistic data
  • factory.SubFactory for related objects; factory.LazyAttribute for fields derived from other fields
  • Traits (class Params) for named variations such as admin=True
  • build() (in memory, no DB) vs create() (persisted, for ORM factories such as factory.django.DjangoModelFactory or SQLAlchemyModelFactory); build_batch/create_batch for lists

Prefer build() wherever the test doesn't need the database; it's much faster. Keep factory defaults minimal and valid. Hidden defaults that tests rely on make tests hard to read.

Code Example:

from dataclasses import dataclass

import factory


@dataclass
class Organization:
name: str
plan: str = "free"


@dataclass
class User:
email: str
name: str
org: Organization
role: str = "member"
is_active: bool = True


class OrganizationFactory(factory.Factory):
class Meta:
model = Organization

name = factory.Faker("company")


class UserFactory(factory.Factory):
class Meta:
model = User

name = factory.Faker("name")
email = factory.Sequence(lambda n: f"user{n}@example.com")
org = factory.SubFactory(OrganizationFactory)

class Params:
admin = factory.Trait(role="admin")
deactivated = factory.Trait(is_active=False)


def test_trait_sets_role():
assert UserFactory.build(admin=True).role == "admin"


def test_override_nested_field():
user = UserFactory.build(org__plan="enterprise")
assert user.org.plan == "enterprise"


def test_batch_has_unique_emails():
users = UserFactory.build_batch(5)
assert len({u.email for u in users}) == 5

🔍 Interviewer Notes: Good answers stress explicitness in tests ("only state what matters") and build() over create() for speed.

❌ Common Mistakes:

  • Giant shared JSON fixtures that every test depends on
  • Factories that create deep object graphs (and DB rows) the test never uses
  • Relying on Faker randomness for values the assertion depends on

Q36: freezegun vs time-machine vs clock injection — how do you test time-dependent code?​

Technical Answer:

  • Clock injection (pass a Callable[[], datetime] or a clock object) is the most explicit option. It needs no global patching, works with threads and async, and makes the time dependency visible in the API. Prefer it for domain code you own.
  • freezegun patches datetime, date, time and friends by replacing module attributes. It's widely used but slower, because it scans loaded modules to replace references, and it can miss C extensions.
  • time-machine patches the clock at the C level, so it's fast and catches datetime.now() everywhere. It provides a time_machine pytest fixture with move_to() and shift().

Always use timezone-aware datetimes (datetime.now(UTC)), and test boundaries exactly (the expiry instant, one second before, one second after).

Code Example:

from collections.abc import Callable
from datetime import UTC, datetime, timedelta

from freezegun import freeze_time


class TokenService:
def __init__(self, clock: Callable[[], datetime] = lambda: datetime.now(UTC)) -> None:
self._clock = clock

def issue(self, ttl: timedelta = timedelta(minutes=15)) -> datetime:
return self._clock() + ttl # returns expiry instant

def is_valid(self, expires_at: datetime) -> bool:
return self._clock() < expires_at


def test_expiry_with_injected_clock():
current = datetime(2026, 1, 1, 12, 0, tzinfo=UTC)
svc = TokenService(clock=lambda: current)
expires_at = svc.issue()

current += timedelta(minutes=15) - timedelta(seconds=1)
assert svc.is_valid(expires_at)
current += timedelta(seconds=1)
assert not svc.is_valid(expires_at) # exactly at expiry -> invalid


def test_expiry_with_time_machine(time_machine):
time_machine.move_to(datetime(2026, 1, 1, 12, 0, tzinfo=UTC), tick=False)
svc = TokenService()
expires_at = svc.issue()
time_machine.shift(timedelta(minutes=15))
assert not svc.is_valid(expires_at)


def test_expiry_with_freezegun():
with freeze_time("2026-01-01 12:00:00") as frozen:
svc = TokenService()
expires_at = svc.issue()
frozen.tick(timedelta(minutes=14))
assert svc.is_valid(expires_at)

🔍 Interviewer Notes: The best answers prefer injection for owned code and use a time library at the edges, and they test exact boundaries.

❌ Common Mistakes:

  • datetime.utcnow() (naive, deprecated since Python 3.12) mixed with aware datetimes
  • time.sleep() in tests to "wait for expiry"
  • Asserting created_at == datetime.now() with no frozen time

Q37: How do you test code that depends on environment variables and cached settings?​

Technical Answer:

  • Use monkeypatch.setenv / monkeypatch.delenv(..., raising=False). Changes are undone automatically after the test, so they never leak.
  • Settings objects are often cached (@lru_cache on get_settings(), a pydantic-settings instance created at import time). Changing the environment then has no effect until the cache is cleared. Call get_settings.cache_clear() before and after, or better, inject settings into the code under test.
  • Values read at import time into module constants need monkeypatch.setattr(module, "CONSTANT", value), because the env var was already consumed.
  • Never let tests read the developer's real .env file or production credentials. Set every required variable explicitly in a fixture.

Code Example:

import os
from dataclasses import dataclass
from functools import lru_cache

import pytest


@dataclass(frozen=True)
class Settings:
database_url: str
new_checkout_enabled: bool


@lru_cache
def get_settings() -> Settings:
return Settings(
database_url=os.environ["DATABASE_URL"],
new_checkout_enabled=os.environ.get("FEATURE_NEW_CHECKOUT", "false").lower() == "true",
)


def checkout_flow() -> str:
return "v2" if get_settings().new_checkout_enabled else "v1"


@pytest.fixture
def env(monkeypatch):
monkeypatch.setenv("DATABASE_URL", "postgresql://localhost/test")
monkeypatch.delenv("FEATURE_NEW_CHECKOUT", raising=False)
get_settings.cache_clear()
yield monkeypatch
get_settings.cache_clear()


def test_flag_defaults_off(env):
assert checkout_flow() == "v1"


def test_flag_on(env):
env.setenv("FEATURE_NEW_CHECKOUT", "true")
assert checkout_flow() == "v2"


def test_missing_required_setting(env):
env.delenv("DATABASE_URL")
with pytest.raises(KeyError):
get_settings()

🔍 Interviewer Notes: The cache-invalidation trap is what separates candidates who have actually tested configuration from those who have only read about monkeypatch.

❌ Common Mistakes:

  • os.environ["X"] = "..." in a test with no cleanup
  • Tests that pass only because the developer's shell has the variable set
  • Forgetting cached settings, so the first test's configuration wins for the whole session

Test Suite Quality and Scale​

Q38: What is mutation testing and what does it tell you that coverage doesn't?​

Technical Answer: Coverage tells you which code ran. Mutation testing tells you whether your tests would notice if that code were wrong. A tool such as mutmut (or cosmic-ray) makes small changes to the source, called mutants: >= → >, + → -, True → False, deleted statements, changed constants. It then runs the tests against each mutant:

  • Killed: at least one test failed. Good.
  • Survived: every test still passed. Either an assertion is missing, or the mutant is equivalent to the original (semantically identical code).

The mutation score (killed / total) measures test strength. It is CPU-expensive (it runs the suite once per mutant), so run it on critical modules, on changed files in a scheduled job, or before refactoring risky logic. Don't run it on every commit for the whole codebase.

Code Example:

# pricing.py
def shipping_fee(subtotal_cents: int) -> int:
if subtotal_cents >= 5_000:
return 0
return 499
# test_pricing_weak.py - 100% line and branch coverage, yet weak
from pricing import shipping_fee


def test_shipping_fee():
assert shipping_fee(10_000) == 0
assert shipping_fee(100) == 499
# The mutant `subtotal_cents > 5_000` SURVIVES: no test sits on the boundary.
# test_pricing_strong.py - kills the boundary mutants
import pytest

from pricing import shipping_fee


@pytest.mark.parametrize(
("subtotal", "fee"),
[(4_999, 499), (5_000, 0), (5_001, 0), (0, 499)],
)
def test_shipping_fee_boundaries(subtotal, fee):
assert shipping_fee(subtotal) == fee
pip install mutmut
mutmut run # generate and test mutants
mutmut results # list surviving mutants to inspect

🔍 Interviewer Notes: A senior candidate frames mutation testing as a targeted audit tool, not a CI gate on every commit, and knows about equivalent mutants.

❌ Common Mistakes:

  • Treating coverage percentage as a measure of test quality
  • Running mutation testing over the whole monolith on every PR
  • Chasing a 100% mutation score, including equivalent mutants

Q39: What is contract testing and how does it differ from integration testing?​

Technical Answer: An integration test runs a consumer and a provider together, which is slow and needs both deployed. A contract test checks each side independently against a shared agreement about request and response shapes:

  • Consumer-driven contracts (Pact): the consumer's tests record the interactions they rely on into a pact file. The provider's CI replays them against the real provider. A broker plus a can-i-deploy check blocks releases that would break a consumer.
  • Schema-based contracts: the provider publishes an OpenAPI or JSON Schema. Consumers validate their fixtures against it, and the provider is fuzzed against its own spec (for example, Schemathesis generates requests from the OpenAPI document).
  • Lightweight in-repo contracts: the consumer defines a strict model of the fields it actually uses, and the provider's test suite validates real responses against it.

Contract tests catch "renamed a field, broke the mobile app" without a full end-to-end environment.

Code Example:

# contracts/order_contract.py - owned by the consumer team, vendored into the provider repo
from typing import Literal

from pydantic import BaseModel, ConfigDict


class OrderContract(BaseModel):
model_config = ConfigDict(extra="ignore") # provider may add fields freely

id: int
status: Literal["pending", "paid", "shipped"]
total_cents: int
# provider/tests/test_order_contract.py
from fastapi import FastAPI
from fastapi.testclient import TestClient

from contracts.order_contract import OrderContract

app = FastAPI()


@app.get("/orders/{order_id}")
def get_order(order_id: int) -> dict:
return {"id": order_id, "status": "paid", "total_cents": 1999, "currency": "usd"}


def test_get_order_satisfies_consumer_contract():
with TestClient(app) as client:
resp = client.get("/orders/7")
assert resp.status_code == 200
OrderContract.model_validate(resp.json()) # raises ValidationError if the contract breaks
# Property-based API fuzzing from the provider's own OpenAPI spec
schemathesis run http://localhost:8000/openapi.json

🔍 Interviewer Notes: Look for the insight that contracts let teams deploy independently, plus awareness of extra="ignore" so that additive changes don't count as breaking.

❌ Common Mistakes:

  • Using a shared staging environment as the only compatibility check
  • Consumers asserting on every field (including ones they don't use), which makes contracts brittle
  • Provider changing response shapes without running consumer contracts in CI

Q40: How do you detect and prevent test-order dependence?​

Technical Answer: Order dependence happens when a test passes only because of state left behind (or not left behind) by another test: module-level caches, singletons, class attributes, environment variables, rows in the database, files on disk, or mocks that were never stopped.

Detection:

  • pytest-randomly shuffles test order on every run and prints the seed. Reproduce with --randomly-seed=<seed> (or --randomly-seed=last). It also reseeds random (and Faker, if installed) for each test.
  • Run a single test on its own: pytest path::test_name. If it fails alone but passes in the suite, it depends on another test's side effects.
  • Tools such as detect-test-pollution bisect the suite to find the polluting test.

Prevention: function-scoped fixtures, monkeypatch for global state, autouse fixtures that reset registries and caches, and transaction rollback for the database.

Code Example:

# app/registry.py
_handlers: dict[str, str] = {}


def register(event: str, handler: str) -> None:
_handlers[event] = handler


def handler_for(event: str) -> str | None:
return _handlers.get(event)
# tests/test_registry.py
import pytest

import app.registry as registry


@pytest.fixture(autouse=True)
def isolated_registry(monkeypatch):
# Fresh dict per test; monkeypatch restores the original afterwards
monkeypatch.setattr(registry, "_handlers", {})


def test_register_handler():
registry.register("signup", "send_welcome")
assert registry.handler_for("signup") == "send_welcome"


def test_unknown_event_has_no_handler():
# Without the autouse fixture this fails whenever it runs after the test above
assert registry.handler_for("signup") is None
pip install pytest-randomly
pytest # random order, seed printed at the top
pytest --randomly-seed=1234 # replay a failing order
pytest -p no:randomly # temporarily disable while bisecting

🔍 Interviewer Notes: Strong candidates make random ordering the default so pollution is caught on the PR that introduces it, not months later.

❌ Common Mistakes:

  • "Fixing" pollution by pinning test order
  • Module-level mutable state in application code with no reset hook
  • Using patch.start() without stop() (use with, the decorator, or monkeypatch)

Q41: How do you parallelize a test suite with pytest-xdist safely?​

Technical Answer: pytest -n auto starts one worker process per CPU and distributes tests. Distribution modes (--dist):

  • load (default with -n): send tests to any free worker, which gives the best balance.
  • loadscope: keep tests from the same module or class on one worker, so module- and class-scoped fixtures run once.
  • loadfile: keep each file on one worker.
  • loadgroup: tests marked @pytest.mark.xdist_group(name=...) run on the same worker.

Gotchas:

  • Session-scoped fixtures run once per worker, not once globally. Shared external resources need per-worker names (use the worker_id fixture: "gw0", "gw1", or "master" without xdist) or a file lock.
  • Tests sharing a database, port, file path or Redis key collide. pytest-django already suffixes the test database name per worker.
  • Parallelism exposes hidden order dependence. Fix the pollution; don't serialize the suite.

Code Example:

# conftest.py
import json

import pytest
from filelock import FileLock


@pytest.fixture(scope="session")
def database_name(worker_id: str) -> str:
return f"test_app_{worker_id}" # test_app_gw0, test_app_gw1, ...


def build_reference_data() -> dict:
return {"currencies": ["EUR", "GBP", "USD"]}


@pytest.fixture(scope="session")
def reference_data(tmp_path_factory, worker_id):
"""Build once across ALL workers, then share via a file (pattern from the xdist docs)."""
if worker_id == "master":
return build_reference_data()
shared_dir = tmp_path_factory.getbasetemp().parent
path = shared_dir / "reference_data.json"
with FileLock(str(path) + ".lock"):
if path.is_file():
return json.loads(path.read_text())
data = build_reference_data()
path.write_text(json.dumps(data))
return data


@pytest.mark.xdist_group(name="ledger")
def test_ledger_opening_balance(reference_data):
assert "USD" in reference_data["currencies"]
pip install pytest-xdist filelock
pytest -n auto # one worker per CPU
pytest -n 8 --dist loadscope # keep module fixtures on one worker
pytest -n auto --dist loadgroup # honour xdist_group markers

🔍 Interviewer Notes: Check that the candidate knows session fixtures run per worker. That misunderstanding causes most xdist breakage.

❌ Common Mistakes:

  • Sharing one test database across workers with no per-worker isolation
  • Adding -n auto on top of an order-dependent suite and blaming xdist
  • Expecting a session fixture to run exactly once under xdist

Q42: How do you find good properties for property-based tests?​

Technical Answer: The hard part of Hypothesis is choosing the property, not the API. Reliable patterns:

  • Round-trip: decode(encode(x)) == x (serializers, parsers, compression).
  • Invariants: facts the output always satisfies (sorted output is ordered, has the same length, and is a permutation of the input).
  • Oracle / reference implementation: compare an optimized function against a simple, obviously correct one.
  • Idempotence: f(f(x)) == f(x) (normalizers, upserts, formatters).
  • Metamorphic relations: how the output changes when the input changes in a known way (adding an item never decreases the cart total).

Use @example(...) to pin specific regressions alongside the generated cases. Design strategies carefully: exclude NaN when equality matters (NaN != NaN), and bound sizes to keep runs fast.

Code Example:

import json
from collections import Counter

from hypothesis import example, given, strategies as st

json_values = st.recursive(
st.none() | st.booleans() | st.integers() | st.text(), # no floats: NaN != NaN
lambda children: st.lists(children) | st.dictionaries(st.text(), children),
max_leaves=20,
)


@given(json_values)
def test_json_round_trip(value):
assert json.loads(json.dumps(value)) == value


@given(st.lists(st.integers()))
def test_sorted_invariants(xs):
out = sorted(xs)
assert len(out) == len(xs)
assert all(a <= b for a, b in zip(out, out[1:]))
assert Counter(out) == Counter(xs)


def dedupe(items: list[int]) -> list[int]:
seen: set[int] = set()
result: list[int] = []
for item in items:
if item not in seen:
seen.add(item)
result.append(item)
return result


@given(st.lists(st.integers(min_value=-5, max_value=5)))
def test_dedupe_matches_oracle(xs):
assert dedupe(xs) == list(dict.fromkeys(xs))


def normalize_whitespace(s: str) -> str:
return " ".join(s.split())


@given(st.text())
@example(" leading and trailing ")
@example("")
def test_normalize_is_idempotent(s):
once = normalize_whitespace(s)
assert normalize_whitespace(once) == once

🔍 Interviewer Notes: Candidates who can name several property patterns can apply Hypothesis to real code, not just to add(a, b) == add(b, a).

❌ Common Mistakes:

  • Properties that re-implement the function under test (and its bugs)
  • Unbounded strategies that make the suite slow
  • Filtering heavily with assume() instead of building a precise strategy

Senior / Principal — Test Strategy and CI Scenarios​

Q43: A flaky test suite is blocking every deploy​

Level: Principal

Scenario: The main branch pipeline fails roughly one run in three. Each failure is a different test, and every one passes on re-run. Engineers press "retry" two or three times per merge, the deploy queue backs up, and a real regression slipped through last week because "it's probably just flaky."

Impact: Lost engineering hours, slower lead time, and the team no longer trusts red builds, which is the worst outcome because real failures now get ignored.

Root Cause: There is no process for flakiness. Flaky tests are never identified systematically, have no owner, and sit in the same blocking job as deterministic tests. The underlying causes (shared state, timing, network, ordering) are never investigated because a retry always "fixes" it.

Fix:

  1. Measure: run CI with pytest-rerunfailures (--reruns 2) in reporting mode. A test that fails and then passes on rerun is flaky by definition. Record these in JUnit XML and aggregate them per test over time.
  2. Quarantine with ownership: move confirmed flaky tests behind a quarantine marker that records the owner and a ticket. The blocking job deselects them; a separate, non-blocking job still runs them so they keep producing signal.
  3. Set a fix SLA: a quarantined test must be fixed or deleted within a fixed window, tracked on a dashboard. Quarantine is a holding area, not a graveyard.
  4. Fix root causes: random ordering (pytest-randomly), frozen or injected time, no real network, per-test database isolation, bounded async waits.
  5. Stop the bleeding: remove blanket retries from the blocking job once quarantine exists, so new flakiness surfaces immediately on the PR that introduces it.

Code Example:

# conftest.py
import pytest


def pytest_addoption(parser):
parser.addoption(
"--run-quarantined",
action="store_true",
default=False,
help="run ONLY tests marked @pytest.mark.quarantine",
)


def pytest_collection_modifyitems(config, items):
run_quarantined = config.getoption("--run-quarantined")
selected, deselected = [], []
for item in items:
is_quarantined = item.get_closest_marker("quarantine") is not None
(selected if is_quarantined == run_quarantined else deselected).append(item)
if deselected:
config.hook.pytest_deselected(items=deselected)
items[:] = selected
# tests/test_checkout.py
import pytest


@pytest.mark.quarantine(owner="payments-team", ticket="PAY-412")
def test_checkout_emits_receipt_event():
events = ["order_paid", "receipt_sent"]
assert "receipt_sent" in events
# pyproject.toml
[tool.pytest.ini_options]
addopts = "--strict-markers"
markers = ["quarantine(owner, ticket): known-flaky test excluded from the blocking job"]
# Blocking job: deterministic tests only, random order
pytest --junitxml=reports/blocking.xml # pytest-randomly installed: random order

# Non-blocking job: quarantined tests with reruns, to collect flake statistics
pytest --run-quarantined --reruns 2 --junitxml=reports/quarantine.xml

Workflow Diagram:

CI failure on main
|
v
Rerun data says "failed then passed"? --no--> real failure: block and fix
|
yes
v
Mark @quarantine(owner, ticket) ---> blocking job deselects it
| |
v v
Non-blocking job keeps running it main pipeline green and trusted again
|
v
Owner fixes root cause within SLA (state / time / network / order)
|
v
Remove marker -> test back in blocking job

🔍 Interviewer Notes: A principal answer treats flakiness as a process and ownership problem, with measurement, quarantine, an SLA and root-cause fixes. A list of retry flags is not enough. Ask how they'd stop quarantine from becoming permanent.

❌ Red Flags: Blanket --reruns 3 on the blocking job; deleting flaky tests without understanding them; quarantine with no owner or deadline; "just re-run it" culture.


Q44: Mocks hid a real bug — all tests green, production crashed​

Level: Senior

Scenario: The payments SDK wrapper was upgraded and PaymentClient.charge() now returns a ChargeResult dataclass instead of a dict. The order service still does result["id"]. Every unit test passes, because each one mocks charge to return a dict. In production, every checkout fails with TypeError: 'ChargeResult' object is not subscriptable.

Impact: Checkout outage until rollback, and lost revenue. The test suite reported 100% success throughout.

Root Cause: The tests encoded the developer's assumption about the dependency, not its actual behavior. Mock() accepts any call and returns whatever it is told to. Even autospec would not have helped here, because it validates call signatures, not return types. No test exercised the real wrapper or a fake that shares its types.

Fix:

  1. Replace ad-hoc mocks of your own wrappers with a fake that subclasses or implements the real interface and returns real types.
  2. Run mypy/pyright in CI. result["id"] on a ChargeResult is a type error that would have been caught before any test ran.
  3. Write a shared contract test that runs against the fake in unit CI and against the provider's sandbox in a scheduled integration job, so the fake can't drift.
  4. Use create_autospec wherever you do still mock, and assert on outcomes rather than on mock configuration.

Code Example:

# payments/client.py
from dataclasses import dataclass


@dataclass(frozen=True)
class ChargeResult:
id: str
status: str


class PaymentClient:
def charge(self, amount_cents: int, currency: str, *, idempotency_key: str) -> ChargeResult:
raise NotImplementedError("calls the provider over HTTP")


class FakePaymentClient(PaymentClient):
def __init__(self) -> None:
self.charges: list[tuple[int, str, str]] = []

def charge(self, amount_cents: int, currency: str, *, idempotency_key: str) -> ChargeResult:
self.charges.append((amount_cents, currency, idempotency_key))
return ChargeResult(id=f"ch_{len(self.charges)}", status="succeeded")
# orders/service.py
from payments.client import PaymentClient


def pay_order(client: PaymentClient, order_id: int, amount_cents: int) -> str:
result = client.charge(amount_cents, "usd", idempotency_key=f"order-{order_id}")
return result.id # the buggy version did result["id"]
# tests/test_pay_order.py
from unittest.mock import Mock

import pytest

from orders.service import pay_order
from payments.client import FakePaymentClient, PaymentClient


def pay_order_buggy(client: PaymentClient, order_id: int, amount_cents: int) -> str:
result = client.charge(amount_cents, "usd", idempotency_key=f"order-{order_id}")
return result["id"] # the pre-fix code


def test_misleading_mock_passes_the_buggy_code():
client = Mock()
client.charge.return_value = {"id": "ch_1"} # a shape the real client never returns
assert pay_order_buggy(client, 1, 100) == "ch_1" # green, yet production crashes


def test_fake_exposes_the_bug():
with pytest.raises(TypeError):
pay_order_buggy(FakePaymentClient(), 1, 100)


def assert_client_contract(client: PaymentClient) -> None:
result = client.charge(500, "usd", idempotency_key="contract-1")
assert isinstance(result.id, str) and result.id
assert result.status in {"succeeded", "pending"}


def test_fake_honours_contract():
assert_client_contract(FakePaymentClient())


def test_pay_order_with_fake():
client = FakePaymentClient()
assert pay_order(client, 42, 1999) == "ch_1"
assert client.charges == [(1999, "usd", "order-42")]

Workflow Diagram:

SDK upgrade changes return type
|
v
Unit tests: Mock returns dict ----> green (false confidence)
|
v
Production: real ChargeResult ----> TypeError, checkout down
|
v
Fix:
type checker in CI -> catches result["id"] statically
fake returns real types -> unit tests exercise real shapes
shared contract test -> fake vs sandbox, run on schedule

🔍 Interviewer Notes: Listen for "autospec checks signatures, not return values" and for type checking being treated as part of the test strategy. Ask: "How do you keep the fake honest?"

❌ Red Flags: "We need more mocks"; mocking your own code at every layer; no type checking; no test that touches the real dependency.


Q45: The test suite takes 40 minutes​

Level: Senior

Scenario: A Django monolith's CI test job has grown from 8 to 40 minutes. Developers stop running tests locally, batch changes into huge PRs, and wait most of the day for feedback.

Impact: Slow lead time, large risky merges, and CI compute costs growing linearly with headcount.

Root Cause: Typically a mix of: expensive work in function-scoped fixtures (schema creation, loading large fixture files), real time.sleep in retry logic under test, slow password hashing in user factories, tests that hit real networks or containers per test, oversized E2E coverage, and everything running serially on one machine.

Fix:

  1. Measure first: pytest --durations=30 --durations-min=1.0 shows the slowest setups, calls and teardowns. Fix the top 20; they usually account for most of the time.
  2. Fixture scope: create schema and reference data once per session and roll back per test (see Q34). Use build() instead of create() in factories where no database is needed.
  3. Remove waiting: inject sleep and clock dependencies so retry and backoff code runs instantly in tests.
  4. Cheap crypto in tests: in Django, set PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"] in test settings only.
  5. Parallelize: pytest -n auto on each machine, then shard across CI machines. Fix any order dependence this exposes.
  6. Run less per change: fast unit tests on every push, full integration tests on merge, E2E tests post-deploy. Use --lf/--ff locally.

Code Example:

# retry.py - sleep is injectable, so tests don't wait
import time
from collections.abc import Callable
from typing import TypeVar

T = TypeVar("T")


def retry(
fn: Callable[[], T],
attempts: int = 5,
base_delay: float = 0.5,
sleep: Callable[[float], None] = time.sleep,
) -> T:
for attempt in range(attempts):
try:
return fn()
except ConnectionError:
if attempt == attempts - 1:
raise
sleep(base_delay * 2**attempt)
raise AssertionError("unreachable")
# test_retry.py - runs in milliseconds instead of ~7.5 seconds of backoff
import pytest

from retry import retry


def test_retry_gives_up_after_attempts():
delays: list[float] = []

def always_fails():
raise ConnectionError("down")

with pytest.raises(ConnectionError):
retry(always_fails, attempts=5, sleep=delays.append)
assert delays == [0.5, 1.0, 2.0, 4.0]
# settings/test.py (Django)
from .base import * # noqa: F403

PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
pytest --durations=30 --durations-min=1.0 # find the real hotspots
pytest -n auto --dist loadscope # parallelize on one machine
pytest --lf # locally: re-run last failures first

Workflow Diagram:

40 min serial suite
|
v
--durations: top offenders = fixtures, sleeps, hashing, network
|
v
Session schema + per-test rollback (setup cost paid once)
Injected sleep/clock (no real waiting)
Fast test-only password hasher (cheap user creation)
|
v
pytest -n auto + CI sharding (wall-clock divided by workers)
|
v
Tiered pipeline: unit on push / integration on merge / e2e post-deploy

🔍 Interviewer Notes: Good candidates profile before optimizing and treat parallelism as the last multiplier, not the first fix. Ask what they'd do if -n auto made tests fail. The answer should be: fix isolation.

❌ Red Flags: Deleting slow tests without understanding what they protect; only adding more CI machines; skipping tests locally because "CI will catch it."


Q46: Tests pass locally but fail in CI​

Level: Senior

Scenario: A set of tests is green on every developer's MacBook and consistently red in the Linux CI container: a report test picks the "wrong" latest file, a date test fails only on the evening CI runs, and an import fails with ModuleNotFoundError for a file that clearly exists.

Impact: Blocked merges, "works on my machine" arguments, and people disabling tests to get green.

Root Cause: Environment differences the tests silently depend on:

  • Filesystem: os.listdir/Path.glob order is arbitrary and differs between filesystems. macOS is case-insensitive by default and Linux is case-sensitive (import Utils vs utils.py).
  • Timezone and locale: CI runs in UTC. Code using date.today() (local time) disagrees with UTC-based expectations near midnight.
  • Environment variables or files that exist on the laptop (.env, credentials, cached data) but not in CI.
  • Dependency drift: unpinned versions resolve differently. The Python version can differ too.
  • Parallelism and speed: CI runs -n auto on slower shared runners, which exposes ordering and timing assumptions.

Fix:

  1. Reproduce CI exactly: run the same container image locally, with the same env vars and the same command (including -n and the random seed).
  2. Make ordering explicit (sorted(...), max(..., key=...)) and keep filename case consistent.
  3. Use timezone-aware UTC time in code and tests; freeze or inject the clock.
  4. Pin dependencies with a lock file and test against the deployed Python version.
  5. Set every required env var explicitly in fixtures; never rely on a developer's .env.

Code Example:

# reports.py
from datetime import UTC, date, datetime
from pathlib import Path


def latest_report_buggy(directory: Path) -> Path:
return list(directory.glob("report-*.csv"))[-1] # order depends on the filesystem


def latest_report(directory: Path) -> Path:
return max(directory.glob("report-*.csv"), key=lambda p: p.name)


def report_date_buggy() -> date:
return date.today() # machine-local timezone


def report_date() -> date:
return datetime.now(UTC).date()
# test_reports.py
from reports import latest_report


def test_latest_report_is_deterministic(tmp_path):
for name in ["report-2026-03-02.csv", "report-2026-03-10.csv", "report-2026-03-01.csv"]:
(tmp_path / name).write_text("id\n")
assert latest_report(tmp_path).name == "report-2026-03-10.csv"
# Reproduce the CI environment locally
docker run --rm -it -e TZ=UTC -v "$PWD":/app -w /app python:3.12-slim \
bash -c "pip install -r requirements-lock.txt && pytest -n auto"

Workflow Diagram:

Green locally, red in CI
|
v
Run CI image + env + command locally ---> reproduces?
| |
yes no -> compare runner resources/services
v
Classify: ordering | timezone | case-sensitivity | env vars | versions | parallelism
|
v
Make the dependency explicit (sort, UTC, pinned deps, fixtures set env)
|
v
Same result on laptop and CI

🔍 Interviewer Notes: Strong candidates go for reproducing the CI environment first, then classify causes systematically. Bonus for knowing about case-insensitive macOS filesystems and arbitrary directory listing order.

❌ Red Flags: Marking tests skipif(CI); adding retries; "CI is just weird."


Q47: Tests use SQLite but production runs PostgreSQL​

Level: Senior

Scenario: To keep tests fast, the team runs the suite against in-memory SQLite. Production uses PostgreSQL. Two incidents in one quarter: a varchar(100) overflow that SQLite silently accepted, and a with_for_update() row-locking path that was silently ignored on SQLite and deadlocked in production.

Impact: Bugs that the test suite was structurally unable to catch, and lost trust in the suite.

Root Cause: SQLite differs from PostgreSQL in ways that matter: it doesn't enforce VARCHAR length, has different type affinity, has different LIKE case-sensitivity, a different JSON operator set, no row-level locking (SQLite has no SELECT ... FOR UPDATE; SQLAlchemy omits the clause and Django's select_for_update() is a no-op there), and different transaction and constraint behavior. Testing against a different database means testing a different program.

Fix:

  1. Run integration tests against the same database engine and major version as production. Testcontainers starts a throwaway PostgreSQL per session.
  2. Keep it fast: one container per session (or per xdist worker), migrations once, transaction rollback per test (Q34).
  3. Keep pure-logic tests database-free (fakes) so they stay instant. Only the persistence layer needs the real engine.
  4. Add tests for the engine-specific behavior you rely on (constraints, locking, JSON queries).

Code Example:

# conftest.py
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from testcontainers.postgres import PostgresContainer

from app.models import Base


@pytest.fixture(scope="session")
def engine():
with PostgresContainer("postgres:16-alpine", driver="psycopg") as pg:
engine = create_engine(pg.get_connection_url())
Base.metadata.create_all(engine)
yield engine
engine.dispose()


@pytest.fixture
def session(engine):
with engine.connect() as connection:
outer = connection.begin()
session = Session(bind=connection, join_transaction_mode="create_savepoint")
try:
yield session
finally:
session.close()
outer.rollback()
# app/models.py
from sqlalchemy import String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
pass


class Customer(Base):
__tablename__ = "customers"
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String(100))
# tests/test_customer_constraints.py
import pytest
from sqlalchemy.exc import DataError

from app.models import Customer


def test_name_length_is_enforced(session):
session.add(Customer(name="x" * 101))
with pytest.raises(DataError): # PostgreSQL enforces varchar(100); SQLite would accept it
session.flush()

Workflow Diagram:

Before: tests -> SQLite (fast, different semantics) -> prod Postgres bugs
After:
pytest session start
|
v
Testcontainers: postgres:16 (same major as prod)
|
v
create schema / migrate once
|
v
each test: BEGIN -> run -> ROLLBACK (fast + isolated)
|
v
engine-specific behavior actually tested

🔍 Interviewer Notes: Look for "test against what you run" and for knowing how to keep it fast (session container plus rollback). Ask which SQLite/Postgres differences they've personally hit.

❌ Red Flags: "SQLite is close enough"; mocking the ORM to avoid a database; spinning up a new container per test.


Q48: The test suite is calling a real third-party API​

Level: Senior

Scenario: CI starts failing with HTTP 429 from the payment provider's sandbox. Investigation shows that dozens of "unit" tests make real HTTP calls: some to the sandbox, and one, through a misconfigured environment variable, to the live API with a real key. The provider has now rate-limited the company's account.

Impact: Flaky and slow tests, rate limiting that also affects staging, potential real charges or data leaks, and secrets exposed to every CI job.

Root Cause: Nothing stops tests from opening network connections. HTTP clients are module-level singletons configured from the environment, and CI injects production-capable credentials into every job "for convenience."

Fix:

  1. Block the network by default in unit tests with pytest-socket (--disable-socket). Any attempted connection raises SocketBlockedError, so leaks fail loudly on the PR that introduces them.
  2. Mock outbound HTTP at the transport layer (respx/responses) or inject fake clients.
  3. Allow network access only in an explicitly marked integration job (@pytest.mark.enable_socket, or --allow-hosts restricted to local services).
  4. Separate credentials: the unit job gets no provider secrets at all. The integration job gets sandbox-only keys. Live keys never reach CI test jobs.

Code Example:

# pyproject.toml - unit test defaults
[tool.pytest.ini_options]
addopts = "--strict-markers --disable-socket --allow-unix-socket"
markers = ["integration: talks to real services"]
# tests/test_rates_client.py
import socket

import httpx
import pytest
import respx
from pytest_socket import SocketBlockedError


def fetch_rate(client: httpx.Client) -> float:
return float(client.get("https://rates.example.com/v1/USD/EUR", timeout=5).json()["rate"])


def test_inet_sockets_are_blocked():
with pytest.raises(SocketBlockedError):
socket.socket(socket.AF_INET, socket.SOCK_STREAM)


@respx.mock
def test_fetch_rate_with_mocked_transport():
respx.get("https://rates.example.com/v1/USD/EUR").mock(
return_value=httpx.Response(200, json={"rate": "0.92"})
)
with httpx.Client() as client:
assert fetch_rate(client) == 0.92


@pytest.mark.integration
@pytest.mark.enable_socket
def test_sandbox_health():
# Runs only in the integration job, which has sandbox credentials
with httpx.Client() as client:
assert client.get("https://sandbox.rates.example.com/health", timeout=5).status_code == 200
# Unit job: no secrets, no network
pytest -m "not integration"

# Integration job: sandbox secrets only
pytest -m integration

Workflow Diagram:

Test opens a socket
|
v
pytest-socket active? --yes--> SocketBlockedError -> test fails on the PR
| |
| v
| developer mocks transport / injects fake
|
(integration job only: enable_socket + sandbox keys)
|
v
Live credentials never present in any test job

🔍 Interviewer Notes: The best answers combine a technical guard (network blocked by default) with a secrets guard (no live keys in CI). Ask how they'd find every test currently making network calls; turning on --disable-socket and reading the failures is the fastest way.

❌ Red Flags: "Use the sandbox, it's free"; production credentials in CI environment variables; mocking only the happy path while real calls still happen in edge cases.


Q49: An async test hangs and the CI job runs until its hard limit​

Level: Senior

Scenario: A PR adds a test for a queue consumer. In CI the job produces no output for hours until the runner's job timeout kills it. There is no traceback, no failing test name, and the PR is blocked with nothing to debug.

Impact: Wasted CI minutes, blocked merges, and no diagnostic information.

Root Cause: The test awaits something that never completes (await queue.get() with no producer, an Event never set, an unclosed client keeping the loop alive), and nothing bounds the wait. pytest has no per-test timeout by default, so a deadlock looks like silence.

Fix:

  1. Bound every wait in the test with asyncio.timeout() (Python 3.11+) so it fails with a clear TimeoutError.
  2. Add a global per-test timeout with pytest-timeout as a safety net. It fails the hanging test and dumps stack traces.
  3. Enable pytest's built-in faulthandler_timeout to dump all thread stacks if a test exceeds a threshold, so you can see where it is stuck.
  4. Set a CI job timeout well below the runner's default, so a hang costs minutes, not hours.

Code Example:

# pyproject.toml
[tool.pytest.ini_options]
timeout = 60 # pytest-timeout: fail any single test after 60s
faulthandler_timeout = 30 # built-in: dump tracebacks of all threads after 30s
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"
# test_consumer.py
import asyncio

import pytest


async def consume_one(queue: asyncio.Queue[str]) -> str:
return await queue.get()


@pytest.mark.asyncio
async def test_consume_one_with_bounded_wait():
queue: asyncio.Queue[str] = asyncio.Queue()

async def producer() -> None:
await asyncio.sleep(0.01)
await queue.put("order-1")

async with asyncio.TaskGroup() as tg:
tg.create_task(producer())
async with asyncio.timeout(1):
assert await consume_one(queue) == "order-1"


@pytest.mark.asyncio
async def test_missing_message_fails_fast_instead_of_hanging():
queue: asyncio.Queue[str] = asyncio.Queue()
with pytest.raises(TimeoutError):
async with asyncio.timeout(0.1):
await consume_one(queue)


@pytest.mark.timeout(5) # tighter per-test override
@pytest.mark.asyncio
async def test_heartbeat_is_quick():
await asyncio.sleep(0)

Workflow Diagram:

Test awaits a future that never resolves
|
+-- asyncio.timeout(1) in test ----> TimeoutError, clear failure (best)
|
+-- faulthandler_timeout=30 -------> stack dump shows where it's stuck
|
+-- pytest-timeout=60 -------------> test failed by name, suite continues
|
+-- CI job timeout (e.g. 20 min) ---> last-resort cap on cost

🔍 Interviewer Notes: Candidates should layer timeouts (in-test, per-test, per-job) and value diagnosability: a named failure with a stack trace instead of silence.

❌ Red Flags: Relying only on the CI runner timeout; await asyncio.sleep(5) to "let things settle"; no idea how to find where an async test is stuck.


Q50: A small refactor breaks 300 tests​

Level: Principal

Scenario: An engineer merges two private pricing helpers into one method. Behavior is unchanged and production output is identical, but 300 tests fail. Most of them mock the internal collaborators and assert exactly which methods were called with which arguments. The team starts avoiding refactoring because "the tests make it too expensive."

Impact: The test suite has become an obstacle to improving the code, the opposite of its purpose. Technical debt grows because cleanup is too costly.

Root Cause: The tests are coupled to how the code works (interaction assertions on internal collaborators) instead of what it does (observable outputs and state). Mocking every class "to isolate the unit" turns each internal call graph into a frozen contract.

Fix:

  1. Test through the public API of a module or service. Treat internal classes as implementation details.
  2. Use real collaborators for in-process logic (sociable unit tests) and fakes only at real boundaries (database, network, clock, queue).
  3. Assert on outputs and state. Keep interaction assertions for side effects that are the behavior (a charge was made once, an email was sent).
  4. Migrate gradually: when a brittle test breaks during a refactor, rewrite it in the behavioral style instead of "fixing the mocks."

Code Example:

# pricing.py
from dataclasses import dataclass


@dataclass(frozen=True)
class Line:
sku: str
unit_cents: int
qty: int


class Pricing:
TAX_RATE_BPS = 800 # 8.00%

def quote(self, lines: list[Line]) -> tuple[int, int]:
subtotal = sum(line.unit_cents * line.qty for line in lines)
tax = subtotal * self.TAX_RATE_BPS // 10_000
return subtotal, tax


class InMemoryOrderRepo:
def __init__(self) -> None:
self.saved: list[dict] = []

def save(self, order: dict) -> None:
self.saved.append(order)


class OrderService:
def __init__(self, repo: InMemoryOrderRepo, pricing: Pricing) -> None:
self.repo = repo
self.pricing = pricing

def place(self, customer_id: int, lines: list[Line]) -> dict:
subtotal, tax = self.pricing.quote(lines)
order = {"customer_id": customer_id, "subtotal": subtotal, "tax": tax, "total": subtotal + tax}
self.repo.save(order)
return order
# Brittle (before): asserts HOW the service talks to its internals.
# It broke when subtotal() and tax() were merged into quote().
#
# pricing = Mock()
# pricing.subtotal.return_value = 2000
# pricing.tax.return_value = 160
# ...
# pricing.subtotal.assert_called_once_with(lines)
# pricing.tax.assert_called_once_with(2000)

# Behavioral (after): survives any internal refactor that keeps results the same.
from pricing import InMemoryOrderRepo, Line, OrderService, Pricing


def test_place_order_totals_and_persists():
repo = InMemoryOrderRepo()
service = OrderService(repo, Pricing())

order = service.place(7, [Line("A", 500, 2), Line("B", 1000, 1)])

assert order == {"customer_id": 7, "subtotal": 2000, "tax": 160, "total": 2160}
assert repo.saved == [order]

Workflow Diagram:

Refactor internal helpers (behavior unchanged)
|
+-- Interaction-coupled tests -> 300 failures -> refactor abandoned
|
+-- Behavioral tests ----------> 0 failures -> refactor merged
|
v
Mocks/fakes only at real boundaries (DB, HTTP, clock, queue)
Assertions on outputs + state, interactions only for side effects

🔍 Interviewer Notes: Principal candidates link test design to the cost of change and can explain the classicist (Detroit) and mockist (London) schools, choosing deliberately between them. Ask: "When is an interaction assertion the right thing?"

❌ Red Flags: "Every class needs its own test file with all dependencies mocked"; treating the test count as a quality metric; fixing broken mocks mechanically after every refactor.


Q51: Tests start failing on a specific date ("time bombs")​

Level: Senior

Scenario: On 1 January, 40 tests fail across the codebase with no code changes. Fixtures contain expires_at=datetime(2026, 12, 31) and a hard-coded "future" subscription renewal date. Separately, one test fails only when CI runs between 23:59 and 00:00 UTC.

Impact: Surprise red builds on a holiday, emergency fixes under pressure, and deploys blocked for reasons that have nothing to do with the change being deployed.

Root Cause: Tests mix absolute hard-coded dates with code that compares against the real current time. They were correct when written and become wrong as the calendar moves. The midnight failure happens because the test computes "today" at a different instant from the code under test.

Fix:

  1. Never compare real "now" against hard-coded dates. Either freeze time for the test or build dates relative to a single captured now.
  2. Inject a clock into domain code so tests decide the time explicitly.
  3. Capture "now" once per test and derive everything from it, so there are no midnight races.
  4. Add a scheduled job that runs the suite with the clock shifted forward (for example, under time-machine or libfaketime) to find remaining time bombs before the calendar does.

Code Example:

# subscriptions.py
from collections.abc import Callable
from datetime import UTC, datetime


def is_active(expires_at: datetime, now: Callable[[], datetime] = lambda: datetime.now(UTC)) -> bool:
return now() < expires_at
# test_subscriptions.py
from datetime import UTC, datetime, timedelta

from subscriptions import is_active


# BAD: correct until 2027-01-01, then fails forever
# def test_active_subscription():
# assert is_active(datetime(2026, 12, 31, tzinfo=UTC))


def test_active_subscription_relative_to_frozen_now():
frozen_now = datetime(2026, 6, 1, 12, 0, tzinfo=UTC)
expires_at = frozen_now + timedelta(days=30)
assert is_active(expires_at, now=lambda: frozen_now)
assert not is_active(expires_at, now=lambda: expires_at)


def test_with_time_machine_fixture(time_machine):
time_machine.move_to(datetime(2026, 12, 31, 23, 59, 59, tzinfo=UTC), tick=False)
now = datetime.now(UTC) # captured once, consistent with the code under test
assert is_active(now + timedelta(seconds=1))
assert not is_active(now)

Workflow Diagram:

Fixture: expires_at = 2026-12-31 Code: compares with real now()
\ /
+--------- passes in 2026 ----+
|
2027-01-01 arrives
|
v
40 tests fail, nothing changed
Fix:
freeze / inject clock -> dates relative to one captured "now"
nightly "clock shifted forward" run -> find remaining time bombs early

🔍 Interviewer Notes: Look for awareness that time is a hidden input to tests, and for the "capture now once" discipline that prevents midnight races.

❌ Red Flags: Bumping the hard-coded year every January; date.today() in both test and code; "it only fails at night, ignore it."


Q52: You must refactor a critical legacy module that has no tests​

Level: Principal

Scenario: A 3,000-line invoicing module from 2015 computes taxes, discounts and rounding for every invoice. It has no tests, no one fully understands it, and it must be refactored to support a new tax regime. Any behavior change, even rounding a cent differently, causes accounting discrepancies.

Impact: High risk of silent financial regressions, and a team too afraid to touch core revenue code.

Root Cause: Behavior is defined only by the code itself. You can't write "correct" specification tests because nobody knows the specification. The current behavior, including its quirks, is the spec until a decision is made to change it.

Fix:

  1. Write characterization (golden master) tests: record what the current code returns for a large, representative set of inputs (sampled, anonymized production data plus generated edge cases) and assert that the code keeps returning exactly that.
  2. Find seams (Michael Feathers): extract pure functions, inject the clock and the database, so the code can be called in a test harness.
  3. Refactor in small steps with the golden master green after every step.
  4. When a behavior change is intended (the new tax regime), regenerate only the affected cases and review the diff like code, with finance sign-off.
  5. Gradually replace golden cases with focused, named specification tests as understanding grows.

Code Example:

# scripts/record_golden_master.py - run once against the CURRENT code
import json
from pathlib import Path

from billing.legacy import calculate_invoice # the untested legacy entry point

CASES_IN = Path("tests/golden/invoice_inputs.json") # sampled + edge-case inputs
CASES_OUT = Path("tests/golden/invoice_cases.json")


def main() -> None:
inputs = json.loads(CASES_IN.read_text())
cases = [
{"name": item["name"], "input": item["input"], "expected": calculate_invoice(item["input"])}
for item in inputs
]
CASES_OUT.write_text(json.dumps(cases, indent=2, sort_keys=True, default=str))


if __name__ == "__main__":
main()
# tests/test_invoice_golden_master.py
import json
from pathlib import Path

import pytest

from billing.legacy import calculate_invoice

CASES = json.loads((Path(__file__).parent / "golden" / "invoice_cases.json").read_text())


@pytest.mark.parametrize("case", CASES, ids=[c["name"] for c in CASES])
def test_invoice_matches_recorded_behavior(case):
actual = json.loads(json.dumps(calculate_invoice(case["input"]), sort_keys=True, default=str))
assert actual == case["expected"]

Workflow Diagram:

Untested legacy module
|
v
Record golden master (current outputs for many real + edge inputs)
|
v
Introduce seams (extract pure functions, inject clock/DB)
|
v
Refactor in small steps ---> golden master must stay green
|
v
Intended change (new tax regime)?
yes -> regenerate affected cases -> diff reviewed with finance
|
v
Replace golden cases with named spec tests over time

🔍 Interviewer Notes: Principal candidates know they must lock in current behavior before improving it, and they treat golden-master diffs as business decisions, not test noise. Ask where the inputs come from and how PII is handled.

❌ Red Flags: A big-bang rewrite with no safety net; writing "what it should do" tests for code no one understands; regenerating golden files without reviewing the diff.