OneHub Technical Interview Masterclass: Pandas, NumPy & Polars
Total Questions: 305 Curriculum: Core Fundamentals (12), Applied Intermediate (12), Principal Advanced (12)
๐๏ธ Section 1: Core Fundamentalsโ
Q1: What is vectorization in NumPy and why is it faster than standard Python loops?โ
Technical Answer: Vectorization performs operations on whole arrays instead of element-by-element Python loops. NumPy pushes the work into optimized C/Fortran code and can use SIMD instructions, which avoids Python interpreter overhead and makes numeric operations much faster.
Code/Syntax Justification:
import numpy as np
a = np.array([1, 2, 3])
b = a * 2
๐ Behind the Question: Performance optimization in numerical code. โ The Red Flags: Looping over large arrays in pure Python.
Q2: What is broadcasting in NumPy and how does it work?โ
Technical Answer: Broadcasting lets NumPy operate on arrays of different shapes without copying data. It aligns dimensions from the end, and two dimensions are compatible if they are equal or one of them is 1. This enables concise element-wise math while keeping the operation vectorized.
Code/Syntax Justification:
import numpy as np
a = np.array([[1], [2], [3]])
b = np.array([10, 20, 30])
print(a + b)
๐ Behind the Question: Shape reasoning and array semantics. โ The Red Flags: Manually expanding arrays when broadcasting already works.
Q3: What is the difference between a NumPy view and a copy?โ
Technical Answer: A view shares the same underlying memory as the original array, while a copy allocates new memory. Views are faster and memory-efficient but can surprise you if you mutate them. Understanding this distinction is critical when slicing large arrays or optimizing memory use.
Code/Syntax Justification:
arr = np.arange(10)
view = arr[2:5]
copy = arr[2:5].copy()
๐ Behind the Question: Memory behavior and mutation safety. โ The Red Flags: Assuming every slice is an independent array.
Q4: What does array dtype affect in NumPy?โ
Technical Answer:
The dtype determines how each element is stored, how much memory the array uses, and what operations are valid or efficient. Picking the right dtype affects speed, precision, and interoperability. For example, float64 is more precise but uses more memory than float32.
Code/Syntax Justification:
x = np.array([1, 2, 3], dtype=np.int32)
๐ Behind the Question: Memory layout and numerical precision. โ The Red Flags: Ignoring dtype and assuming everything is a Python integer.
Q5: What does a NumPy stride tell you?โ
Technical Answer: A stride is the byte step needed to move from one element to the next along an axis. Strides reveal how the array is laid out in memory and whether an operation will be cache-friendly. They matter for slicing, transposes, and interoperability with low-level libraries.
Code/Syntax Justification:
print(arr.strides)
๐ Behind the Question: Memory layout and performance reasoning. โ The Red Flags: Ignoring that array layout affects speed.
Q6: What is the difference between flatten() and ravel()?โ
Technical Answer: flatten() returns a copy, so changes do not affect the original array. ravel() returns a view when possible, so it is more memory efficient but mutations may affect the parent array. Interviewers often use this to test whether you understand view semantics.
Code/Syntax Justification:
flat = arr.flatten()
view = arr.ravel()
๐ Behind the Question: Copy semantics and memory efficiency. โ The Red Flags: Assuming both behave identically.
Q7: Why can Pandas apply() be slow?โ
Technical Answer: apply() often executes Python-level functions row by row or group by group, which adds interpreter overhead and blocks vectorized execution. When a built-in aggregation or vectorized expression exists, it is usually much faster and easier to reason about.
Code/Syntax Justification:
df.groupby("country")["sales"].sum()
๐ Behind the Question: Tabular performance optimization.
โ The Red Flags: Using apply as the default solution.
Q8: What makes Polars lazy execution efficient?โ
Technical Answer: Lazy execution lets Polars build a query plan before running it, so it can push down filters, prune unused columns, and reorder operations for efficiency. This is especially valuable for large datasets because it reduces I/O and memory pressure.
Code/Syntax Justification:
import polars as pl
q = pl.scan_csv("data.csv").filter(pl.col("x") > 10).select(["x", "y"])
๐ Behind the Question: Query planning and columnar optimization. โ The Red Flags: Loading every row when only a subset is needed.
๐ง Section 2: Applied Intermediateโ
Q9: How does Pandas groupby work?โ
Technical Answer:
groupby splits data into groups, applies an aggregation or transformation, and then combines the results. It is often used for summaries like counts, sums, and means per category. The key interview point is knowing the difference between agg, transform, and apply.
Code/Syntax Justification:
df.groupby("country")["sales"].sum()
๐ Behind the Question: Aggregation semantics and tabular analysis.
โ The Red Flags: Using apply for everything.
Q10: What is the difference between agg, transform, and apply in Pandas?โ
Technical Answer:
agg returns one row per group with summarized values. transform returns a result aligned to the original shape, which is useful for per-row group statistics. apply is the most flexible but often the slowest and least predictable because it can return arbitrary shapes.
Code/Syntax Justification:
df["zscore"] = df.groupby("team")["score"].transform(lambda s: (s - s.mean()) / s.std())
๐ Behind the Question: Group operations and output shape.
โ The Red Flags: Reaching for apply when a vectorized method exists.
Q11: How do joins and merges differ from concatenation?โ
Technical Answer:
merge combines tables based on key columns, like SQL joins. concat stacks objects along an axis, either row-wise or column-wise, without matching keys unless you align them manually. The choice depends on whether your data relationship is relational or just structural.
Code/Syntax Justification:
pd.merge(users, orders, on="user_id", how="left")
๐ Behind the Question: Relational thinking in data manipulation. โ The Red Flags: Using concat where a key-based join is needed.
Q12: How do missing values work in Pandas?โ
Technical Answer:
Pandas represents missing values with NaN, NaT, or None depending on type. Handling missing data usually means detecting, dropping, filling, or leaving it intentionally. The correct choice depends on whether missingness is informative or just corruption.
Code/Syntax Justification:
df["age"] = df["age"].fillna(df["age"].median())
๐ Behind the Question: Data cleaning and imputation. โ The Red Flags: Blindly dropping all nulls.
๐ Section 3: Principal Advancedโ
Q13: What are strides in NumPy and why do they matter?โ
Technical Answer: Strides describe how many bytes to move in memory to reach the next element along each dimension. They determine whether an array view is contiguous, how slicing behaves, and how efficiently operations can traverse memory. Understanding strides helps explain performance differences and the cost of transposes and reshapes.
Code/Syntax Justification:
arr = np.arange(12).reshape(3, 4)
print(arr.strides)
๐ Behind the Question: Memory layout and cache locality. โ The Red Flags: Treating arrays as abstract boxes with no storage model.
Q14: Why can DataFrame.apply() be slow, and what should you use instead?โ
Technical Answer:
apply() often falls back to Python-level iteration, which defeats vectorization and adds overhead. Prefer built-in vectorized operations, groupby aggregations, merge, map, where, or NumPy functions whenever possible. The best interview answer usually ties slowness to Python overhead and object handling.
Code/Syntax Justification:
df["tax"] = df["amount"] * 0.2
๐ Behind the Question: Performance tuning in Pandas. โ The Red Flags: Writing row-wise Python callbacks for everything.
Q15: How does Polars lazy execution help performance?โ
Technical Answer: Lazy execution builds a query plan first, then optimizes it before running it. This allows predicate pushdown, projection pruning, and fewer scans over the data. It is one of the major reasons Polars can outperform eager DataFrame workflows on larger datasets.
Code/Syntax Justification:
import polars as pl
df = pl.scan_csv("data.csv").filter(pl.col("val") > 0).select(["id", "val"])
๐ Behind the Question: Query optimization and execution planning. โ The Red Flags: Thinking Polars is only about syntax similarity to Pandas.
Q16: How does Apache Arrow fit into Pandas and Polars?โ
Technical Answer: Arrow provides a columnar memory format that is efficient for analytics and zero-copy data exchange. Polars is built on Arrow-style columnar concepts, and modern Pandas increasingly interoperates with Arrow-backed arrays. Arrow matters because it reduces conversion overhead and improves interoperability between tools.
Code/Syntax Justification:
import pyarrow as pa
table = pa.Table.from_pandas(df)
๐ Behind the Question: Columnar storage and interoperability. โ The Red Flags: Ignoring memory format and only thinking about APIs.
๐ง Section 4: Expert Deep Divesโ
Q17: How do NumPy views and copies affect performance and correctness?โ
Technical Answer:
Views share memory with the original array, so they are cheap but can mutate the source unexpectedly. Copies allocate new memory, which is safer when you need isolation but more expensive for large arrays. Interviewers expect you to know when a slice is a view, when to call .copy(), and why that matters for memory pressure.
Code/Syntax Justification:
arr = np.arange(10)
view = arr[2:5]
copy = arr[2:5].copy()
๐ Behind the Question: Memory aliasing and mutation safety. โ The Red Flags: Assuming all slices are independent arrays.
Q18: What do NumPy strides tell you about an array?โ
Technical Answer: Strides describe how many bytes to move in memory to reach the next element along each axis. They explain why transposes and certain slices can be views and why some operations are cache-friendly while others are not. Understanding strides helps you reason about layout, contiguity, and performance.
Code/Syntax Justification:
arr = np.arange(12).reshape(3, 4)
print(arr.strides)
๐ Behind the Question: Memory layout and cache behavior. โ The Red Flags: Treating arrays as abstract nested lists only.
Q19: How does broadcasting work in NumPy?โ
Technical Answer: Broadcasting lets NumPy align shapes from the right and expand dimensions of size 1 implicitly. This enables vectorized operations without manually replicating data. A strong interview answer includes the compatibility rule and the benefit of avoiding Python loops and extra memory use.
Code/Syntax Justification:
a = np.array([[1], [2], [3]])
b = np.array([10, 20, 30])
print(a + b)
๐ Behind the Question: Shape alignment and vectorized math. โ The Red Flags: Manually tiling arrays instead of using broadcasting.
Q20: Why is flatten() different from ravel()?โ
Technical Answer:
flatten() returns a copy, while ravel() returns a view when possible. The difference matters when you want to reduce memory use or avoid side effects. This question is often used to test whether you understand array semantics beyond surface-level syntax.
Code/Syntax Justification:
flat = arr.flatten()
view = arr.ravel()
๐ Behind the Question: Copy semantics and memory efficiency. โ The Red Flags: Assuming both methods behave identically.
Q21: What is the difference between iloc, loc, and boolean indexing in Pandas?โ
Technical Answer:
iloc is position-based, loc is label-based, and boolean indexing filters rows by condition. The interview answer should mention that loc is usually safer for label-aware code and that boolean masks are often the clearest way to filter data. Mixing these up is a common source of bugs.
Code/Syntax Justification:
df.loc[df["age"] > 30, ["name", "age"]]
๐ Behind the Question: Indexing semantics and row selection. โ The Red Flags: Using the wrong indexer and getting silent row errors.
Q22: What is the difference between groupby().agg() and groupby().transform()?โ
Technical Answer:
agg reduces each group to one or more summary rows, while transform returns output aligned to the original shape. transform is ideal for group-wise normalization or per-row statistics, whereas agg is ideal for summaries. The distinction is important for both correctness and shape expectations.
Code/Syntax Justification:
df["zscore"] = df.groupby("team")["score"].transform(lambda s: (s - s.mean()) / s.std())
๐ Behind the Question: Output shape and group operations.
โ The Red Flags: Using apply where transform is the right fit.
Q23: Why can DataFrame.apply() be slow?โ
Technical Answer:
apply() often falls back to Python-level iteration and function calls, which defeats vectorization and adds interpreter overhead. Prefer built-in vectorized expressions, map, where, groupby reductions, or NumPy operations when possible. The real interview point is recognizing when you are leaving optimized code paths.
Code/Syntax Justification:
df["tax"] = df["amount"] * 0.2
๐ Behind the Question: Performance tuning in tabular workflows. โ The Red Flags: Writing row-wise Python callbacks as the default approach.
Q24: How do merge types differ in Pandas?โ
Technical Answer:
merge supports join styles similar to SQL: inner, left, right, and outer. Inner returns matches only, left preserves the left table, right preserves the right table, and outer keeps everything with nulls where data is missing. Knowing the join type is crucial for avoiding silent data loss or duplication.
Code/Syntax Justification:
pd.merge(users, orders, on="user_id", how="left")
๐ Behind the Question: Relational reasoning and join semantics. โ The Red Flags: Using concat when a key-based join is required.
Q25: How do missing values differ across NumPy and Pandas?โ
Technical Answer:
NumPy typically uses np.nan for floating-point missingness, while Pandas also has NaT for datetimes and nullable extension dtypes for more precise missing-value support. The important point is that missing values are type-sensitive and can affect comparisons, aggregations, and serialization. Robust code handles missingness explicitly.
Code/Syntax Justification:
df["age"] = df["age"].fillna(df["age"].median())
๐ Behind the Question: Data cleaning and null semantics. โ The Red Flags: Dropping all null rows without understanding the data.
Q26: What is Polars lazy execution and why is it fast?โ
Technical Answer: Lazy execution builds a query plan before execution, which allows filter pushdown, projection pruning, and optimized scan planning. This often reduces memory use and I/O, especially on large datasets. The core interview insight is that Polars can optimize the plan because it sees the whole query first.
Code/Syntax Justification:
import polars as pl
q = pl.scan_csv("data.csv").filter(pl.col("x") > 10).select(["x", "y"])
๐ Behind the Question: Query planning and execution optimization. โ The Red Flags: Assuming lazy just means delayed loading.
Q27: How does Apache Arrow improve interoperability between libraries?โ
Technical Answer: Arrow provides a columnar in-memory format that supports efficient analytics and zero-copy exchange between compatible systems. It reduces conversion overhead when moving data between Pandas, Polars, and other analytics tools. This matters for performance and for keeping pipelines from spending time reshaping the same data repeatedly.
Code/Syntax Justification:
import pyarrow as pa
table = pa.Table.from_pandas(df)
๐ Behind the Question: Columnar formats and data interchange. โ The Red Flags: Ignoring the cost of conversion between libraries.
Q28: What is the best way to optimize a slow Pandas pipeline?โ
Technical Answer:
Profile first, then reduce Python-level loops, select fewer columns, avoid unnecessary copies, use categorical dtypes when appropriate, and prefer vectorized operations over apply. For very large workloads, consider Polars or pushing the work closer to the storage layer. The best answer is specific, measured, and workload-aware.
Code/Syntax Justification:
df["country"] = df["country"].astype("category")
๐ Behind the Question: End-to-end performance tuning. โ The Red Flags: Guessing at speedups without measuring bottlenecks.
Q29: What is the difference between iloc and loc in Pandas?โ
Technical Answer: Use iloc for integer-position based slicing and loc for label-based indexing. Knowing the difference is essential because it affects correctness when your index is not a simple range.
Code/Syntax Justification:
df.loc[:, 'col']
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q30: How do you inspect an ndarray for shape, dtype, and strides?โ
Technical Answer: Shape tells you dimensions, dtype tells you storage type, and strides tell you byte stepping between elements. Together they reveal how the array is laid out and how expensive operations may be.
Code/Syntax Justification:
arr.shape; arr.dtype; arr.strides
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q31: Why is .copy() sometimes necessary when slicing?โ
Technical Answer: Slices often produce views, which share memory with the original array. A copy is necessary when you want isolation and do not want mutations to leak back to the source array.
Code/Syntax Justification:
safe = arr.copy()
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q32: What is a boolean mask and why is it useful?โ
Technical Answer: Boolean masks let you filter data using vectorized conditions rather than loops. They are concise, fast, and commonly used in numeric and tabular filtering.
Code/Syntax Justification:
arr[arr > 0]
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q33: What is the practical difference between groupby().agg() and groupby().transform()?โ
Technical Answer: Aggregation reduces each group to one row, while transform returns an aligned result with the original shape. Use transform when you need per-row group statistics instead of a collapsed summary.
Code/Syntax Justification:
df.groupby('team')['score'].transform('mean')
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q34: Why is apply() often discouraged in Pandas?โ
Technical Answer: Apply often falls back to Python-level loops, which are slower than native vectorized operations. When a built-in method exists, it is usually faster and easier to read.
Code/Syntax Justification:
df['x'] * 2
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q35: How do you merge tables safely in Pandas?โ
Technical Answer: Use merge with explicit keys and join type, and validate assumptions about cardinality when necessary. Incorrect merge logic is one of the most common sources of duplicated or dropped data.
Code/Syntax Justification:
pd.merge(left, right, on='id', how='left')
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q36: What is the benefit of categorical dtype?โ
Technical Answer: Categorical dtype stores repeated string values as integer codes plus a category table, which saves memory and can speed up groupby and sorting. It is especially useful for low-cardinality columns.
Code/Syntax Justification:
df['country'] = df['country'].astype('category')
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q37: How do views and contiguity relate to performance?โ
Technical Answer: Contiguous arrays are easier for CPU caches and vectorized kernels to process. Non-contiguous views can still work, but some operations may need copies or slower access patterns.
Code/Syntax Justification:
np.ascontiguousarray(arr)
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q38: What is lazy execution in Polars?โ
Technical Answer: Lazy execution builds a logical plan first and then optimizes it before running. This allows filter pushdown, projection pruning, and better memory use on large datasets.
Code/Syntax Justification:
pl.scan_csv('data.csv').select(['a','b'])
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q39: Why does Arrow matter in modern data tools?โ
Technical Answer: Arrow provides a columnar in-memory layout that many tools can share efficiently, often with zero-copy interoperability. That reduces conversion overhead between libraries.
Code/Syntax Justification:
pyarrow.Table
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q40: How do you reduce memory in Pandas dataframes?โ
Technical Answer: Use smaller dtypes, categorical columns, column selection, and avoid unnecessary object columns. Memory optimization is often as important as computation speed.
Code/Syntax Justification:
df.astype({'id':'int32'})
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q41: What is the difference between .head() and .sample() in testing data?โ
Technical Answer: Head gives the first rows and is useful for deterministic inspection, while sample gives a randomized subset. Sampling is better for exploring distribution or spotting anomalies in a larger table.
Code/Syntax Justification:
df.sample(5)
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q42: How do you handle missing data in analysis?โ
Technical Answer: Choose between dropping, imputing, or preserving nulls based on the meaning of the data and downstream model requirements. There is no universal best option.
Code/Syntax Justification:
df.fillna(0)
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q43: What is broadcasting used for in feature engineering?โ
Technical Answer: Broadcasting lets you apply scalar or shape-compatible operations across whole arrays without loops. It is one of the main reasons NumPy is fast and concise.
Code/Syntax Justification:
arr * 1.1
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q44: How do rolling windows help in time-series analysis?โ
Technical Answer: Rolling windows compute statistics over sliding subsets, such as moving averages or volatility measures. They are foundational for financial and sensor analytics.
Code/Syntax Justification:
df['ma'] = df['price'].rolling(7).mean()
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q45: What is the tradeoff between eager and lazy execution?โ
Technical Answer: Eager execution is simpler and immediate, while lazy execution enables global optimization. Use eager when simplicity matters and lazy when you need query planning and performance.
Code/Syntax Justification:
lazy.collect()
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q46: How do you think about shape transformations in NumPy?โ
Technical Answer: Every reshape or transpose must respect the total number of elements and the underlying memory layout. Good interview answers explain when an operation is a view versus a copy and why that matters.
Code/Syntax Justification:
arr.reshape(3, 2)
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q47: What makes Polars well suited to large-scale transformations?โ
Technical Answer: Polars uses Rust, parallel execution, and a query optimizer to reduce unnecessary work. It is especially good when you need scalable transformations with predictable performance.
Code/Syntax Justification:
pl.DataFrame(...)
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
Q48: Why should you care about cache locality in numeric work?โ
Technical Answer: Cache locality affects how quickly the CPU can load data for computation. Contiguous and columnar-friendly layouts often outperform object-heavy structures by a large margin.
Code/Syntax Justification:
arr.T
๐ Behind the Question: Added for deeper coverage and interview readiness. โ The Red Flags: Weak or vague answers that fail to justify design choices.
๐ฏ Section 4: Deep Coverageโ
Q49: What is the difference between iloc and loc in Pandas?โ
Technical Answer: iloc selects by integer position, while loc selects by label. loc is usually clearer for indexed business data, and iloc is ideal when you need positional slicing.
Code/Syntax Justification:
df.loc[df['status'] == 'active', ['id', 'email']]
๐ Behind the Question: Indexing semantics. โ The Red Flags: Using positional access when label-based intent is clearer.
Q50: What is the practical value of astype()?โ
Technical Answer: astype() controls dtypes explicitly, which affects memory use, arithmetic behavior, and join compatibility. It is often necessary when cleaning messy CSV data or normalizing categorical fields.
Code/Syntax Justification:
df['age'] = df['age'].astype('int64')
๐ Behind the Question: Data normalization and memory control. โ The Red Flags: Leaving everything as object strings.
Q51: Why is categorical dtype useful?โ
Technical Answer: Categorical dtype stores repeated values efficiently and can speed up grouping and comparisons. It is well suited for low-cardinality columns like status, country, or tier.
Code/Syntax Justification:
df['status'] = df['status'].astype('category')
๐ Behind the Question: Memory optimization for repeated labels. โ The Red Flags: Using object dtype for every repeated string column.
Q52: What does merge(validate=...) protect you from?โ
Technical Answer: Validation helps catch unexpected join cardinality like many-to-many merges when you expected one-to-one or many-to-one. It is a strong guard against silent data duplication.
Code/Syntax Justification:
pd.merge(left, right, on='id', validate='one_to_one')
๐ Behind the Question: Join correctness. โ The Red Flags: Assuming a merge preserved row counts without checking.
Q53: Why is sort_values() sometimes important before groupby logic?โ
Technical Answer: Sorting is not required for basic aggregation, but it matters for order-sensitive operations such as top-N selection, cumulative logic, and time-series processing. Understanding when order matters prevents subtle bugs.
Code/Syntax Justification:
df = df.sort_values(['user_id', 'timestamp'])
๐ Behind the Question: Order-dependent analytics. โ The Red Flags: Ignoring temporal ordering in event data.
Q54: What is the benefit of query() in Pandas?โ
Technical Answer: query() can make filtering expressions more readable for complex boolean logic and sometimes simpler to maintain. It is a style choice more than a performance feature.
Code/Syntax Justification:
df.query("country == 'US' and sales > 1000")
๐ Behind the Question: Readability in filtering. โ The Red Flags: Using query syntax when plain boolean indexing is clearer.
Q55: What is the difference between pivot and pivot_table?โ
Technical Answer: pivot reshapes data and requires unique index/column combinations, while pivot_table can aggregate duplicates with a function like mean or sum. pivot_table is safer for real-world data.
Code/Syntax Justification:
df.pivot_table(index='date', columns='region', values='sales', aggfunc='sum')
๐ Behind the Question: Reshaping with aggregation.
โ The Red Flags: Using pivot when duplicate combinations exist.
Q56: How do you reason about NumPy memory contiguity?โ
Technical Answer: Contiguous arrays store elements in a single predictable memory layout, which is often faster for low-level operations and library interop. Non-contiguous views can still be valid but may incur extra work.
Code/Syntax Justification:
arr.flags['C_CONTIGUOUS']
๐ Behind the Question: Memory layout and performance. โ The Red Flags: Assuming every view is cheap for downstream libraries.
Q57: What is the use of np.where()?โ
Technical Answer: np.where() performs vectorized conditional selection, which is faster and clearer than Python loops for element-wise branching. It is useful for feature engineering and masked transforms.
Code/Syntax Justification:
out = np.where(x > 0, x, 0)
๐ Behind the Question: Vectorized branching. โ The Red Flags: Writing element-by-element if/else loops over arrays.
Q58: Why can mixed types hurt Pandas performance?โ
Technical Answer: Mixed or object-heavy columns force Python-level operations and reduce vectorization opportunities. Normalizing types to numeric, datetime, or categorical often gives big speedups and cleaner semantics.
Code/Syntax Justification:
df['amount'] = pd.to_numeric(df['amount'], errors='coerce')
๐ Behind the Question: Type hygiene and speed. โ The Red Flags: Leaving a numeric column as strings.
Q59: What is the advantage of explode()?โ
Technical Answer: explode() converts list-like cells into separate rows, which is useful when denormalized data needs to be analyzed relationally. It is often a better first step than manual loops.
Code/Syntax Justification:
df.explode('tags')
๐ Behind the Question: Normalization of nested data. โ The Red Flags: Iterating row by row to unpack lists.
Q60: What is merge_asof() for?โ
Technical Answer: merge_asof() joins on nearest keys, typically timestamps, rather than exact equality. It is ideal for aligning events with the latest prior observation in time-series data.
Code/Syntax Justification:
pd.merge_asof(left, right, on='timestamp', direction='backward')
๐ Behind the Question: Time-aware joining. โ The Red Flags: Using a regular merge for time proximity problems.
Q61: How does groupby.transform() differ from aggregation?โ
Technical Answer: transform() returns output aligned to the original rows, which makes it useful for group-relative features like z-scores or percentages. Aggregation collapses rows into fewer results.
Code/Syntax Justification:
df['mean_sales'] = df.groupby('team')['sales'].transform('mean')
๐ Behind the Question: Output-shape control. โ The Red Flags: Using aggregation when you need row-aligned results.
Q62: Why is Polars often faster than Pandas?โ
Technical Answer: Polars uses a Rust backend, columnar execution, and parallelism that reduce overhead for many workloads. It can also optimize lazy queries more aggressively than typical eager Pandas workflows.
Code/Syntax Justification:
import polars as pl
df = pl.read_csv('data.csv')
๐ Behind the Question: Engine and execution model. โ The Red Flags: Comparing libraries without considering workload shape.
Q63: What is the value of lazy.scan_csv() over read_csv() in Polars?โ
Technical Answer: scan_csv() builds a lazy query plan without loading the entire file immediately, enabling projection and predicate pushdown. read_csv() is eager and loads data right away.
Code/Syntax Justification:
q = pl.scan_csv('data.csv').select(['x', 'y']).filter(pl.col('x') > 10)
๐ Behind the Question: Lazy I/O optimization. โ The Red Flags: Loading full datasets when only a few columns are needed.
Q64: How do you handle nulls differently in numeric and categorical data?โ
Technical Answer: Numeric nulls may be filled with medians, means, or sentinel values depending on meaning. Categorical nulls often need explicit missing labels or category-aware handling rather than numeric-style imputation.
Code/Syntax Justification:
df['region'] = df['region'].fillna('unknown')
๐ Behind the Question: Missing-data strategy. โ The Red Flags: Applying one blanket null policy to every column.
Q65: What is np.nan versus None in data workflows?โ
Technical Answer: np.nan is the standard floating missing marker in NumPy and Pandas numeric contexts, while None is a Python object that often forces object dtype. The choice affects dtype inference and performance.
Code/Syntax Justification:
arr = np.array([1.0, np.nan, 3.0])
๐ Behind the Question: Missing-value representation. โ The Red Flags: Mixing object nulls into numeric workflows casually.
Q66: Why is to_numpy() sometimes useful?โ
Technical Answer: to_numpy() converts a DataFrame or Series to a NumPy array when you need lower-level numeric processing or library interop. It should be used with awareness of dtype conversion and copy behavior.
Code/Syntax Justification:
values = df[['x', 'y']].to_numpy()
๐ Behind the Question: Interop and array-oriented computation. โ The Red Flags: Converting back and forth without a reason.
Q67: What does rolling().mean() tell you about a series?โ
Technical Answer: Rolling windows compute a statistic over a moving subset of the data, which is useful for trends, smoothing, and volatility analysis. The window size determines the time horizon of the measure.
Code/Syntax Justification:
df['ma7'] = df['price'].rolling(7).mean()
๐ Behind the Question: Time-series smoothing. โ The Red Flags: Using rolling stats without considering window semantics.
Q68: What is a memory-efficient way to handle large tabular data?โ
Technical Answer: Read only needed columns, use appropriate dtypes, process in chunks, and avoid unnecessary copies. In Polars, lazy projection pushdown and streaming can reduce memory pressure further.
Code/Syntax Justification:
pd.read_csv('data.csv', usecols=['id', 'value'])
๐ Behind the Question: Memory reduction tactics. โ The Red Flags: Loading every column and then dropping most of it.
Q69: Why are index choices important in Pandas?โ
Technical Answer: A good index can make alignment, joins, and label-based selection more expressive, but a bad or misleading index can create confusion and accidental duplicates. Index design is part of data modeling.
Code/Syntax Justification:
df = df.set_index('user_id')
๐ Behind the Question: Label semantics and data modeling. โ The Red Flags: Ignoring duplicate or meaningless indexes.
Q70: What is the benefit of pipe()?โ
Technical Answer: pipe() allows function composition while keeping transformations readable and testable. It is useful for building clean data pipelines without excessive intermediate variables.
Code/Syntax Justification:
df.pipe(clean_names).pipe(add_features)
๐ Behind the Question: Functional composition in data workflows. โ The Red Flags: Creating hidden side effects in pipeline functions.
Q71: How do you think about broadcasting errors?โ
Technical Answer: Broadcasting errors usually mean shapes are incompatible under NumPyโs trailing-dimension rules. The fix is to reshape intentionally rather than forcing operations that do not align mathematically.
Code/Syntax Justification:
arr.reshape(-1, 1) + vec
๐ Behind the Question: Shape reasoning. โ The Red Flags: Guessing shapes instead of inspecting them.
Q72: Why does groupby().apply() often underperform?โ
Technical Answer: apply() can fall back to Python callbacks per group, which is slower than vectorized aggregation or transform methods. Prefer built-in reductions whenever possible.
Code/Syntax Justification:
df.groupby('country')['sales'].sum()
๐ Behind the Question: Performance and API choice.
โ The Red Flags: Using apply() as a first resort for group operations.
Q73: When should you choose Polars over Pandas?โ
Technical Answer: Choose Polars when you need fast columnar processing, lazy query optimization, or large-scale transformations with lower memory overhead. Choose Pandas when ecosystem compatibility or notebook familiarity is more important.
Code/Syntax Justification:
pl.scan_csv('data.csv').filter(pl.col('value') > 0).collect()
๐ Behind the Question: Library selection tradeoffs. โ The Red Flags: Choosing a tool by popularity instead of workload fit.
๐ง Section 4: Massive Expansionโ
Q74: What is the difference between iloc and loc in Pandas?โ
Technical Answer:
iloc selects by integer position, while loc selects by label.
Code/Syntax Justification:
df.iloc[0]
df.loc["row_id"]
๐ Behind the Question: Indexing semantics. โ The Red Flags: Using positional selection when label selection is required.
Q75: What is the role of the index in Pandas?โ
Technical Answer: The index is the label axis used for alignment, joins, slicing, and time-series operations.
Code/Syntax Justification:
df = df.set_index("user_id")
๐ Behind the Question: Label alignment. โ The Red Flags: Ignoring duplicate or meaningless indexes.
Q76: What is alignment in Pandas arithmetic?โ
Technical Answer: Pandas aligns on labels before performing arithmetic, which can produce unexpected NaNs if indexes do not match.
Code/Syntax Justification:
df1 + df2
๐ Behind the Question: Label-aware computation. โ The Red Flags: Assuming arrays and DataFrames align by row order only.
Q77: Why is object dtype often a performance smell?โ
Technical Answer: Object dtype usually means Python objects instead of compact native types, which hurts speed and memory use.
Code/Syntax Justification:
df["category"] = df["category"].astype("category")
๐ Behind the Question: Memory efficiency. โ The Red Flags: Leaving everything as generic Python objects.
Q78: What is the difference between copy() and a view in NumPy?โ
Technical Answer:
A view shares memory with the original array, while copy() allocates new storage.
Code/Syntax Justification:
arr2 = arr.copy()
๐ Behind the Question: Memory ownership. โ The Red Flags: Mutating a slice and assuming the parent will not change.
Q79: What is broadcasting in NumPy really doing?โ
Technical Answer: It virtually expands smaller dimensions to make array shapes compatible without copying data.
Code/Syntax Justification:
mat + vec
๐ Behind the Question: Shape mechanics. โ The Red Flags: Reshaping blindly instead of checking compatibility.
Q80: What is a boolean mask?โ
Technical Answer:
A boolean mask is an array or Series of True/False values used to filter or select matching rows/elements.
Code/Syntax Justification:
filtered = df[df["age"] > 18]
๐ Behind the Question: Vectorized filtering. โ The Red Flags: Looping to filter rows one at a time.
Q81: What is the difference between merge and join in Pandas?โ
Technical Answer:
merge is a flexible SQL-like join on columns or indexes. join is a convenience method that often joins on index by default.
Code/Syntax Justification:
pd.merge(left, right, on="id", how="left")
๐ Behind the Question: Relational joins. โ The Red Flags: Using concatenation when a join is needed.
Q82: What is the difference between concat and merge?โ
Technical Answer:
concat stacks along an axis without key matching, while merge combines based on join keys.
Code/Syntax Justification:
pd.concat([df1, df2], axis=0)
๐ Behind the Question: Data combination semantics.
โ The Red Flags: Expecting concat to reconcile keys automatically.
Q83: What is groupby split-apply-combine?โ
Technical Answer:
groupby partitions rows into groups, applies an operation, and then combines the results.
Code/Syntax Justification:
df.groupby("country")["sales"].sum()
๐ Behind the Question: Aggregation model.
โ The Red Flags: Using apply when a direct reduction exists.
Q84: Why is apply() often slow in Pandas?โ
Technical Answer: It frequently executes Python callbacks per row or group, which blocks vectorization and adds interpreter overhead.
Code/Syntax Justification:
df["tax"] = df["amount"] * 0.2
๐ Behind the Question: Performance tuning.
โ The Red Flags: Treating apply as the default tool.
Q85: What is the difference between agg and transform?โ
Technical Answer:
agg reduces a group to summary values, while transform returns results aligned with the original rows.
Code/Syntax Justification:
df["mean_score"] = df.groupby("team")["score"].transform("mean")
๐ Behind the Question: Output shape reasoning.
โ The Red Flags: Using apply for a simple per-row group statistic.
Q86: What is a categorical dtype good for?โ
Technical Answer: It reduces memory use and speeds up operations on repeated labels.
Code/Syntax Justification:
df["status"] = df["status"].astype("category")
๐ Behind the Question: Compression and grouping speed. โ The Red Flags: Leaving repeated labels as plain Python strings.
Q87: What is dropna() good for and what is its risk?โ
Technical Answer: It removes missing values cleanly, but it can discard a lot of information if missingness is meaningful.
Code/Syntax Justification:
df = df.dropna(subset=["price"])
๐ Behind the Question: Missing-data handling. โ The Red Flags: Dropping all nulls by default.
Q88: What is fillna() used for?โ
Technical Answer: It replaces missing values with a chosen constant, statistic, or forward/backward fill strategy.
Code/Syntax Justification:
df["age"] = df["age"].fillna(df["age"].median())
๐ Behind the Question: Imputation strategy. โ The Red Flags: Filling all nulls with zero without thinking.
Q89: What is the difference between shift() and diff()?โ
Technical Answer:
shift() moves values relative to the index, while diff() computes differences between current and previous values.
Code/Syntax Justification:
df["delta"] = df["value"].diff()
๐ Behind the Question: Sequence operations. โ The Red Flags: Rewriting lag logic manually in loops.
Q90: What does resample() do in time series?โ
Technical Answer: It changes the frequency of a datetime-indexed Series or DataFrame, enabling aggregation over time buckets.
Code/Syntax Justification:
df.resample("D").sum()
๐ Behind the Question: Temporal aggregation. โ The Red Flags: Treating time series like ordinary rows.
Q91: What is a rolling window?โ
Technical Answer: A rolling window computes statistics over a moving subset of data, such as moving averages.
Code/Syntax Justification:
df["ma7"] = df["price"].rolling(7).mean()
๐ Behind the Question: Windowed analytics. โ The Red Flags: Looping through rows to compute rolling stats.
Q92: What is the difference between pivot() and pivot_table()?โ
Technical Answer:
pivot() reshapes data when index-value combinations are unique. pivot_table() adds aggregation.
Code/Syntax Justification:
df.pivot_table(index="day", columns="category", values="sales", aggfunc="sum")
๐ Behind the Question: Reshaping with aggregation.
โ The Red Flags: Using pivot() on duplicated keys and expecting it to work.
Q93: What is melt() used for?โ
Technical Answer: It converts wide data into long format, which is often better for analysis and plotting.
Code/Syntax Justification:
df.melt(id_vars=["id"], var_name="metric", value_name="value")
๐ Behind the Question: Tidy data transformation. โ The Red Flags: Keeping wide format when long format would simplify analysis.
Q94: What is astype() used for?โ
Technical Answer: It explicitly converts data to another dtype, which is important for performance and correctness.
Code/Syntax Justification:
df["qty"] = df["qty"].astype("int64")
๐ Behind the Question: Type control. โ The Red Flags: Leaving everything as inferred object dtype.
Q95: Why does to_datetime() matter?โ
Technical Answer: It parses strings into real datetime values so you can sort and resample correctly.
Code/Syntax Justification:
df["ts"] = pd.to_datetime(df["ts"])
๐ Behind the Question: Temporal correctness. โ The Red Flags: Doing date math on strings.
Q96: What is read_csv() often used with in performance-sensitive code?โ
Technical Answer:
Use usecols, dtype, and sometimes chunking to reduce memory and parsing overhead.
Code/Syntax Justification:
pd.read_csv("data.csv", usecols=["a", "b"], dtype={"a": "int32"})
๐ Behind the Question: Ingestion efficiency. โ The Red Flags: Loading huge files with default inference.
Q97: What is memory_usage() useful for?โ
Technical Answer: It helps identify columns or objects consuming too much memory.
Code/Syntax Justification:
df.memory_usage(deep=True)
๐ Behind the Question: Memory profiling. โ The Red Flags: Optimizing blindly without measuring footprint.
Q98: What is vectorization in NumPy?โ
Technical Answer: It applies operations across arrays in compiled code instead of Python loops.
Code/Syntax Justification:
arr = np.array([1, 2, 3])
arr * 2
๐ Behind the Question: High-performance numeric computation. โ The Red Flags: Writing Python loops over large arrays.
Q99: What is the difference between np.where() and boolean indexing?โ
Technical Answer:
np.where() can produce conditional outputs or indices, while boolean indexing selects rows or elements directly.
Code/Syntax Justification:
np.where(arr > 0, arr, 0)
๐ Behind the Question: Conditional vectorization. โ The Red Flags: Replacing a simple mask with manual loops.
Q100: What is np.arange() commonly used for?โ
Technical Answer: It creates evenly spaced integer sequences for indexing, testing, and synthetic arrays.
Code/Syntax Justification:
np.arange(10)
๐ Behind the Question: Array generation basics. โ The Red Flags: Using Python lists when a numeric array is required.
Q101: What is reshape() doing?โ
Technical Answer: It changes the view of data into a new shape when the number of elements is compatible.
Code/Syntax Justification:
arr.reshape(3, 4)
๐ Behind the Question: Shape manipulation. โ The Red Flags: Confusing reshaping with sorting or copying.
Q102: What is a contiguous array?โ
Technical Answer: It is laid out in a single dense memory block, which is often faster for low-level operations.
Code/Syntax Justification:
arr.flags["C_CONTIGUOUS"]
๐ Behind the Question: Memory layout efficiency. โ The Red Flags: Ignoring layout when performance matters.
Q103: What is the difference between flatten() and ravel()?โ
Technical Answer:
flatten() returns a copy, while ravel() returns a view when possible.
Code/Syntax Justification:
flat = arr.flatten()
view = arr.ravel()
๐ Behind the Question: Copy versus view semantics. โ The Red Flags: Assuming both are interchangeable.
Q104: Why do dtypes matter for numeric precision?โ
Technical Answer: Different dtypes trade precision, range, and memory.
Code/Syntax Justification:
np.array([1.0], dtype=np.float32)
๐ Behind the Question: Numerical correctness. โ The Red Flags: Treating all numeric types as equivalent.
Q105: What is a sparse representation?โ
Technical Answer: Sparse structures store only non-zero or meaningful values, saving memory for mostly empty data.
Code/Syntax Justification:
# use sparse types when data is mostly zeros
๐ Behind the Question: Memory-efficient representation. โ The Red Flags: Using dense arrays for mostly empty matrices.
Q106: What is pipe() useful for in Pandas?โ
Technical Answer: It enables readable function composition by passing a DataFrame through named transformation steps.
Code/Syntax Justification:
df.pipe(clean_names).pipe(add_features)
๐ Behind the Question: Functional pipeline composition. โ The Red Flags: Hiding side effects inside pipeline functions.
Q107: What is the difference between eager and lazy execution in Polars?โ
Technical Answer: Eager execution runs immediately, while lazy execution builds a query plan first.
Code/Syntax Justification:
pl.scan_csv("data.csv").filter(pl.col("x") > 10).collect()
๐ Behind the Question: Query planning. โ The Red Flags: Loading everything before filtering.
Q108: What is predicate pushdown in Polars?โ
Technical Answer: It pushes filters down toward the data source so irrelevant rows are skipped earlier.
Code/Syntax Justification:
pl.scan_csv("data.csv").filter(pl.col("x") > 10)
๐ Behind the Question: Scan optimization. โ The Red Flags: Filtering only after full materialization.
Q109: What is projection pruning?โ
Technical Answer: It removes unused columns from the query plan so fewer fields are read and processed.
Code/Syntax Justification:
pl.scan_csv("data.csv").select(["x", "y"])
๐ Behind the Question: Column minimization. โ The Red Flags: Reading every column when only a few are needed.
Q110: Why can groupby().apply() underperform?โ
Technical Answer: It often falls back to Python-level callbacks instead of optimized reductions.
Code/Syntax Justification:
df.groupby("country")["sales"].sum()
๐ Behind the Question: Groupby performance.
โ The Red Flags: Using apply as the default group operation.
Q111: What is the benefit of sort=False in some group operations?โ
Technical Answer: Skipping unnecessary sorting can improve performance when group order does not matter.
Code/Syntax Justification:
df.groupby("key", sort=False).sum()
๐ Behind the Question: Performance tuning. โ The Red Flags: Paying for ordering you do not use.
Q112: What is the benefit of category for joins and groupby?โ
Technical Answer: Categorical columns can make repeated-label operations faster and more memory efficient.
Code/Syntax Justification:
df["region"] = df["region"].astype("category")
๐ Behind the Question: Efficient label handling. โ The Red Flags: Leaving high-repeat string columns as object dtype.
Q113: What is the explode() method used for?โ
Technical Answer: It turns list-like values into separate rows, which is useful for normalization and analysis of nested data.
Code/Syntax Justification:
df.explode("tags")
๐ Behind the Question: Nested data normalization. โ The Red Flags: Writing custom loops to flatten list columns.
Q114: What is the point of assign()?โ
Technical Answer: It adds or replaces columns in a chainable way, making transformations easier to read.
Code/Syntax Justification:
df.assign(total=df["a"] + df["b"])
๐ Behind the Question: Pipeline readability. โ The Red Flags: Reusing many mutable intermediate DataFrames unnecessarily.
Q115: What is the benefit of query()?โ
Technical Answer: It provides a readable expression syntax for filtering rows, especially for complex boolean conditions.
Code/Syntax Justification:
df.query("age > 18 and country == 'US'")
๐ Behind the Question: Expressive filtering. โ The Red Flags: Using it where simple boolean indexing is clearer.
Q116: Why can chained indexing be dangerous?โ
Technical Answer:
Chained indexing can create ambiguous views or copies, leading to SettingWithCopy issues and silent bugs.
Code/Syntax Justification:
df.loc[df["x"] > 0, "y"] = 1
๐ Behind the Question: Assignment safety.
โ The Red Flags: Writing df[df.x > 0].y = 1.
Q117: What is to_numpy() useful for?โ
Technical Answer: It converts a DataFrame or Series into a NumPy array for downstream numeric computation or library interoperability.
Code/Syntax Justification:
arr = df[["x", "y"]].to_numpy()
๐ Behind the Question: Interop with array libraries. โ The Red Flags: Converting too early and losing labels unnecessarily.
Q118: What is the difference between values and to_numpy()?โ
Technical Answer:
to_numpy() is the preferred explicit API and handles dtype conversion more clearly.
Code/Syntax Justification:
df.to_numpy()
๐ Behind the Question: API preference. โ The Red Flags: Relying on legacy attribute behavior without understanding it.
Q119: What is memory locality and why does it matter?โ
Technical Answer: Operations that access contiguous memory are faster because they are cache-friendly.
Code/Syntax Justification:
arr.strides
๐ Behind the Question: Performance fundamentals. โ The Red Flags: Ignoring memory layout in numeric workloads.
Q120: What is the practical difference between Pandas and Polars for large data?โ
Technical Answer: Pandas is flexible and mature, but Polars often scales better for columnar, query-heavy workloads with lower memory overhead.
Code/Syntax Justification:
pl.scan_csv("data.csv").select(["a", "b"]).collect()
๐ Behind the Question: Library selection tradeoffs. โ The Red Flags: Selecting by habit instead of workload.
Q121: What is the danger of row-wise Python logic in data processing?โ
Technical Answer: Row-wise logic defeats vectorization and becomes a performance bottleneck on large datasets.
Code/Syntax Justification:
df["flag"] = np.where(df["score"] > 50, 1, 0)
๐ Behind the Question: Vectorization discipline. โ The Red Flags: Using Python loops for everything.
Q122: How do you think about reproducibility in data analysis?โ
Technical Answer: Reproducibility means fixing inputs, versions, random seeds, and transformation steps so results can be recreated.
Code/Syntax Justification:
np.random.seed(42)
๐ Behind the Question: Scientific rigor. โ The Red Flags: Rerunning notebooks with hidden state and no version control.
Q123: What is the senior-level answer for selecting Pandas, NumPy, or Polars?โ
Technical Answer: Use NumPy for raw numerical arrays, Pandas for labeled tabular analysis, and Polars for high-performance columnar transformations and lazy querying.
Code/Syntax Justification:
np.array([...])
pd.DataFrame(...)
pl.scan_csv("data.csv")
๐ Behind the Question: Workload-fit reasoning. โ The Red Flags: Picking a library based only on familiarity or hype.
๐ Section 4: Expert Extensionsโ
Q124: Why does column selection matter so much in Polars and Pandas?โ
Technical Answer: Selecting only needed columns reduces memory use and I/O, especially on large datasets. Column pruning is one of the easiest ways to make tabular workflows faster without changing the business logic.
Code/Syntax Justification:
df = df[["id", "score"]]
๐ Behind the Question: I/O and memory efficiency. โ The Red Flags: Loading or carrying around unused columns everywhere.
Q125: What is the practical difference between merge and join in Pandas?โ
Technical Answer:
merge is the general SQL-like table combine operation, while join is a convenience method often indexed on the left frame. The key is knowing which side supplies keys and how duplicates or many-to-many relationships behave.
Code/Syntax Justification:
pd.merge(left, right, on="user_id")
๐ Behind the Question: Relational operations and key alignment. โ The Red Flags: Joining without understanding key cardinality.
Q126: Why can chained assignment in Pandas be dangerous?โ
Technical Answer:
Chained assignment can operate on a temporary object rather than the original DataFrame, so writes may not behave as expected. Using .loc makes intent explicit and avoids ambiguous mutation.
Code/Syntax Justification:
df.loc[df["score"] > 50, "flag"] = 1
๐ Behind the Question: Mutation safety and indexing semantics.
โ The Red Flags: Ignoring SettingWithCopy warnings.
Q127: What is astype() used for and why should it be deliberate?โ
Technical Answer:
astype() changes dtypes, which can improve memory use or enable operations, but it can also silently truncate data or introduce bad conversions. Type changes should be intentional because they affect correctness and performance.
Code/Syntax Justification:
df["age"] = df["age"].astype("int32")
๐ Behind the Question: Dtype control. โ The Red Flags: Blindly casting without checking for nulls or precision loss.
Q128: How do missing values affect numeric workflows?โ
Technical Answer: Missing values can change dtypes, break arithmetic, and alter aggregations if not handled deliberately. Good workflows decide early whether to impute, drop, or propagate missing data based on the analytical goal.
Code/Syntax Justification:
df["score"] = df["score"].fillna(0)
๐ Behind the Question: Data cleanliness and null semantics. โ The Red Flags: Letting nulls silently distort calculations.
Q129: What is the advantage of using groupby().agg() over manual loops?โ
Technical Answer:
groupby().agg() delegates aggregation to optimized library code and keeps the intent concise. It is usually faster, easier to test, and less error-prone than writing Python loops for each group.
Code/Syntax Justification:
df.groupby("team").agg({"sales": "sum", "age": "mean"})
๐ Behind the Question: Aggregation efficiency. โ The Red Flags: Reimplementing basic statistics row by row.
Q130: Why is Categorical useful in Pandas?โ
Technical Answer: Categorical data reduces memory for repeated labels and can speed up comparisons and group operations. It is especially helpful for low-cardinality text columns like status, country, or tier.
Code/Syntax Justification:
df["status"] = df["status"].astype("category")
๐ Behind the Question: Memory optimization for repeated labels. โ The Red Flags: Keeping every repeated string as plain object dtype.
Q131: What is the difference between iloc and loc?โ
Technical Answer:
iloc selects by integer position, while loc selects by label. Interviewers use this to see whether you understand index semantics and avoid off-by-one or label confusion.
Code/Syntax Justification:
df.loc["row_1", "score"]
๐ Behind the Question: Indexing semantics. โ The Red Flags: Using position-based access when labels matter.
Q132: What makes Polars lazy queries easier to optimize than eager ones?โ
Technical Answer: Lazy queries let the engine analyze the whole plan before running it, so it can push filters down, remove unused columns, and streamline execution. That produces better performance on large pipelines.
Code/Syntax Justification:
pl.scan_csv("data.csv").filter(pl.col("x") > 10).select(["x"])
๐ Behind the Question: Query planning and execution optimization. โ The Red Flags: Collecting data too early in a pipeline.
Q133: Why do window functions matter in analytics?โ
Technical Answer: Window functions calculate metrics across a rolling or partitioned set of rows without collapsing the table. They are useful for running totals, rankings, and time-based comparisons.
Code/Syntax Justification:
df["rank"] = df.groupby("team")["score"].rank()
๐ Behind the Question: Per-row analytics within groups. โ The Red Flags: Forcing a full aggregation when you need row-level output.
Q134: How do you think about memory copies when slicing arrays?โ
Technical Answer: Slicing often creates views rather than copies in NumPy, which is efficient but can cause hidden mutations. Understanding when data is shared is critical for correctness and memory tuning.
Code/Syntax Justification:
sub = arr[1:5]
๐ Behind the Question: Shared-memory semantics. โ The Red Flags: Assuming every slice is independent.
Q135: What is the purpose of np.where?โ
Technical Answer:
np.where performs vectorized conditional logic across arrays, replacing explicit Python loops. It is a clean way to build classification flags or conditional transformations.
Code/Syntax Justification:
df["flag"] = np.where(df["score"] > 50, 1, 0)
๐ Behind the Question: Vectorized branching. โ The Red Flags: Writing loops for simple conditional assignment.
Q136: What is the value of describe() in exploratory analysis?โ
Technical Answer:
describe() gives quick summary statistics that reveal skew, spread, missingness, and suspicious ranges. It is a fast first pass before deeper analysis or modeling.
Code/Syntax Justification:
df.describe(include="all")
๐ Behind the Question: Fast data profiling. โ The Red Flags: Jumping into modeling without basic distribution checks.
Q137: Why is explode() useful for nested data?โ
Technical Answer: It normalizes list-like data into one row per element, making analysis and joins easier. This is common when ingesting APIs or event payloads with repeated fields.
Code/Syntax Justification:
df.explode("tags")
๐ Behind the Question: Data normalization. โ The Red Flags: Keeping nested lists in a shape that blocks analysis.
Q138: What is the practical downside of using apply everywhere?โ
Technical Answer:
apply often forces Python-level execution and can be much slower than vectorized alternatives. It is fine when there is no better built-in, but it should not be the default pattern.
Code/Syntax Justification:
df["score2"] = df["score"] * 2
๐ Behind the Question: Performance discipline.
โ The Red Flags: Choosing apply when arithmetic or map would do.
Q139: How do you think about reproducibility in notebooks?โ
Technical Answer: Reproducibility means controlling random seeds, library versions, input files, and execution order so results can be recreated later. Notebooks are convenient, but hidden state makes them easy to misuse.
Code/Syntax Justification:
np.random.seed(42)
๐ Behind the Question: Scientific workflow reliability. โ The Red Flags: Relying on notebook state from a previous run.
Q140: What is the benefit of using Polars for large scans?โ
Technical Answer: Polars can defer execution, prune work, and use a columnar engine that is efficient for large transformations. That often leads to better memory behavior than eager, row-oriented approaches.
Code/Syntax Justification:
pl.scan_parquet("data.parquet").select(["a", "b"]).collect()
๐ Behind the Question: Large-data execution strategy. โ The Red Flags: Loading more data than needed before filtering.
Q141: What is the difference between transform and agg in groupby?โ
Technical Answer:
agg reduces each group to a summary row, while transform returns an output aligned to the original rows. This distinction matters when you need a group metric but want to preserve row shape.
Code/Syntax Justification:
df["group_mean"] = df.groupby("team")["score"].transform("mean")
๐ Behind the Question: Shape-preserving vs reducing operations. โ The Red Flags: Using the wrong groupby method for the desired output shape.
Q142: Why should dtype selection be part of performance tuning?โ
Technical Answer: Different dtypes change memory footprint, precision, and CPU efficiency. Choosing the smallest correct type can reduce memory pressure and improve cache behavior in large workloads.
Code/Syntax Justification:
df["id"] = df["id"].astype("int32")
๐ Behind the Question: Memory and speed optimization. โ The Red Flags: Using default object dtype for everything.
Q143: What is the strongest concise senior summary of this library stack?โ
Technical Answer: NumPy powers fast numerical arrays, Pandas excels at labeled tabular analysis, and Polars provides high-performance columnar transformations with a strong lazy engine for large-scale data workflows.
Code/Syntax Justification:
np.array([1, 2])
pd.DataFrame({"x": [1]})
pl.scan_csv("data.csv")
๐ Behind the Question: Workload-fit reasoning across the stack. โ The Red Flags: Choosing libraries by habit instead of data shape and scale.
๐ฌ Section 5: Advanced Pandas Operationsโ
Q144: How does pd.eval() accelerate expression evaluation on DataFrames?โ
Technical Answer:
pd.eval() uses Numexpr under the hood to compile string expressions into optimized bytecode that avoids creating intermediate arrays. For large DataFrames, this reduces memory allocation overhead and can be 2โ5ร faster than chained Python operators.
Code/Syntax Justification:
import pandas as pd
import numpy as np
df = pd.DataFrame({"a": np.random.rand(1_000_000), "b": np.random.rand(1_000_000)})
# Slow: creates temporaries
result = df["a"] + df["b"] * 2 - df["a"] ** 0.5
# Fast: single-pass with numexpr
result = pd.eval("df.a + df.b * 2 - df.a ** 0.5")
# In-place with DataFrame.eval
df.eval("c = a + b * 2 - a ** 0.5", inplace=True)
๐ Behind the Question: Understanding expression-level optimization beyond vectorization.
โ The Red Flags: Never using pd.eval() or df.eval() for complex multi-column arithmetic on large frames.
Q145: What is the difference between DataFrame.pipe() and chaining methods directly?โ
Technical Answer:
pipe() passes the entire DataFrame as the first argument to a callable, enabling clean functional-style pipelines where each step is a standalone function. Direct chaining is limited to built-in methods and becomes unreadable with custom logic.
Code/Syntax Justification:
def drop_nulls(df, threshold=0.5):
return df.dropna(thresh=int(len(df) * threshold), axis=1)
def add_ratio(df, col_a, col_b, name):
return df.assign(**{name: df[col_a] / df[col_b]})
result = (
df
.pipe(drop_nulls, threshold=0.7)
.pipe(add_ratio, "revenue", "cost", "roi")
.query("roi > 1.5")
)
๐ Behind the Question: Composable data transformations. โ The Red Flags: Writing monolithic transformation blocks instead of composable pipeline steps.
Q146: How do you handle memory-efficient iteration over large DataFrames when vectorization isn't possible?โ
Technical Answer:
Use itertuples() over iterrows() โ it returns named tuples with C-speed attribute access and avoids per-row Series construction. For truly row-by-row logic that can't be vectorized, itertuples() is 10โ100ร faster than iterrows().
Code/Syntax Justification:
# Slow: constructs a Series per row
for idx, row in df.iterrows():
process(row["a"], row["b"])
# Fast: named tuples, no Series overhead
for row in df.itertuples(index=False):
process(row.a, row.b)
# Fastest for apply-style: use .values or .to_numpy()
values = df[["a", "b"]].to_numpy()
for a, b in values:
process(a, b)
๐ Behind the Question: Understanding the cost hierarchy of row access patterns.
โ The Red Flags: Using iterrows() in production code or not considering vectorized alternatives first.
Q147: How does pd.Categorical improve performance for low-cardinality string columns?โ
Technical Answer: Categorical stores an integer code array plus a small dictionary of unique values. For columns with few unique strings repeated millions of times, this reduces memory by 90%+ and speeds up groupby/sort/merge because integer comparisons replace string comparisons.
Code/Syntax Justification:
df["status"] = df["status"].astype("category")
# Memory comparison
print(df["status"].nbytes) # Categorical: ~1 byte per row
print(df["status"].cat.codes.nbytes)
# Ordered categories for proper sorting
df["priority"] = pd.Categorical(
df["priority"],
categories=["low", "medium", "high", "critical"],
ordered=True
)
df.sort_values("priority") # Sorts by defined order, not alphabetical
๐ Behind the Question: Memory optimization for real-world datasets with repeated labels. โ The Red Flags: Keeping high-cardinality string columns as object dtype without evaluation.
Q148: What is the difference between stack() and melt() in Pandas?โ
Technical Answer:
stack() pivots columns into the innermost row index level, producing a MultiIndex Series or DataFrame. melt() unpivots wide-format columns into long-format rows with explicit variable/value columns. stack() preserves index hierarchy; melt() creates a flat result.
Code/Syntax Justification:
wide = pd.DataFrame({
"name": ["Alice", "Bob"],
"math": [90, 85],
"science": [88, 92]
})
# stack: columns become inner index level
stacked = wide.set_index("name").stack()
# Alice math 90
# science 88
# Bob math 85
# science 92
# melt: explicit long format
melted = wide.melt(id_vars="name", var_name="subject", value_name="score")
# name subject score
# 0 Alice math 90
# 1 Bob math 85
# 2 Alice science 88
# 3 Bob science 92
๐ Behind the Question: Reshaping data for different analytical needs. โ The Red Flags: Confusing stack/unstack with melt/pivot or not knowing when each is appropriate.
Q149: How do you create efficient rolling window calculations with variable window sizes?โ
Technical Answer:
Use rolling() with an offset string for time-based variable windows, or use expanding() for cumulative windows. For truly custom variable windows, use apply() with Numba JIT or vectorized stride tricks.
Code/Syntax Justification:
# Fixed window
df["ma_7"] = df["price"].rolling(window=7).mean()
# Time-based variable window (requires DatetimeIndex)
df = df.set_index("timestamp")
df["ma_7d"] = df["price"].rolling("7D").mean()
# Exponential weighted moving average
df["ewm"] = df["price"].ewm(span=20, adjust=False).mean()
# Rolling with min_periods for partial windows
df["ma_7_partial"] = df["price"].rolling(window=7, min_periods=1).mean()
# Custom rolling with numba for speed
from numba import jit
@jit(nopython=True)
def weighted_mean(values):
w = np.arange(1, len(values) + 1, dtype=np.float64)
return np.sum(values * w) / np.sum(w)
df["wma"] = df["price"].rolling(7).apply(weighted_mean, engine="numba", raw=True)
๐ Behind the Question: Time-series computation patterns. โ The Red Flags: Implementing rolling calculations with explicit Python loops.
Q150: How does pd.merge_asof() differ from a regular merge?โ
Technical Answer:
merge_asof() performs an approximate join by matching the nearest key rather than requiring exact equality. It requires both DataFrames to be sorted by the join key and supports direction control (backward, forward, nearest) and tolerance limits.
Code/Syntax Justification:
trades = pd.DataFrame({
"time": pd.to_datetime(["10:01:02", "10:01:05", "10:01:10"]),
"ticker": ["AAPL", "AAPL", "AAPL"],
"price": [150.0, 150.5, 151.0]
})
quotes = pd.DataFrame({
"time": pd.to_datetime(["10:01:00", "10:01:03", "10:01:06"]),
"ticker": ["AAPL", "AAPL", "AAPL"],
"bid": [149.9, 150.3, 150.8]
})
# Match each trade to the most recent quote
merged = pd.merge_asof(
trades, quotes,
on="time",
by="ticker",
direction="backward",
tolerance=pd.Timedelta("5s")
)
๐ Behind the Question: Financial/time-series data joining patterns. โ The Red Flags: Using exact merge on timestamps or implementing manual nearest-match loops.
Q151: What are the performance implications of MultiIndex vs flat index with tuple columns?โ
Technical Answer:
MultiIndex enables hierarchical slicing with xs(), loc tuple indexing, and level-based groupby, but adds overhead for construction and some operations. Flat columns with string concatenation are simpler but lose structured access. Use MultiIndex when you need cross-section slicing; flatten when you need simple column selection.
Code/Syntax Justification:
# MultiIndex creation
arrays = [["bar", "bar", "baz", "baz"], ["one", "two", "one", "two"]]
mi = pd.MultiIndex.from_arrays(arrays, names=["first", "second"])
df = pd.DataFrame(np.random.randn(3, 4), columns=mi)
# Cross-section slicing
df.xs("one", level="second", axis=1)
# Flatten MultiIndex columns
df.columns = ["_".join(col).strip() for col in df.columns.values]
# Reset MultiIndex to flat
df_flat = df.reset_index()
๐ Behind the Question: Hierarchical data organization trade-offs. โ The Red Flags: Always using MultiIndex without considering whether flat structure is simpler.
Q152: How do you use pd.json_normalize() for deeply nested JSON?โ
Technical Answer:
json_normalize() flattens nested dictionaries and lists into a flat table, handling arbitrary nesting depth with dot-separated column names. Use record_path for nested arrays and meta to carry parent fields down.
Code/Syntax Justification:
data = [
{
"id": 1,
"info": {"name": "Alice", "address": {"city": "NYC", "zip": "10001"}},
"orders": [
{"oid": 101, "amount": 50},
{"oid": 102, "amount": 75}
]
}
]
# Flatten top-level nesting
flat = pd.json_normalize(data, sep="_")
# Columns: id, info_name, info_address_city, info_address_zip, orders
# Flatten nested array with parent metadata
orders = pd.json_normalize(
data,
record_path="orders",
meta=["id", ["info", "name"]],
sep="_"
)
# Columns: oid, amount, id, info_name
๐ Behind the Question: API response processing into tabular format. โ The Red Flags: Manually looping through nested JSON to build DataFrames.
Q153: How does DataFrame.query() compare to boolean indexing for readability and performance?โ
Technical Answer:
query() uses pd.eval() internally with Numexpr, so it's faster for large frames and avoids repetitive df["col"] syntax. Boolean indexing is more flexible for complex conditions involving function calls. For simple comparisons on large data, query() wins on both clarity and speed.
Code/Syntax Justification:
# Boolean indexing โ verbose
result = df[(df["age"] > 25) & (df["salary"] < 100000) & (df["dept"] == "eng")]
# query() โ cleaner, faster on large frames
result = df.query("age > 25 and salary < 100000 and dept == 'eng'")
# query with variables using @
min_age = 25
max_salary = 100000
result = df.query("age > @min_age and salary < @max_salary")
๐ Behind the Question: Idiomatic filtering patterns and their performance characteristics.
โ The Red Flags: Always using boolean indexing without considering query() for complex filters.
Q154: What is pd.StringDtype() and why prefer it over object dtype for text?โ
Technical Answer:
pd.StringDtype() (backed by "string" or "string[pyarrow]") provides proper NA handling (returns pd.NA instead of NaN), consistent return types from string methods, and enables Arrow-backed string storage that is 50-70% more memory-efficient than object dtype.
Code/Syntax Justification:
# Default object dtype โ NaN for missing, mixed types
s_obj = pd.Series(["hello", None, "world"]) # dtype: object
# StringDtype โ proper NA, consistent types
s_str = pd.Series(["hello", None, "world"], dtype="string")
# s_str.isna() returns pd.NA-based booleans
# Arrow-backed strings โ best memory and performance
s_arrow = pd.Series(["hello", None, "world"], dtype="string[pyarrow]")
print(s_arrow.memory_usage(deep=True)) # Much smaller than object
๐ Behind the Question: Modern Pandas type system evolution. โ The Red Flags: Using object dtype for all text columns in Pandas 2.x+.
Q155: How do you implement efficient conditional column creation without apply()?โ
Technical Answer:
Use np.where() for binary conditions, np.select() for multi-condition mapping, or pd.cut()/pd.qcut() for binning. All are vectorized and orders of magnitude faster than row-wise apply().
Code/Syntax Justification:
# Binary condition
df["flag"] = np.where(df["score"] > 80, "pass", "fail")
# Multi-condition
conditions = [
df["score"] >= 90,
df["score"] >= 70,
df["score"] >= 50,
]
choices = ["A", "B", "C"]
df["grade"] = np.select(conditions, choices, default="F")
# Binning
df["bucket"] = pd.cut(df["score"], bins=[0, 50, 70, 90, 100], labels=["F", "C", "B", "A"])
df["quartile"] = pd.qcut(df["score"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])
๐ Behind the Question: Vectorized conditional logic patterns.
โ The Red Flags: Using df.apply(lambda row: ..., axis=1) for simple conditional column creation.
Q156: How does DataFrame.explode() work and when do you use it?โ
Technical Answer:
explode() transforms list-like elements in a column into separate rows, replicating the index. It's the inverse of a groupby-agg-list operation. Essential for normalizing denormalized data where a single row contains a list of values.
Code/Syntax Justification:
df = pd.DataFrame({
"user": ["Alice", "Bob"],
"tags": [["python", "pandas"], ["numpy", "scipy", "polars"]]
})
exploded = df.explode("tags")
# user tags
# 0 Alice python
# 0 Alice pandas
# 1 Bob numpy
# 1 Bob scipy
# 1 Bob polars
# Multi-column explode (Pandas 1.3+)
df["scores"] = [[90, 85], [70, 80, 95]]
exploded = df.explode(["tags", "scores"]) # Must be same length per row
๐ Behind the Question: Denormalization reversal in data pipelines. โ The Red Flags: Manually looping to split list columns into rows.
Q157: What is the Styler API and how do you use it for presentation-quality DataFrames?โ
Technical Answer:
DataFrame.style returns a Styler object that applies CSS-based formatting for Jupyter/HTML output. It supports conditional formatting, gradients, bar charts in cells, and export to Excel with formatting preserved.
Code/Syntax Justification:
styled = (
df.style
.background_gradient(subset=["revenue"], cmap="Greens")
.highlight_max(subset=["profit"], color="gold")
.format({"revenue": "${:,.0f}", "margin": "{:.1%}"})
.bar(subset=["growth"], color=["#d65f5f", "#5fba7d"])
.set_caption("Q4 Financial Summary")
)
styled.to_excel("report.xlsx", engine="openpyxl")
๐ Behind the Question: Data presentation in analytical workflows. โ The Red Flags: Manually formatting DataFrames for presentation instead of using the Styler API.
Q158: How do you handle timezone-aware datetime operations in Pandas?โ
Technical Answer:
Use tz_localize() to assign a timezone to naive datetimes, tz_convert() to convert between timezones. Store timestamps as UTC internally and convert to local time only for display. Mixing timezone-aware and naive datetimes raises errors.
Code/Syntax Justification:
# Localize naive to UTC
ts = pd.to_datetime("2024-01-15 10:30:00")
ts_utc = ts.tz_localize("UTC")
# Convert to local
ts_est = ts_utc.tz_convert("US/Eastern")
# Series operations
df["created_utc"] = pd.to_datetime(df["created"]).dt.tz_localize("UTC")
df["created_local"] = df["created_utc"].dt.tz_convert("America/New_York")
# Timezone-aware date range
idx = pd.date_range("2024-01-01", periods=24, freq="h", tz="Europe/London")
๐ Behind the Question: Correct timezone handling for global applications. โ The Red Flags: Storing local times without timezone info or mixing aware/naive timestamps.
Q159: What is pd.IntervalIndex and when is it useful?โ
Technical Answer:
IntervalIndex represents a set of intervals (ranges) that support containment checks, overlap detection, and binned data indexing. It's created automatically by pd.cut() and is useful for range-based lookups like pricing tiers or time windows.
Code/Syntax Justification:
# Created by pd.cut
bins = pd.cut([15, 25, 45, 65, 80], bins=[0, 18, 35, 55, 100])
# Categories: [(0, 18], (18, 35], (35, 55], (55, 100]]
# Manual IntervalIndex
ii = pd.IntervalIndex.from_breaks([0, 10, 20, 30, 40])
s = pd.Series(["free", "basic", "pro", "enterprise"], index=ii)
# Lookup by value
print(s.loc[15]) # "basic" โ 15 falls in (10, 20]
print(s.loc[35]) # "enterprise" โ 35 falls in (30, 40]
๐ Behind the Question: Range-based indexing for pricing, scoring, and binning. โ The Red Flags: Implementing range lookups with manual loops instead of IntervalIndex.
Q160: How do you efficiently compare two DataFrames to find differences?โ
Technical Answer:
Use DataFrame.compare() for element-wise diff, merge() with indicator for set operations, or pd.testing.assert_frame_equal() for test assertions. For large frames, hash-based comparison with pd.util.hash_pandas_object() identifies changed rows quickly.
Code/Syntax Justification:
# Element-wise comparison (Pandas 1.1+)
diff = df1.compare(df2, keep_shape=False, keep_equal=False)
# Shows self/other values for each differing cell
# Set operations with merge indicator
merged = df1.merge(df2, how="outer", indicator=True)
added = merged[merged["_merge"] == "right_only"]
removed = merged[merged["_merge"] == "left_only"]
# Hash-based row comparison for large frames
h1 = pd.util.hash_pandas_object(df1)
h2 = pd.util.hash_pandas_object(df2)
changed_rows = df1[h1 != h2]
๐ Behind the Question: Data validation and change detection in pipelines.
โ The Red Flags: Comparing DataFrames with == without handling NaN, shape, or dtype mismatches.
Q161: What is pd.NamedAgg and how does it improve grouped aggregation?โ
Technical Answer:
pd.NamedAgg provides explicit column naming in grouped aggregations, replacing the deprecated dict-of-dicts syntax. It produces flat column names directly and supports multiple aggregations on the same source column.
Code/Syntax Justification:
result = df.groupby("department").agg(
avg_salary=pd.NamedAgg(column="salary", aggfunc="mean"),
max_salary=pd.NamedAgg(column="salary", aggfunc="max"),
headcount=pd.NamedAgg(column="employee_id", aggfunc="count"),
total_bonus=pd.NamedAgg(column="bonus", aggfunc="sum"),
)
# Flat columns: avg_salary, max_salary, headcount, total_bonus
# Shorthand with keyword arguments
result = df.groupby("dept").agg(
avg_sal=("salary", "mean"),
max_sal=("salary", "max"),
count=("id", "nunique"),
)
๐ Behind the Question: Clean aggregation APIs for readable analytics code.
โ The Red Flags: Using .agg() with dicts and then manually renaming/flattening columns.
Q162: How do you use DataFrame.assign() for functional-style column creation?โ
Technical Answer:
assign() returns a new DataFrame with added/modified columns, enabling method chaining without mutating the original. Lambda arguments receive the calling DataFrame, so new columns can reference columns created in the same call.
Code/Syntax Justification:
result = (
df
.assign(
revenue_k=lambda d: d["revenue"] / 1000,
cost_k=lambda d: d["cost"] / 1000,
profit_k=lambda d: d.revenue_k - d.cost_k, # References earlier assign
margin=lambda d: d.profit_k / d.revenue_k,
)
.query("margin > 0.2")
)
๐ Behind the Question: Immutable DataFrame transformations for pipeline clarity.
โ The Red Flags: Mutating DataFrames in place with df["col"] = ... inside complex pipelines.
Q163: What are Pandas Extension Arrays and how do you create custom ones?โ
Technical Answer:
Extension Arrays allow custom data types that integrate with the Pandas type system. Built-in examples include Int64 (nullable integer), StringDtype, and ArrowDtype. Custom Extension Arrays must subclass ExtensionArray and register an ExtensionDtype.
Code/Syntax Justification:
# Built-in nullable integer (capital I)
s = pd.array([1, 2, None], dtype="Int64") # Not np.int64
print(s[2]) # <NA>, not NaN
# Arrow-backed extension types (Pandas 2.0+)
import pyarrow as pa
s = pd.Series([1, 2, 3], dtype="int64[pyarrow]")
s_str = pd.Series(["a", "b", "c"], dtype="string[pyarrow]")
# Check if a dtype is an extension type
print(pd.api.types.is_extension_array_dtype(s.dtype)) # True
๐ Behind the Question: Understanding the extensible type system in modern Pandas.
โ The Red Flags: Not knowing the difference between int64 and Int64 (nullable) in Pandas.
Q164: How do you optimize memory when reading large CSV files?โ
Technical Answer:
Specify dtype to prevent upcasting, use usecols to read only needed columns, parse_dates to avoid post-load conversion, and chunksize for streaming. For categorical columns, pass a dtype dict. Converting to Parquet after first read eliminates repeated CSV parsing costs.
Code/Syntax Justification:
dtypes = {
"id": "int32",
"status": "category",
"name": "string",
"amount": "float32",
}
df = pd.read_csv(
"large_file.csv",
dtype=dtypes,
usecols=["id", "status", "name", "amount", "created_at"],
parse_dates=["created_at"],
na_values=["", "NULL", "N/A"],
low_memory=True,
)
# Streaming with chunks
chunks = pd.read_csv("huge.csv", chunksize=100_000, dtype=dtypes)
result = pd.concat([chunk.query("amount > 100") for chunk in chunks])
# Convert to Parquet for repeated access
df.to_parquet("data.parquet", engine="pyarrow", compression="zstd")
๐ Behind the Question: Production data loading optimization. โ The Red Flags: Reading CSVs with default settings and converting types after loading.
Q165: How does pd.Grouper enable time-based grouping?โ
Technical Answer:
pd.Grouper specifies groupby keys with frequency resampling. It allows grouping by time periods (daily, weekly, monthly) combined with categorical columns in a single groupby() call, replacing separate resample() calls.
Code/Syntax Justification:
# Group by month and category
df["date"] = pd.to_datetime(df["date"])
monthly = df.groupby([pd.Grouper(key="date", freq="ME"), "category"]).agg(
total=("amount", "sum"),
count=("id", "count"),
)
# Weekly grouping starting on Monday
weekly = df.groupby(pd.Grouper(key="date", freq="W-MON")).mean()
# Business quarter grouping
quarterly = df.groupby(pd.Grouper(key="date", freq="QS")).sum()
๐ Behind the Question: Time-based aggregation patterns for reporting. โ The Red Flags: Manually extracting year/month columns for time-based groupby.
Q166: What is the .str accessor and what are its most powerful methods?โ
Technical Answer:
The .str accessor exposes vectorized string operations on Series with object or StringDtype. Key methods include extract() for regex capture groups, contains() for pattern matching, split() with expand=True, and replace() with regex support.
Code/Syntax Justification:
# Regex extraction with named groups
df["area_code"] = df["phone"].str.extract(r"\((\d{3})\)")
# Pattern matching
mask = df["email"].str.contains(r"@company\.com$", regex=True, na=False)
# Split and expand
df[["first", "last"]] = df["name"].str.split(" ", n=1, expand=True)
# Chained string operations
df["slug"] = (
df["title"]
.str.lower()
.str.strip()
.str.replace(r"[^\w\s]", "", regex=True)
.str.replace(r"\s+", "-", regex=True)
)
๐ Behind the Question: Text processing within DataFrames.
โ The Red Flags: Using apply(lambda x: x.lower()) instead of .str.lower().
Q167: How do you implement efficient cross-tabulation with pd.crosstab()?โ
Technical Answer:
crosstab() computes a frequency table of two or more factors. It supports normalization (row/column/all), custom aggregation functions, and margins (row/column totals). It's syntactic sugar over pivot_table() with aggfunc="count".
Code/Syntax Justification:
# Frequency table
ct = pd.crosstab(df["department"], df["level"])
# Normalized by row (percentages)
ct_pct = pd.crosstab(df["department"], df["level"], normalize="index")
# With margins (totals)
ct_margins = pd.crosstab(df["dept"], df["level"], margins=True, margins_name="Total")
# Custom aggregation
ct_mean = pd.crosstab(
df["dept"], df["level"],
values=df["salary"], aggfunc="mean"
)
๐ Behind the Question: Categorical data analysis and contingency tables. โ The Red Flags: Manually pivoting and counting for cross-tabulations.
Q168: How do you use pd.SparseDtype for memory-efficient storage of mostly-zero data?โ
Technical Answer: Sparse arrays store only non-fill values plus an offset array, dramatically reducing memory for data with a dominant fill value (typically 0 or NaN). Useful for one-hot encoded features, indicator matrices, and scientific data.
Code/Syntax Justification:
# Dense: 8 bytes ร 1M = 8 MB
dense = pd.Series(np.zeros(1_000_000))
# Sparse: stores only non-zero positions
sparse = dense.astype(pd.SparseDtype("float64", fill_value=0.0))
print(f"Dense: {dense.memory_usage():,} bytes")
print(f"Sparse: {sparse.memory_usage():,} bytes")
# Create sparse from get_dummies
dummies = pd.get_dummies(df["category"], sparse=True)
print(dummies.dtypes) # Sparse[uint8, 0]
๐ Behind the Question: Memory optimization for sparse datasets. โ The Red Flags: Using dense arrays for one-hot encoded data with hundreds of categories.
Q169: How do you perform window functions with expanding() vs rolling() vs ewm()?โ
Technical Answer:
rolling(n) uses a fixed-size sliding window. expanding() uses all data from the start up to the current point (cumulative). ewm() applies exponential weighting where recent observations have more influence. Each serves a different analytical purpose.
Code/Syntax Justification:
prices = pd.Series([100, 102, 101, 105, 103, 107, 106])
# Rolling: fixed window
prices.rolling(3).mean() # NaN, NaN, 101.0, 102.67, 103.0, ...
# Expanding: cumulative from start
prices.expanding(min_periods=1).mean() # 100.0, 101.0, 101.0, 102.0, ...
# EWM: exponential decay (recent data weighted more)
prices.ewm(span=3).mean() # Smoothed with exponential decay
# Comparison for volatility
rolling_std = prices.rolling(20).std() # Fixed lookback
expanding_std = prices.expanding().std() # All historical data
ewm_std = prices.ewm(span=20).std() # Emphasize recent volatility
๐ Behind the Question: Different window semantics for time-series analysis. โ The Red Flags: Using only rolling windows without considering whether expanding or EWM is more appropriate.
Q170: What is DataFrame.select_dtypes() and how do you use it for type-based column selection?โ
Technical Answer:
select_dtypes() filters columns by their dtype, accepting include and exclude parameters. Essential for preprocessing pipelines where numeric, categorical, and text columns require different transformations.
Code/Syntax Justification:
# Select numeric columns only
numeric_cols = df.select_dtypes(include=["number"])
# Select categorical and object columns
cat_cols = df.select_dtypes(include=["category", "object"])
# Exclude datetime columns
no_dates = df.select_dtypes(exclude=["datetime64"])
# Common in sklearn pipelines
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
num_features = df.select_dtypes(include="number").columns.tolist()
cat_features = df.select_dtypes(include=["object", "category"]).columns.tolist()
preprocessor = ColumnTransformer([
("num", StandardScaler(), num_features),
("cat", OneHotEncoder(), cat_features),
])
๐ Behind the Question: Automated feature engineering based on data types. โ The Red Flags: Hardcoding column names instead of selecting by dtype in preprocessing.
Q171: How does pd.read_sql() compare to pd.read_sql_query() and pd.read_sql_table()?โ
Technical Answer:
read_sql() is a convenience wrapper that dispatches to read_sql_query() for SQL strings or read_sql_table() for table names. read_sql_query() accepts arbitrary SQL and supports chunksize for streaming. read_sql_table() reads an entire table with optional column selection.
Code/Syntax Justification:
from sqlalchemy import create_engine
engine = create_engine("postgresql://user:pass@host/db")
# Full query with parameters (prevents SQL injection)
df = pd.read_sql_query(
"SELECT * FROM orders WHERE status = %(status)s AND created_at > %(since)s",
engine,
params={"status": "active", "since": "2024-01-01"},
parse_dates=["created_at"],
)
# Streaming large results
for chunk in pd.read_sql_query("SELECT * FROM big_table", engine, chunksize=50_000):
process(chunk)
# Write back to database
df.to_sql("results", engine, if_exists="append", index=False, method="multi")
๐ Behind the Question: Database integration patterns in analytics workflows. โ The Red Flags: Using string formatting for SQL parameters or loading entire tables into memory.
Q172: How do you handle duplicate rows and values in Pandas?โ
Technical Answer:
Use duplicated() to detect duplicates (returns boolean mask), drop_duplicates() to remove them, and keep parameter to control which occurrence survives. For fuzzy deduplication, combine with string similarity libraries.
Code/Syntax Justification:
# Detect duplicates
mask = df.duplicated(subset=["email"], keep="first")
print(f"Duplicate rows: {mask.sum()}")
# Remove duplicates keeping last occurrence
df_clean = df.drop_duplicates(subset=["email"], keep="last")
# Remove all occurrences of duplicated values
df_unique = df.drop_duplicates(subset=["email"], keep=False)
# Mark duplicates with count
df["dup_count"] = df.groupby("email")["email"].transform("count")
df_dups = df[df["dup_count"] > 1].sort_values("email")
๐ Behind the Question: Data quality and deduplication in ETL. โ The Red Flags: Not checking for duplicates before joins or aggregations.
Q173: What is the pd.option_context() manager and how does it control display settings?โ
Technical Answer:
option_context() temporarily changes Pandas display settings within a context block, automatically reverting on exit. Useful for debugging, logging, and notebook display without permanently altering global state.
Code/Syntax Justification:
# Temporary display settings
with pd.option_context(
"display.max_rows", 100,
"display.max_columns", 50,
"display.width", 200,
"display.float_format", "{:.4f}".format,
"display.max_colwidth", 80,
):
print(df)
# Permanently set options
pd.set_option("display.max_rows", 200)
# Reset to defaults
pd.reset_option("display.max_rows")
# Show all options
pd.describe_option("display")
๐ Behind the Question: Controlling output for debugging and presentation. โ The Red Flags: Permanently changing global options or truncating output without knowing how to expand it.
๐งฎ Section 6: NumPy Advanced Computingโ
Q174: How does NumPy broadcasting work and what are the rules?โ
Technical Answer: Broadcasting allows operations between arrays of different shapes by virtually expanding the smaller array. Rules: (1) dimensions are compared right-to-left; (2) dimensions are compatible when equal or one is 1; (3) missing dimensions are treated as 1. No data is copied โ the expansion is virtual.
Code/Syntax Justification:
import numpy as np
a = np.array([[1], [2], [3]]) # Shape: (3, 1)
b = np.array([10, 20, 30]) # Shape: (3,) โ broadcast to (1, 3)
c = a + b # Shape: (3, 3) โ outer sum
# [[ 11, 21, 31],
# [ 12, 22, 32],
# [ 13, 23, 33]]
# Subtract row mean from each element
data = np.random.rand(1000, 50)
row_means = data.mean(axis=1, keepdims=True) # Shape: (1000, 1)
centered = data - row_means # Broadcasting along axis=1
๐ Behind the Question: Core NumPy paradigm for eliminating loops. โ The Red Flags: Using explicit loops or tile/repeat to match shapes manually.
Q175: What is the difference between np.copy(), np.view(), and array slicing?โ
Technical Answer:
Slicing creates a view (shared memory โ mutations propagate). view() explicitly creates a view with optional dtype reinterpretation. copy() creates an independent array with its own memory. Use .base to check if an array is a view.
Code/Syntax Justification:
a = np.array([1, 2, 3, 4, 5])
# View: shared memory
b = a[1:4] # View
b[0] = 99
print(a) # [1, 99, 3, 4, 5] โ mutated!
# Copy: independent memory
c = a.copy()
c[0] = -1
print(a) # [1, 99, 3, 4, 5] โ unchanged
# Check if view
print(b.base is a) # True โ b is a view of a
print(c.base is None) # True โ c owns its data
# View with dtype reinterpretation
x = np.array([1.0, 2.0], dtype=np.float64)
y = x.view(np.int64) # Reinterpret bytes as int64
๐ Behind the Question: Memory model and mutation safety. โ The Red Flags: Not knowing whether a slice is a view or copy, leading to unexpected mutations.
Q176: How do you use np.einsum() for complex tensor operations?โ
Technical Answer:
einsum() uses Einstein summation notation to express arbitrary tensor contractions, transpositions, traces, and element-wise operations in a single call. It's often faster than chaining multiple NumPy functions and can auto-optimize computation order.
Code/Syntax Justification:
A = np.random.rand(100, 200)
B = np.random.rand(200, 300)
# Matrix multiplication: C_ij = sum_k A_ik * B_kj
C = np.einsum("ik,kj->ij", A, B) # Equivalent to A @ B
# Batch matrix multiply: (batch, m, k) ร (batch, k, n) โ (batch, m, n)
X = np.random.rand(32, 64, 128)
Y = np.random.rand(32, 128, 256)
Z = np.einsum("bmk,bkn->bmn", X, Y)
# Trace: sum of diagonal
trace = np.einsum("ii->", A[:100, :100])
# Outer product
outer = np.einsum("i,j->ij", np.arange(5), np.arange(5))
# Optimized contraction order
result = np.einsum("ij,jk,kl->il", A, B[:, :50], np.random.rand(50, 10), optimize=True)
๐ Behind the Question: Efficient tensor algebra without intermediate arrays. โ The Red Flags: Chaining multiple matmul/transpose/sum calls when einsum would be clearer and faster.
Q177: What are structured arrays in NumPy and when should you use them?โ
Technical Answer: Structured arrays store heterogeneous data in a contiguous buffer with named fields, like a lightweight C struct array. They offer better cache locality than Python objects and integrate with memory-mapped files, but Pandas DataFrames are usually preferred for analytical work.
Code/Syntax Justification:
# Define structured dtype
dt = np.dtype([
("name", "U20"),
("age", "i4"),
("salary", "f8"),
])
employees = np.array([
("Alice", 30, 95000.0),
("Bob", 25, 82000.0),
("Charlie", 35, 110000.0),
], dtype=dt)
# Field access
print(employees["name"]) # ['Alice' 'Bob' 'Charlie']
print(employees["salary"].mean()) # 95666.67
# Record access
print(employees[0]) # ('Alice', 30, 95000.)
# Memory-mapped structured array
mmap = np.memmap("data.bin", dtype=dt, mode="w+", shape=(1000,))
๐ Behind the Question: Low-level data representation for performance-critical applications. โ The Red Flags: Using structured arrays when Pandas or dataclasses would be clearer and more maintainable.
Q178: How does np.vectorize() differ from true vectorization?โ
Technical Answer:
np.vectorize() wraps a scalar function to accept arrays but does NOT provide true vectorization โ it still loops in Python. It's a convenience for broadcasting, not a performance tool. For real speedups, use ufuncs, Numba @jit, or pure NumPy operations.
Code/Syntax Justification:
import numpy as np
from numba import vectorize as numba_vectorize
# np.vectorize โ convenient but slow (still Python loop)
@np.vectorize
def slow_classify(x):
if x > 0.8: return "high"
elif x > 0.5: return "medium"
else: return "low"
# True vectorization โ fast
def fast_classify(arr):
return np.where(arr > 0.8, "high", np.where(arr > 0.5, "medium", "low"))
# Numba vectorize โ compiles to machine code ufunc
@numba_vectorize(["float64(float64, float64)"])
def fast_hypotenuse(x, y):
return np.sqrt(x**2 + y**2)
๐ Behind the Question: Understanding the vectorization illusion vs. reality.
โ The Red Flags: Using np.vectorize() and assuming it runs at C speed.
Q179: What is the difference between np.concatenate(), np.stack(), np.vstack(), and np.hstack()?โ
Technical Answer:
concatenate() joins along an existing axis. stack() creates a new axis and joins along it. vstack() stacks vertically (along axis 0). hstack() stacks horizontally (along axis 1). All are thin wrappers around concatenate() with different axis defaults.
Code/Syntax Justification:
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
np.concatenate([a, b]) # [1, 2, 3, 4, 5, 6] โ join along axis 0
np.stack([a, b]) # [[1, 2, 3], [4, 5, 6]] โ new axis 0
np.stack([a, b], axis=1) # [[1, 4], [2, 5], [3, 6]] โ new axis 1
np.vstack([a, b]) # [[1, 2, 3], [4, 5, 6]] โ vertical
np.hstack([a, b]) # [1, 2, 3, 4, 5, 6] โ horizontal
# 2D arrays
A = np.ones((3, 4))
B = np.zeros((3, 4))
np.vstack([A, B]).shape # (6, 4)
np.hstack([A, B]).shape # (3, 8)
np.stack([A, B]).shape # (2, 3, 4) โ new batch dimension
๐ Behind the Question: Array assembly for data pipeline construction. โ The Red Flags: Confusing concatenate (existing axis) with stack (new axis).
Q180: How do you use np.where() for conditional element selection?โ
Technical Answer:
np.where(condition, x, y) returns elements from x where condition is True, elements from y where False. With only a condition argument, it returns indices of True elements (like np.nonzero()). It's fully vectorized and supports broadcasting.
Code/Syntax Justification:
arr = np.array([1, -2, 3, -4, 5])
# Conditional replacement
result = np.where(arr > 0, arr, 0) # [1, 0, 3, 0, 5]
# Multi-array conditional
a = np.array([10, 20, 30])
b = np.array([100, 200, 300])
mask = np.array([True, False, True])
result = np.where(mask, a, b) # [10, 200, 30]
# Index form โ returns tuple of arrays
indices = np.where(arr > 0) # (array([0, 2, 4]),)
positive_values = arr[np.where(arr > 0)] # [1, 3, 5]
# 2D conditional
matrix = np.random.randn(5, 5)
clipped = np.where(matrix > 0, matrix, 0) # ReLU-like operation
๐ Behind the Question: Vectorized conditional logic โ foundation of array programming.
โ The Red Flags: Using Python if/else in loops instead of np.where().
Q181: What is np.frompyfunc() and how does it create custom ufuncs?โ
Technical Answer:
np.frompyfunc() converts a Python function into a ufunc that supports broadcasting and array input. Unlike np.vectorize(), the output dtype is always object. For typed ufuncs with real performance, use Numba's @vectorize decorator.
Code/Syntax Justification:
# Create ufunc from Python function
def clamp(x, lo, hi):
return min(max(x, lo), hi)
clamp_ufunc = np.frompyfunc(clamp, 3, 1) # 3 inputs, 1 output
arr = np.array([1, 5, 10, 15, 20])
result = clamp_ufunc(arr, 5, 15) # array([5, 5, 10, 15, 15], dtype=object)
result = result.astype(float) # Convert from object dtype
# Real performance ufunc with Numba
from numba import vectorize
@vectorize(["float64(float64, float64, float64)"], nopython=True)
def fast_clamp(x, lo, hi):
return min(max(x, lo), hi)
result = fast_clamp(arr.astype(float), 5.0, 15.0) # float64 output, C speed
๐ Behind the Question: Custom array operations with proper broadcasting support. โ The Red Flags: Not knowing the difference between frompyfunc (object dtype) and Numba vectorize (typed, fast).
Q182: How does np.linalg handle common linear algebra operations?โ
Technical Answer:
np.linalg provides matrix decomposition (SVD, QR, Cholesky, eigendecomposition), solving linear systems, matrix inversion, determinants, norms, and pseudo-inverse. For production use, these call optimized BLAS/LAPACK routines.
Code/Syntax Justification:
A = np.random.rand(100, 100)
b = np.random.rand(100)
# Solve Ax = b (prefer over inv(A) @ b)
x = np.linalg.solve(A, b)
# SVD decomposition
U, s, Vt = np.linalg.svd(A, full_matrices=False)
# Eigenvalues
eigenvalues, eigenvectors = np.linalg.eigh(A @ A.T) # Symmetric matrix
# Matrix norm
frobenius = np.linalg.norm(A, "fro")
spectral = np.linalg.norm(A, 2)
# Pseudo-inverse for underdetermined/overdetermined systems
A_pinv = np.linalg.pinv(A[:50, :]) # 50ร100 โ 100ร50
# Condition number (numerical stability indicator)
cond = np.linalg.cond(A)
๐ Behind the Question: Numerical linear algebra for ML and scientific computing.
โ The Red Flags: Computing inv(A) @ b instead of solve(A, b), which is numerically unstable and slower.
Q183: What is np.random.Generator and why is it preferred over legacy np.random?โ
Technical Answer:
np.random.Generator (introduced in NumPy 1.17) uses the PCG64 bit generator by default, which has better statistical properties and performance than the legacy Mersenne Twister. It provides explicit state management, no global state pollution, and supports multiple independent streams.
Code/Syntax Justification:
# Modern API (preferred)
rng = np.random.default_rng(seed=42)
samples = rng.standard_normal(1000)
integers = rng.integers(0, 100, size=50)
shuffled = rng.permutation(np.arange(100))
choice = rng.choice(["a", "b", "c"], size=10, p=[0.5, 0.3, 0.2])
# Reproducible parallel streams with SeedSequence
from numpy.random import SeedSequence, PCG64, Generator
ss = SeedSequence(42)
child_seeds = ss.spawn(4) # 4 independent streams
rngs = [Generator(PCG64(s)) for s in child_seeds]
# Legacy API (avoid in new code)
np.random.seed(42) # Global state โ bad for parallel/testing
np.random.randn(100)
๐ Behind the Question: Modern reproducible random number generation.
โ The Red Flags: Using np.random.seed() global state in production or test code.
Q184: How does np.memmap enable out-of-core array processing?โ
Technical Answer:
np.memmap maps a file on disk as a virtual array, loading pages into memory on demand via the OS virtual memory system. This allows processing arrays larger than RAM, with automatic disk I/O managed by the OS page cache.
Code/Syntax Justification:
# Create a memory-mapped file
fp = np.memmap("large_data.dat", dtype="float64", mode="w+", shape=(1_000_000, 100))
# Write data in chunks
for i in range(0, 1_000_000, 10_000):
fp[i:i+10_000] = np.random.rand(10_000, 100)
fp.flush() # Ensure data is written to disk
# Read memory-mapped file (lazy loading)
data = np.memmap("large_data.dat", dtype="float64", mode="r", shape=(1_000_000, 100))
# Operations work transparently
mean_per_col = data.mean(axis=0) # Only loads needed pages
subset = data[500_000:500_100] # Loads only 100 rows from disk
๐ Behind the Question: Processing datasets larger than available RAM. โ The Red Flags: Loading entire large arrays into memory when only a subset is needed.
Q185: What are np.ufunc reduce, accumulate, and outer methods?โ
Technical Answer:
Every ufunc has .reduce() (fold along axis), .accumulate() (running fold), .outer() (all pairwise combinations), and .at() (unbuffered in-place operation). These eliminate explicit loops for common patterns.
Code/Syntax Justification:
a = np.array([1, 2, 3, 4, 5])
# reduce: fold with the ufunc
np.add.reduce(a) # 15 (sum)
np.multiply.reduce(a) # 120 (product)
# accumulate: running fold
np.add.accumulate(a) # [1, 3, 6, 10, 15] (cumsum)
np.multiply.accumulate(a) # [1, 2, 6, 24, 120] (cumprod)
np.maximum.accumulate(a) # Running maximum
# outer: all pairwise combinations
np.multiply.outer(a, a) # 5ร5 multiplication table
# at: unbuffered in-place (useful for scatter operations)
counts = np.zeros(10, dtype=int)
indices = np.array([1, 2, 2, 3, 3, 3])
np.add.at(counts, indices, 1) # counts = [0, 1, 2, 3, 0, 0, 0, 0, 0, 0]
๐ Behind the Question: Leveraging ufunc methods for loop-free computation. โ The Red Flags: Reimplementing reduce/accumulate patterns with explicit loops.
Q186: How do you use advanced integer and boolean indexing in NumPy?โ
Technical Answer: Integer array indexing selects arbitrary elements by index arrays. Boolean indexing selects elements where a mask is True. Both return copies, not views. Combining them enables complex selection patterns like top-k, conditional sampling, and scatter/gather.
Code/Syntax Justification:
data = np.random.rand(1000, 50)
# Integer indexing โ arbitrary row/column selection
row_idx = np.array([0, 5, 10, 999])
col_idx = np.array([0, 3, 7, 49])
selected = data[row_idx, col_idx] # 4 specific elements
selected_rows = data[row_idx] # 4 full rows
selected_block = data[np.ix_(row_idx, col_idx)] # 4ร4 submatrix
# Boolean indexing
mask = data[:, 0] > 0.5
filtered = data[mask] # Rows where first column > 0.5
# Top-k per column
top_k = 5
top_indices = np.argpartition(data, -top_k, axis=0)[-top_k:]
top_values = np.take_along_axis(data, top_indices, axis=0)
๐ Behind the Question: Advanced data selection patterns in array programming. โ The Red Flags: Not knowing that integer/boolean indexing returns copies while slicing returns views.
Q187: What is np.stride_tricks.as_strided() and when is it useful?โ
Technical Answer:
as_strided() creates a view with custom shape and strides, enabling zero-copy sliding windows, patches, and tiling. It's powerful but dangerous โ incorrect strides cause memory corruption. Use np.lib.stride_tricks.sliding_window_view() (NumPy 1.20+) for the safe version.
Code/Syntax Justification:
from numpy.lib.stride_tricks import sliding_window_view
# Safe sliding window (NumPy 1.20+)
a = np.arange(10)
windows = sliding_window_view(a, window_shape=3)
# [[0, 1, 2], [1, 2, 3], [2, 3, 4], ..., [7, 8, 9]]
# 2D sliding window (image patches)
image = np.random.rand(100, 100)
patches = sliding_window_view(image, (8, 8))
# Shape: (93, 93, 8, 8) โ all 8ร8 patches
# Moving average without loops
ma = sliding_window_view(a, 3).mean(axis=1)
๐ Behind the Question: Zero-copy array transformations for signal processing and convolution.
โ The Red Flags: Using as_strided() directly without understanding memory safety implications.
Q188: How do you use np.searchsorted() for efficient bin assignment?โ
Technical Answer:
searchsorted() finds insertion points in a sorted array using binary search (O(log n)). It's the backbone of pd.cut(), histogram binning, and quantile mapping. Supports side="left" (lower bound) and side="right" (upper bound).
Code/Syntax Justification:
bins = np.array([0, 10, 20, 50, 100])
values = np.array([5, 15, 25, 75, 100])
# Find bin indices
indices = np.searchsorted(bins, values, side="right") - 1
# [0, 1, 2, 3, 4] โ bin index for each value
# Efficient percentile lookup
percentiles = np.percentile(data, np.arange(0, 101, 10))
value_percentile = np.searchsorted(percentiles, new_values) * 10
# Merge sorted arrays
merged_indices = np.searchsorted(sorted_a, sorted_b)
๐ Behind the Question: O(log n) lookup patterns for numerical data. โ The Red Flags: Using linear search for bin assignment on sorted data.
Q189: What is np.fft and how do you use it for frequency analysis?โ
Technical Answer:
np.fft provides Fast Fourier Transform routines for converting signals between time domain and frequency domain. Use fft() for complex signals, rfft() for real signals (half the output), and fftfreq() for frequency axis labels.
Code/Syntax Justification:
# Generate signal: 50 Hz + 120 Hz components
sample_rate = 1000
t = np.arange(0, 1, 1/sample_rate)
signal = np.sin(2 * np.pi * 50 * t) + 0.5 * np.sin(2 * np.pi * 120 * t)
# FFT
spectrum = np.fft.rfft(signal)
freqs = np.fft.rfftfreq(len(signal), d=1/sample_rate)
magnitudes = np.abs(spectrum) / len(signal)
# Find dominant frequency
dominant_freq = freqs[np.argmax(magnitudes[1:]) + 1]
# Inverse FFT (reconstruction)
reconstructed = np.fft.irfft(spectrum)
๐ Behind the Question: Signal processing fundamentals for time-series and audio. โ The Red Flags: Not normalizing FFT output or confusing one-sided vs. two-sided spectrum.
Q190: How do you use np.apply_along_axis() and when should you avoid it?โ
Technical Answer:
apply_along_axis() applies a function to 1-D slices along a given axis, iterating in Python. It's convenient for functions that don't have native axis support but is slow for large arrays. Prefer native axis parameters or Numba @guvectorize for performance.
Code/Syntax Justification:
# Slow but convenient
def custom_normalize(x):
return (x - x.mean()) / (x.std() + 1e-8)
# apply_along_axis โ Python loop over axis
result = np.apply_along_axis(custom_normalize, axis=0, arr=data)
# Fast native equivalent
mean = data.mean(axis=0, keepdims=True)
std = data.std(axis=0, keepdims=True)
result_fast = (data - mean) / (std + 1e-8)
๐ Behind the Question: Understanding when convenience functions hide Python loops.
โ The Red Flags: Using apply_along_axis() for operations that have vectorized equivalents.
Q191: What is the difference between np.allclose() and np.array_equal()?โ
Technical Answer:
array_equal() tests exact element-wise equality. allclose() tests approximate equality within absolute and relative tolerances, which is necessary for floating-point comparisons. Use allclose() for numerical results and array_equal() for integer/boolean arrays.
Code/Syntax Justification:
a = np.array([1.0, 2.0, 3.0])
b = np.array([1.0, 2.0000001, 3.0])
np.array_equal(a, b) # False โ exact comparison
np.allclose(a, b, atol=1e-6) # True โ within tolerance
# Custom tolerances
np.allclose(a, b, rtol=1e-5, atol=1e-8) # rtol = relative, atol = absolute
# Testing with NaN
c = np.array([1.0, np.nan, 3.0])
np.allclose(a, c) # False (NaN โ NaN)
np.allclose(a, c, equal_nan=True) # Still False (values differ)
np.allclose(c, c, equal_nan=True) # True (NaN == NaN with flag)
๐ Behind the Question: Floating-point comparison correctness in tests and validation.
โ The Red Flags: Using == or array_equal() for floating-point comparison.
Q192: How do you create custom dtypes in NumPy?โ
Technical Answer:
Custom dtypes can represent fixed-size records, sub-arrays, and byte-level layouts. Define with a list of (name, type) tuples. Sub-array dtypes embed arrays within each element. Useful for interop with C structs and binary file formats.
Code/Syntax Justification:
# Record dtype with sub-array
dt = np.dtype([
("position", np.float64, (3,)), # 3D coordinate
("velocity", np.float64, (3,)), # 3D velocity
("mass", np.float64),
("label", "U10"),
])
particles = np.zeros(1000, dtype=dt)
particles["position"] = np.random.randn(1000, 3)
particles["mass"] = np.random.uniform(0.1, 10.0, 1000)
# Access sub-arrays
positions = particles["position"] # Shape: (1000, 3)
x_coords = particles["position"][:, 0]
# Byte-aligned dtype for binary interop
dt_packed = np.dtype([("x", "<f4"), ("y", "<f4"), ("z", "<f4")], align=True)
๐ Behind the Question: Low-level data layout control for scientific and systems programming. โ The Red Flags: Not understanding memory layout implications for performance and interoperability.
Q193: What is np.ma (masked arrays) and when do you use it?โ
Technical Answer: Masked arrays attach a boolean mask to data, marking elements as invalid without removing them. Operations automatically skip masked elements. Useful for sensor data with dropouts, satellite imagery with clouds, and any dataset with structured missing values.
Code/Syntax Justification:
data = np.array([1.0, 2.0, -999.0, 4.0, -999.0, 6.0])
# Create masked array
masked = np.ma.masked_equal(data, -999.0)
print(masked.mean()) # 3.25 (only valid elements)
print(masked.count()) # 4
# Mask by condition
masked2 = np.ma.masked_where(data < 0, data)
# Operations propagate mask
result = masked * 2 + 1 # Masked elements stay masked
# Fill masked values for output
filled = masked.filled(fill_value=0.0) # Replace masked with 0
๐ Behind the Question: Handling irregular missing data without dropping array elements. โ The Red Flags: Using NaN for all missing data instead of considering masked arrays for complex masking patterns.
Q194: How do you use np.piecewise() for multi-condition functions?โ
Technical Answer:
np.piecewise() evaluates different functions for different conditions, similar to a mathematical piecewise function. Each condition selects elements, and the corresponding function is applied only to those elements. It's fully vectorized.
Code/Syntax Justification:
x = np.linspace(-5, 5, 1000)
# Piecewise function: different formulas for different ranges
y = np.piecewise(x,
[x < -2, (x >= -2) & (x < 2), x >= 2],
[lambda x: -x - 2, # x < -2
lambda x: x ** 2, # -2 <= x < 2
lambda x: x + 2] # x >= 2
)
# Tax bracket calculation
income = np.array([20000, 50000, 100000, 250000])
tax = np.piecewise(income.astype(float),
[income <= 40000, (income > 40000) & (income <= 85000), income > 85000],
[lambda i: i * 0.12, lambda i: 4800 + (i - 40000) * 0.22, lambda i: 14700 + (i - 85000) * 0.32]
)
๐ Behind the Question: Vectorized multi-branch computation.
โ The Red Flags: Using loops or nested np.where() for piecewise functions.
Q195: What is np.seterr() and how do you handle floating-point errors?โ
Technical Answer:
np.seterr() controls how NumPy handles floating-point errors: divide by zero, overflow, underflow, and invalid operations. Options are ignore, warn, raise, or print. Use np.errstate() context manager for localized settings.
Code/Syntax Justification:
# Global setting
old_settings = np.seterr(divide="ignore", invalid="ignore")
# Context manager (preferred โ localized)
with np.errstate(divide="ignore", invalid="ignore"):
result = np.log(0) # -inf, no warning
result2 = 0.0 / 0.0 # nan, no warning
# Raise on errors for debugging
with np.errstate(all="raise"):
try:
result = np.sqrt(-1.0)
except FloatingPointError:
print("Caught invalid operation")
# Check for special values
data = np.array([1.0, np.nan, np.inf, -np.inf, 0.0])
print(np.isfinite(data)) # [True, False, False, False, True]
print(np.isnan(data)) # [False, True, False, False, False]
๐ Behind the Question: Numerical stability and error handling in scientific code. โ The Red Flags: Globally ignoring all floating-point errors without understanding the consequences.
Q196: How do you use np.polynomial for polynomial fitting and evaluation?โ
Technical Answer:
np.polynomial provides classes for Chebyshev, Legendre, Hermite, and power-series polynomials with stable fitting via least squares. Prefer np.polynomial.polynomial.Polynomial.fit() over the legacy np.polyfit() which uses the monomial basis (numerically unstable for high degrees).
Code/Syntax Justification:
from numpy.polynomial import Polynomial
x = np.linspace(0, 10, 100)
y = 2 * x ** 2 - 3 * x + 1 + np.random.randn(100) * 5
# Modern API (preferred, numerically stable)
p = Polynomial.fit(x, y, deg=2)
y_pred = p(x)
print(p.convert().coef) # Coefficients in standard form
# Legacy API (avoid for high-degree fits)
coeffs = np.polyfit(x, y, 2)
y_pred_legacy = np.polyval(coeffs, x)
# Polynomial operations
p1 = Polynomial([1, 2, 3]) # 1 + 2x + 3xยฒ
p2 = Polynomial([0, 1]) # x
product = p1 * p2 # x + 2xยฒ + 3xยณ
derivative = p1.deriv() # 2 + 6x
integral = p1.integ() # x + xยฒ + xยณ
๐ Behind the Question: Numerical polynomial computations and curve fitting.
โ The Red Flags: Using np.polyfit() with high degree polynomials (numerically unstable).
Q197: How does np.tensordot() generalize matrix multiplication?โ
Technical Answer:
tensordot() contracts (sums over) specified axes of two tensors. With axes=1 it's a dot product, with axes=2 it's a double contraction. It generalizes matrix multiplication to higher-dimensional arrays.
Code/Syntax Justification:
A = np.random.rand(3, 4, 5)
B = np.random.rand(5, 4, 6)
# Contract last axis of A with first axis of B (like batched matmul)
C = np.tensordot(A, B, axes=([2], [0])) # Shape: (3, 4, 4, 6)
# Standard matrix multiplication
M1 = np.random.rand(3, 4)
M2 = np.random.rand(4, 5)
np.tensordot(M1, M2, axes=1) # Same as M1 @ M2
# Double contraction (Frobenius inner product)
np.tensordot(M1, M1, axes=2) # Same as np.sum(M1 * M1)
๐ Behind the Question: Tensor algebra for ML and physics simulations. โ The Red Flags: Not understanding axis contraction or confusing tensordot axes parameter.
Q198: What are the differences between C-order and Fortran-order memory layout?โ
Technical Answer: C-order (row-major) stores rows contiguously; Fortran-order (column-major) stores columns contiguously. Operations that iterate along the contiguous dimension are cache-friendly and faster. NumPy defaults to C-order. BLAS/LAPACK expect Fortran-order.
Code/Syntax Justification:
# C-order (default): rows are contiguous
c_arr = np.array([[1, 2, 3], [4, 5, 6]], order="C")
print(c_arr.strides) # (24, 8) โ 24 bytes between rows, 8 between columns
# Fortran-order: columns are contiguous
f_arr = np.array([[1, 2, 3], [4, 5, 6]], order="F")
print(f_arr.strides) # (8, 16) โ 8 bytes between rows, 16 between columns
# Performance impact: iterate along contiguous axis
data = np.random.rand(10000, 10000)
# Fast: sum along rows (axis=1) for C-order
row_sums = data.sum(axis=1)
# Check and convert order
print(data.flags["C_CONTIGUOUS"]) # True
data_f = np.asfortranarray(data) # Convert for LAPACK calls
print(data_f.flags["F_CONTIGUOUS"]) # True
๐ Behind the Question: Memory layout impact on computation performance. โ The Red Flags: Ignoring memory order when interfacing with BLAS/LAPACK or iterating along the wrong axis.
Q199: How do you use np.char for vectorized string operations on arrays?โ
Technical Answer:
np.char provides vectorized string operations on character arrays (dtype U or S). Functions include add, upper, lower, strip, startswith, find, replace, and count. Useful for NumPy-only workflows without Pandas.
Code/Syntax Justification:
names = np.array(["Alice Smith", "Bob Jones", "Charlie Brown"])
# Vectorized string operations
upper = np.char.upper(names)
contains = np.char.find(names, "Jones") # [-1, 4, -1]
first_names = np.char.partition(names, " ")[:, 0]
replaced = np.char.replace(names, "Smith", "Johnson")
# Comparison
starts_with_b = np.char.startswith(names, "B") # [False, True, False]
lengths = np.char.str_len(names) # [11, 9, 13]
# Join and split
joined = np.char.join("-", ["abc", "def"]) # ["a-b-c", "d-e-f"]
๐ Behind the Question: String processing in pure NumPy environments.
โ The Red Flags: Using Python loops for string operations on arrays when np.char is available.
Q200: How do you implement efficient batch operations with np.add.reduceat()?โ
Technical Answer:
reduceat() applies a ufunc reduction to contiguous segments of an array defined by index boundaries. It's much faster than looping over slices and avoids the overhead of np.split() + map().
Code/Syntax Justification:
data = np.arange(20)
# Sum segments: [0:5], [5:10], [10:15], [15:20]
segment_starts = np.array([0, 5, 10, 15])
segment_sums = np.add.reduceat(data, segment_starts)
# [10, 35, 60, 85]
# Variable-length segments
values = np.array([1, 2, 3, 10, 20, 100, 200, 300, 400])
breaks = np.array([0, 3, 5])
np.add.reduceat(values, breaks) # [6, 30, 1000]
# Max per segment
np.maximum.reduceat(values, breaks) # [3, 20, 400]
๐ Behind the Question: Segment-wise operations without Python-level looping.
โ The Red Flags: Using np.split() + list comprehension for segment operations.
Q201: What is np.testing and how do you write NumPy-aware tests?โ
Technical Answer:
np.testing provides assertion functions that handle floating-point tolerance, NaN, shape mismatches, and descriptive error messages. Essential for testing numerical code where exact equality is inappropriate.
Code/Syntax Justification:
import numpy.testing as npt
# Assert arrays are close (with tolerance)
npt.assert_allclose(actual, expected, rtol=1e-7, atol=1e-10)
# Assert exact equality (for integers)
npt.assert_array_equal(np.array([1, 2, 3]), np.array([1, 2, 3]))
# Assert shape
npt.assert_equal(result.shape, (100, 50))
# Assert raises
npt.assert_raises(ValueError, np.linalg.solve, singular_matrix, b)
# In pytest
def test_normalization():
data = np.random.rand(100, 10)
result = normalize(data)
npt.assert_allclose(result.mean(axis=0), 0.0, atol=1e-10)
npt.assert_allclose(result.std(axis=0), 1.0, atol=1e-10)
๐ Behind the Question: Testing numerical code correctly.
โ The Red Flags: Using Python assert with == for floating-point array comparisons.
Q202: How do you optimize NumPy operations with Numba JIT compilation?โ
Technical Answer:
Numba compiles NumPy-style Python to LLVM machine code with @jit(nopython=True). It supports a large subset of NumPy operations, parallel execution with prange, and GPU targeting. Ideal for loops that can't be easily vectorized.
Code/Syntax Justification:
from numba import jit, prange
import numpy as np
@jit(nopython=True)
def custom_rolling_max(arr, window):
n = len(arr)
result = np.empty(n)
result[:window-1] = np.nan
for i in range(window - 1, n):
result[i] = np.max(arr[i - window + 1:i + 1])
return result
# Parallel execution
@jit(nopython=True, parallel=True)
def pairwise_distances(X):
n = X.shape[0]
D = np.empty((n, n))
for i in prange(n):
for j in range(i, n):
d = 0.0
for k in range(X.shape[1]):
d += (X[i, k] - X[j, k]) ** 2
D[i, j] = D[j, i] = np.sqrt(d)
return D
๐ Behind the Question: Bridging the gap between Python convenience and C performance. โ The Red Flags: Not using Numba for computationally intensive loops that resist vectorization.
Q203: How do you profile NumPy code to identify bottlenecks?โ
Technical Answer:
Use %timeit for microbenchmarks, line_profiler for line-by-line timing, and memory_profiler for memory usage. Check for unnecessary copies with np.shares_memory() and monitor allocations with tracemalloc.
Code/Syntax Justification:
import tracemalloc
import numpy as np
# Timing
import timeit
setup = "import numpy as np; a = np.random.rand(1_000_000)"
t = timeit.timeit("np.sort(a.copy())", setup=setup, number=100)
# Memory tracking
tracemalloc.start()
a = np.random.rand(10_000_000)
b = a * 2 + 1
current, peak = tracemalloc.get_traced_memory()
print(f"Current: {current / 1e6:.1f} MB, Peak: {peak / 1e6:.1f} MB")
tracemalloc.stop()
# Check for shared memory (views vs copies)
a = np.arange(100)
b = a[10:20]
print(np.shares_memory(a, b)) # True โ b is a view
# In-place operations to reduce allocations
np.multiply(a, 2, out=a) # In-place, no temporary
๐ Behind the Question: Performance debugging methodology for numerical code. โ The Red Flags: Optimizing code without profiling or not checking for unnecessary array copies.
โก Section 7: Polars Deep Diveโ
Q204: What is Polars' lazy execution model and why does it outperform Pandas for large datasets?โ
Technical Answer: Polars' lazy API builds a logical query plan that is optimized before execution. The query optimizer applies predicate pushdown, projection pushdown, common subexpression elimination, and join reordering. This means Polars reads only the columns/rows needed, parallelizes across all cores automatically, and avoids materializing intermediate DataFrames.
Code/Syntax Justification:
import polars as pl
# Lazy: builds plan, executes nothing yet
result = (
pl.scan_parquet("orders/*.parquet")
.filter(pl.col("amount") > 100)
.select(["user_id", "amount", "created_at"])
.group_by("user_id")
.agg([
pl.col("amount").sum().alias("total"),
pl.col("amount").count().alias("order_count"),
pl.col("created_at").max().alias("last_order"),
])
.sort("total", descending=True)
)
# View the optimized plan
print(result.explain())
# Execute
df = result.collect()
# Streaming mode for datasets larger than RAM
df = result.collect(streaming=True)
๐ Behind the Question: Understanding lazy evaluation as the key differentiator from Pandas.
โ The Red Flags: Using eager .collect() after every operation instead of building a full lazy plan.
Q205: How do Polars expressions work and why are they central to the API?โ
Technical Answer: Expressions are the building blocks of all Polars operations. An expression describes a computation on columns without executing it. They compose with method chaining, support arbitrary nesting, and the engine parallelizes independent expressions automatically. This is fundamentally different from Pandas where operations mutate or create Series objects.
Code/Syntax Justification:
# Expressions are composable computation descriptions
expr_revenue = pl.col("price") * pl.col("quantity")
expr_discount = pl.when(pl.col("is_member")).then(0.9).otherwise(1.0)
expr_final = (expr_revenue * expr_discount).alias("final_price")
# Multiple independent expressions run in parallel
df = df.select([
pl.col("name"),
expr_final,
pl.col("category").n_unique().over("store").alias("category_diversity"),
pl.col("price").rank().alias("price_rank"),
])
# Expression reuse
high_value = pl.col("amount") > pl.col("amount").mean()
df_filtered = df.filter(high_value)
df_labeled = df.with_columns(high_value.alias("is_high_value"))
๐ Behind the Question: Polars' expression-based paradigm vs. Pandas' method-based paradigm. โ The Red Flags: Writing Polars code like Pandas โ column-by-column mutation instead of expression composition.
Q206: How does over() implement window functions in Polars?โ
Technical Answer:
over() partitions the DataFrame by specified columns and applies the expression within each partition, broadcasting the result back to the original frame shape. It replaces Pandas' groupby().transform() with cleaner syntax and automatic parallelization.
Code/Syntax Justification:
df = pl.DataFrame({
"dept": ["eng", "eng", "sales", "sales", "eng"],
"name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
"salary": [120000, 110000, 95000, 98000, 115000],
})
# Window function: rank within department
result = df.with_columns([
pl.col("salary").rank(descending=True).over("dept").alias("dept_rank"),
pl.col("salary").mean().over("dept").alias("dept_avg"),
(pl.col("salary") - pl.col("salary").mean().over("dept")).alias("vs_dept_avg"),
pl.col("salary").max().over("dept").alias("dept_max"),
])
# Multiple partition keys
result = df.with_columns(
pl.col("salary").rank().over(["dept", "level"]).alias("level_rank")
)
๐ Behind the Question: Window functions without verbose groupby-transform patterns.
โ The Red Flags: Using group_by().agg() + join to achieve what over() does natively.
Q207: How do you handle missing values in Polars compared to Pandas?โ
Technical Answer:
Polars uses null (not NaN) for missing values across all dtypes, backed by Arrow's validity bitmask. This means integers can have nulls without upcasting to float (unlike Pandas). Operations like fill_null(), drop_nulls(), interpolate(), and forward_fill() are expression-based.
Code/Syntax Justification:
df = pl.DataFrame({
"a": [1, None, 3, None, 5],
"b": [None, "x", None, "y", "z"],
})
# Fill with strategies
result = df.with_columns([
pl.col("a").fill_null(strategy="forward").alias("a_ffill"),
pl.col("a").fill_null(pl.col("a").mean()).alias("a_mean_fill"),
pl.col("b").fill_null(pl.lit("unknown")).alias("b_filled"),
pl.col("a").interpolate().alias("a_interpolated"),
])
# Null counting
null_counts = df.null_count()
# Coalesce: first non-null across columns
df.with_columns(
pl.coalesce(["primary_email", "secondary_email", "backup_email"]).alias("email")
)
๐ Behind the Question: Proper null handling without the NaN/None confusion of Pandas.
โ The Red Flags: Confusing Polars null with Pandas NaN or not using fill_null(strategy=...).
Q208: What is struct dtype in Polars and how does it enable nested data?โ
Technical Answer:
Polars Struct type holds named fields of different types within a single column, similar to a JSON object or named tuple. It enables nested data without separate columns, is fully supported in expressions, and can be unnested back to flat columns.
Code/Syntax Justification:
# Create struct column
df = pl.DataFrame({
"name": ["Alice", "Bob"],
"age": [30, 25],
"city": ["NYC", "LA"],
})
# Pack columns into struct
df = df.select(
pl.struct(["name", "age"]).alias("person"),
pl.col("city"),
)
# Access struct fields
df.with_columns(
pl.col("person").struct.field("name").alias("person_name"),
pl.col("person").struct.field("age").alias("person_age"),
)
# Unnest back to flat columns
df.unnest("person")
# Struct in aggregation
stats = df.group_by("category").agg(
pl.struct([
pl.col("price").mean().alias("avg"),
pl.col("price").std().alias("std"),
pl.col("price").count().alias("n"),
]).alias("price_stats")
)
๐ Behind the Question: Nested data modeling in columnar DataFrames. โ The Red Flags: Flattening all nested data into separate columns when struct would preserve structure.
Q209: How does Polars handle string operations differently from Pandas?โ
Technical Answer:
Polars uses Apache Arrow's UTF-8 string representation (not Python objects), making string operations zero-copy where possible and cache-friendly. The .str namespace provides vectorized operations that run in Rust at native speed, often 5-50ร faster than Pandas.
Code/Syntax Justification:
df = pl.DataFrame({"text": ["Hello World", "foo bar BAZ", "123-abc"]})
result = df.with_columns([
pl.col("text").str.to_lowercase().alias("lower"),
pl.col("text").str.contains("(?i)hello").alias("has_hello"),
pl.col("text").str.extract(r"(\d+)", 1).alias("numbers"),
pl.col("text").str.split(" ").alias("words"),
pl.col("text").str.replace_all(r"[^a-zA-Z]", "").alias("alpha_only"),
pl.col("text").str.len_chars().alias("char_count"),
pl.col("text").str.strip_chars().alias("stripped"),
])
# Regex with named groups
df.with_columns(
pl.col("email").str.extract(r"^(?P<user>.+)@(?P<domain>.+)$", 1).alias("user")
)
๐ Behind the Question: Arrow-backed string processing performance.
โ The Red Flags: Converting Polars strings to Python for processing instead of using .str expressions.
Q210: How do you perform time-series operations in Polars?โ
Technical Answer:
Polars has first-class temporal support with group_by_dynamic() for time-based windowing, rolling() for rolling aggregations anchored to time, and efficient date/time extraction expressions. All temporal operations leverage Arrow's temporal types.
Code/Syntax Justification:
df = pl.DataFrame({
"timestamp": pl.date_range(
start=datetime(2024, 1, 1),
end=datetime(2024, 12, 31),
interval="1h",
eager=True,
),
"value": np.random.rand(8760),
})
# Dynamic time-based grouping
daily = df.group_by_dynamic("timestamp", every="1d").agg([
pl.col("value").mean().alias("daily_avg"),
pl.col("value").max().alias("daily_max"),
pl.col("value").std().alias("daily_std"),
])
# Rolling window by time
rolling = df.rolling("timestamp", period="7d").agg(
pl.col("value").mean().alias("7d_avg")
)
# Temporal extraction
df.with_columns([
pl.col("timestamp").dt.month().alias("month"),
pl.col("timestamp").dt.weekday().alias("dow"),
pl.col("timestamp").dt.hour().alias("hour"),
pl.col("timestamp").dt.quarter().alias("quarter"),
])
๐ Behind the Question: Time-series analytics in a columnar engine. โ The Red Flags: Converting timestamps to strings for grouping instead of using temporal expressions.
Q211: How does Polars handle joins and what join strategies are available?โ
Technical Answer:
Polars supports inner, left, right, outer, cross, semi, anti, and asof joins. The engine automatically selects hash join or sort-merge join based on data characteristics. join_asof() handles inexact temporal joins. All joins can run in parallel on multiple cores.
Code/Syntax Justification:
orders = pl.DataFrame({
"order_id": [1, 2, 3],
"user_id": [10, 20, 10],
"amount": [100, 200, 150],
})
users = pl.DataFrame({
"user_id": [10, 20, 30],
"name": ["Alice", "Bob", "Charlie"],
})
# Standard joins
inner = orders.join(users, on="user_id", how="inner")
left = orders.join(users, on="user_id", how="left")
# Semi join: keep rows in left that have a match in right
active_users = users.join(orders, on="user_id", how="semi")
# Anti join: keep rows in left that have NO match in right
inactive_users = users.join(orders, on="user_id", how="anti")
# Asof join: nearest match
trades.join_asof(quotes, on="timestamp", by="ticker", strategy="backward")
# Suffix control for overlapping column names
result = orders.join(users, on="user_id", suffix="_user")
๐ Behind the Question: Join semantics and strategy selection. โ The Red Flags: Not knowing semi/anti joins or using filter + isin instead.
Q212: What is Polars' map_elements() vs map_batches() and when to use each?โ
Technical Answer:
map_elements() applies a Python function element-by-element (slow, like Pandas apply()). map_batches() receives the entire column as a Series and should return a Series (can use NumPy inside). Always prefer native expressions; use map_batches() for vectorized external libraries; use map_elements() only as last resort.
Code/Syntax Justification:
# AVOID: map_elements (Python loop, slow)
df.with_columns(
pl.col("text").map_elements(lambda x: x.upper(), return_dtype=pl.Utf8).alias("upper")
)
# BETTER: map_batches (vectorized, medium speed)
df.with_columns(
pl.col("values").map_batches(
lambda s: pl.Series(np.log1p(s.to_numpy())),
return_dtype=pl.Float64,
).alias("log_values")
)
# BEST: native expression (Rust speed)
df.with_columns(
pl.col("text").str.to_uppercase().alias("upper")
)
# When map_elements is unavoidable: external API calls
df.with_columns(
pl.col("address").map_elements(
geocode_address, # External API โ can't vectorize
return_dtype=pl.Struct({"lat": pl.Float64, "lng": pl.Float64}),
).alias("coords")
)
๐ Behind the Question: Understanding the performance hierarchy in Polars.
โ The Red Flags: Using map_elements() for operations that have native Polars equivalents.
Q213: How do you read and write Parquet files efficiently with Polars?โ
Technical Answer:
Polars uses Arrow's Parquet reader/writer with predicate pushdown and projection pushdown in lazy mode. scan_parquet() reads only needed columns and rows matching filters. It supports row-group-level filtering, hive partitioning, and glob patterns for partitioned datasets.
Code/Syntax Justification:
# Lazy scan with pushdown optimization
df = (
pl.scan_parquet("data/year=*/month=*/*.parquet", hive_partitioning=True)
.filter(pl.col("year") == 2024)
.select(["user_id", "amount", "year", "month"])
.collect()
)
# Write with compression and statistics
df.write_parquet(
"output.parquet",
compression="zstd",
compression_level=3,
statistics=True,
row_group_size=100_000,
)
# Streaming for files larger than RAM
result = (
pl.scan_parquet("huge_dataset.parquet")
.filter(pl.col("status") == "active")
.group_by("region")
.agg(pl.col("revenue").sum())
.collect(streaming=True)
)
# Write partitioned dataset
df.write_parquet(
"output/",
use_pyarrow=True,
pyarrow_options={"partition_cols": ["year", "month"]},
)
๐ Behind the Question: Optimal columnar file I/O for analytics. โ The Red Flags: Reading entire Parquet files eagerly when lazy scan with pushdown is available.
Q214: How do you implement custom aggregations in Polars?โ
Technical Answer:
Use expression composition for most aggregations. For complex custom logic, use map_groups() which receives each group as a DataFrame. Polars supports multiple simultaneous aggregations, conditional aggregations, and struct-based compound results.
Code/Syntax Justification:
# Multiple aggregations
stats = df.group_by("category").agg([
pl.col("price").mean().alias("avg_price"),
pl.col("price").quantile(0.95).alias("p95_price"),
pl.col("price").filter(pl.col("status") == "sold").sum().alias("sold_revenue"),
pl.col("user_id").n_unique().alias("unique_users"),
(pl.col("price") * pl.col("quantity")).sum().alias("total_gmv"),
])
# Conditional aggregation
df.group_by("region").agg([
pl.col("amount").filter(pl.col("type") == "credit").sum().alias("credits"),
pl.col("amount").filter(pl.col("type") == "debit").sum().alias("debits"),
])
# Custom group function (last resort)
def custom_metric(group_df: pl.DataFrame) -> pl.DataFrame:
return group_df.select(
weighted_avg=(pl.col("value") * pl.col("weight")).sum() / pl.col("weight").sum()
)
result = df.group_by("category").map_groups(custom_metric)
๐ Behind the Question: Aggregation expressiveness and performance.
โ The Red Flags: Using map_groups() for aggregations expressible with native expressions.
Q215: How does Polars handle categorical data?โ
Technical Answer:
Polars Categorical type uses a global string cache for consistent encoding across DataFrames, enabling efficient joins and comparisons on categoricals. Enum type provides fixed-category sets with compile-time validation. Both are more memory-efficient than strings.
Code/Syntax Justification:
# Enable global string cache for cross-DataFrame consistency
with pl.StringCache():
df1 = pl.DataFrame({"status": ["active", "inactive"]}).cast({"status": pl.Categorical})
df2 = pl.DataFrame({"status": ["active", "pending"]}).cast({"status": pl.Categorical})
# Joining works because categories share global cache
joined = df1.join(df2, on="status")
# Enum: fixed category set (stricter than Categorical)
status_type = pl.Enum(["active", "inactive", "pending"])
df = pl.DataFrame({"status": ["active", "pending"]}, schema={"status": status_type})
# df with status "unknown" would raise an error
# Casting and ordering
df.with_columns(
pl.col("priority").cast(pl.Categorical).cat.set_ordering("physical")
)
๐ Behind the Question: Typed categorical semantics in a columnar engine. โ The Red Flags: Using raw strings for low-cardinality columns or not enabling StringCache for multi-DataFrame work.
Q216: How do you use when/then/otherwise for conditional logic in Polars?โ
Technical Answer:
when().then().otherwise() is Polars' SQL CASE-WHEN equivalent. It supports chaining multiple conditions, nested expressions, and null handling. Unlike np.where(), it's fully integrated into Polars' expression engine and benefits from query optimization.
Code/Syntax Justification:
df = pl.DataFrame({
"score": [95, 72, 58, 88, 45],
"attendance": [0.95, 0.80, 0.60, 0.92, 0.70],
})
# Multi-condition (chained when/then)
result = df.with_columns(
pl.when(pl.col("score") >= 90)
.then(pl.lit("A"))
.when(pl.col("score") >= 80)
.then(pl.lit("B"))
.when(pl.col("score") >= 70)
.then(pl.lit("C"))
.when(pl.col("score") >= 60)
.then(pl.lit("D"))
.otherwise(pl.lit("F"))
.alias("grade")
)
# Complex conditional with multiple columns
df.with_columns(
pl.when((pl.col("score") >= 80) & (pl.col("attendance") >= 0.9))
.then(pl.lit("honors"))
.when(pl.col("score") >= 60)
.then(pl.lit("pass"))
.otherwise(pl.lit("fail"))
.alias("status")
)
๐ Behind the Question: Declarative conditional logic in expression-based APIs.
โ The Red Flags: Using map_elements() with Python if/else for conditional column creation.
Q217: How do you perform SQL queries on Polars DataFrames?โ
Technical Answer:
Polars includes a built-in SQL engine via pl.SQLContext that lets you write SQL queries directly against DataFrames. It supports SELECT, JOIN, WHERE, GROUP BY, ORDER BY, window functions, and CTEs. The SQL is translated to Polars' optimized query plan.
Code/Syntax Justification:
ctx = pl.SQLContext(
orders=orders_df,
users=users_df,
products=products_df,
)
result = ctx.execute("""
SELECT
u.name,
COUNT(o.order_id) AS order_count,
SUM(o.amount) AS total_spent,
AVG(o.amount) AS avg_order,
RANK() OVER (ORDER BY SUM(o.amount) DESC) AS spend_rank
FROM orders o
JOIN users u ON o.user_id = u.user_id
WHERE o.created_at >= '2024-01-01'
GROUP BY u.name
HAVING COUNT(o.order_id) >= 3
ORDER BY total_spent DESC
""").collect()
# Register lazy frames for predicate pushdown
ctx = pl.SQLContext(
events=pl.scan_parquet("events/*.parquet"),
)
๐ Behind the Question: SQL interoperability for teams transitioning from SQL-based workflows. โ The Red Flags: Not knowing Polars has SQL support or converting to Pandas just to use pandasql.
Q218: What is sink_parquet() and how does it enable streaming writes?โ
Technical Answer:
sink_parquet() writes a lazy frame directly to a Parquet file in streaming mode, processing data in chunks without materializing the full result in memory. This is essential for ETL pipelines where both input and output exceed available RAM.
Code/Syntax Justification:
# Streaming ETL: read, transform, write โ never fully in memory
(
pl.scan_parquet("raw_data/*.parquet")
.filter(pl.col("quality_score") > 0.8)
.with_columns([
pl.col("timestamp").dt.truncate("1h").alias("hour"),
(pl.col("price") * pl.col("quantity")).alias("revenue"),
])
.group_by(["hour", "category"])
.agg([
pl.col("revenue").sum().alias("hourly_revenue"),
pl.col("order_id").n_unique().alias("order_count"),
])
.sink_parquet("output/hourly_stats.parquet")
)
# Streaming to CSV
(
pl.scan_csv("input.csv")
.filter(pl.col("status") == "active")
.sink_csv("output.csv")
)
๐ Behind the Question: End-to-end streaming data pipelines without RAM constraints.
โ The Red Flags: Calling .collect() then .write_parquet() for large datasets that don't fit in memory.
Q219: How do you handle list columns in Polars?โ
Technical Answer:
Polars has first-class List dtype support with the .list namespace for operations like lengths(), get(), contains(), mean(), explode(), and eval(). List columns store variable-length arrays per row efficiently using Arrow's list arrays.
Code/Syntax Justification:
df = pl.DataFrame({
"user": ["Alice", "Bob", "Charlie"],
"scores": [[90, 85, 92], [78, 88], [95, 91, 87, 93]],
})
result = df.with_columns([
pl.col("scores").list.len().alias("num_scores"),
pl.col("scores").list.mean().alias("avg_score"),
pl.col("scores").list.max().alias("max_score"),
pl.col("scores").list.get(0).alias("first_score"),
pl.col("scores").list.contains(90).alias("has_90"),
pl.col("scores").list.sort(descending=True).alias("sorted_desc"),
pl.col("scores").list.slice(0, 2).alias("top_two"),
])
# Explode list to rows
exploded = df.explode("scores")
# List eval: apply expressions within each list
df.with_columns(
pl.col("scores").list.eval(pl.element().filter(pl.element() > 85)).alias("high_scores")
)
๐ Behind the Question: Variable-length data in columnar storage.
โ The Red Flags: Exploding list columns unnecessarily when .list operations suffice.
Q220: How do you do Polars-native data validation?โ
Technical Answer:
Use expressions to build validation rules that produce boolean columns or filter invalid rows. Combine with when/then/otherwise for tagging, filter() for removal, and schema enforcement via cast() with strict=True. For complex validation, chain expressions into a validation pipeline.
Code/Syntax Justification:
# Define validation expressions
validations = {
"email_valid": pl.col("email").str.contains(r"^[\w.+-]+@[\w-]+\.[\w.]+$"),
"age_valid": pl.col("age").is_between(0, 150),
"salary_positive": pl.col("salary") > 0,
"name_not_null": pl.col("name").is_not_null(),
}
# Apply all validations
df_validated = df.with_columns([
expr.alias(name) for name, expr in validations.items()
])
# Filter to valid rows only
all_valid = pl.all_horizontal([pl.col(name) for name in validations])
df_clean = df.filter(all_valid)
# Report violations
violations = df_validated.filter(~all_valid).select(
["id"] + [pl.col(name) for name in validations]
)
๐ Behind the Question: Data quality enforcement in Polars pipelines. โ The Red Flags: Validating data with Python loops instead of expression-based validation.
Q221: How does Polars integrate with scikit-learn and ML workflows?โ
Technical Answer:
Polars DataFrames can be converted to NumPy via .to_numpy() or to Arrow via .to_arrow() for ML library interop. Many ML libraries now accept Arrow natively. For feature engineering, Polars' expression engine is far faster than Pandas for creating features before model training.
Code/Syntax Justification:
import polars as pl
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
# Feature engineering in Polars (fast)
features = (
raw_df
.with_columns([
pl.col("amount").log1p().alias("log_amount"),
pl.col("amount").over("user_id").mean().alias("user_avg_amount"),
pl.col("timestamp").dt.hour().alias("hour"),
pl.col("category").cast(pl.Categorical),
])
.select(["log_amount", "user_avg_amount", "hour", "label"])
)
# Convert to NumPy for sklearn
X = features.select(pl.exclude("label")).to_numpy()
y = features["label"].to_numpy()
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
model = RandomForestClassifier().fit(X_train, y_train)
๐ Behind the Question: Bridging columnar DataFrames with array-based ML libraries. โ The Red Flags: Converting to Pandas as an intermediate step between Polars and scikit-learn.
Q222: How do you use pl.concat() and pl.align_frames() for DataFrame assembly?โ
Technical Answer:
pl.concat() stacks DataFrames vertically (default) or horizontally. align_frames() aligns multiple DataFrames on shared columns, filling missing values with null. Use how="diagonal" in concat for union-like behavior with different schemas.
Code/Syntax Justification:
# Vertical concatenation (append rows)
combined = pl.concat([df1, df2, df3])
# Horizontal concatenation (side-by-side)
wide = pl.concat([features, labels], how="horizontal")
# Diagonal: union with different schemas (fills missing with null)
jan = pl.DataFrame({"date": ["2024-01"], "sales": [100]})
feb = pl.DataFrame({"date": ["2024-02"], "sales": [120], "returns": [5]})
combined = pl.concat([jan, feb], how="diagonal")
# date, sales, returns (jan has null for returns)
# Align frames on common columns
aligned = pl.align_frames(df_2023, df_2024, on="product_id")
๐ Behind the Question: DataFrame assembly patterns for multi-source data.
โ The Red Flags: Not knowing how="diagonal" for heterogeneous schema concatenation.
Q223: What is pl.Config and how do you customize Polars behavior?โ
Technical Answer:
pl.Config controls display formatting, table rendering, string lengths, and verbose mode. Settings are thread-local and can be scoped with context managers. It also controls streaming chunk sizes and memory limits.
Code/Syntax Justification:
# Display configuration
pl.Config.set_tbl_rows(50)
pl.Config.set_tbl_cols(20)
pl.Config.set_fmt_str_lengths(100)
pl.Config.set_tbl_width_chars(200)
# Context manager for temporary settings
with pl.Config(tbl_rows=100, fmt_float="full"):
print(df)
# Verbose mode for debugging query plans
pl.Config.set_verbose(True)
# Control streaming chunk size
with pl.Config(streaming_chunk_size=50_000):
result = lazy_df.collect(streaming=True)
# Reset all settings
pl.Config.restore_defaults()
๐ Behind the Question: Runtime configuration management. โ The Red Flags: Not customizing display settings when debugging large DataFrames.
Q224: How do you implement complex data transformations with fold() and reduce() in Polars?โ
Technical Answer:
pl.fold() accumulates across columns using a binary function (column-wise reduce). pl.reduce() applies a function across multiple expressions. Both avoid creating intermediate DataFrames and run in a single pass.
Code/Syntax Justification:
# Sum across specific columns (row-wise sum)
df.with_columns(
pl.fold(
acc=pl.lit(0),
function=lambda a, b: a + b,
exprs=pl.col("score_*"), # All columns matching pattern
).alias("total_score")
)
# Row-wise any/all
df.with_columns(
pl.any_horizontal("flag_*").alias("any_flag"),
pl.all_horizontal("check_*").alias("all_checks"),
)
# Sum/min/max across columns
df.with_columns(
pl.sum_horizontal("metric_a", "metric_b", "metric_c").alias("total"),
pl.min_horizontal("price_a", "price_b").alias("best_price"),
pl.max_horizontal("score_1", "score_2", "score_3").alias("max_score"),
)
๐ Behind the Question: Horizontal (row-wise) operations across columns.
โ The Red Flags: Using map_elements() for row-wise operations that have horizontal equivalents.
Q225: How does Polars handle large-scale CSV ingestion?โ
Technical Answer:
Polars' CSV reader is written in Rust, parallelizes across cores, and supports automatic type inference, null value detection, schema overrides, and row-count limits. scan_csv() with lazy mode enables predicate pushdown on row-level filters.
Code/Syntax Justification:
# Lazy scan with schema override
df = (
pl.scan_csv(
"large_data.csv",
dtypes={"id": pl.Int32, "amount": pl.Float32, "status": pl.Categorical},
null_values=["", "NULL", "N/A", "\\N"],
ignore_errors=True,
n_rows=None, # Read all rows
)
.filter(pl.col("amount") > 0)
.select(["id", "amount", "status"])
.collect()
)
# Read with row count limit for sampling
sample = pl.read_csv("data.csv", n_rows=10_000)
# Batch processing pattern
df = (
pl.scan_csv("data.csv")
.filter(pl.col("date") >= "2024-01-01")
.collect(streaming=True)
)
# Write CSV with options
df.write_csv("output.csv", separator="|", null_value="\\N", quote_style="necessary")
๐ Behind the Question: High-performance text file ingestion. โ The Red Flags: Using Pandas to read CSV then converting to Polars.
Q226: How do you implement rolling and expanding windows in Polars?โ
Technical Answer:
Polars rolling() on expressions applies fixed-size sliding windows. group_by_dynamic() provides time-based rolling with anchored intervals. Both support min_periods, center, and custom aggregation expressions within the window.
Code/Syntax Justification:
# Integer-based rolling
df.with_columns([
pl.col("price").rolling_mean(window_size=7).alias("ma_7"),
pl.col("price").rolling_std(window_size=20).alias("vol_20"),
pl.col("volume").rolling_sum(window_size=5).alias("vol_sum_5"),
])
# Time-based rolling with group_by_dynamic
(
df.sort("timestamp")
.group_by_dynamic("timestamp", every="1h", period="24h")
.agg([
pl.col("price").mean().alias("24h_avg"),
pl.col("price").std().alias("24h_vol"),
])
)
# Rolling with grouping
df.with_columns(
pl.col("price")
.rolling_mean(window_size=10)
.over("ticker")
.alias("ticker_ma_10")
)
๐ Behind the Question: Window computation patterns for financial and time-series data. โ The Red Flags: Implementing rolling calculations with manual loops or slicing.
Q227: What are Polars plugins and how do you extend Polars with Rust?โ
Technical Answer: Polars plugins let you write custom expressions in Rust that integrate with the query engine. They compile to shared libraries and run at native Polars speed with full access to Arrow arrays. Use for domain-specific operations not covered by built-in expressions.
Code/Syntax Justification:
# Python side: register and use plugin
# After building the Rust plugin crate
import polars_business # Example: business day calculations
df.with_columns(
pl.col("date").business.advance_n_days(n=5).alias("delivery_date")
)
# Common community plugins
# polars-distance: distance calculations
# polars-hash: hashing functions
# polars-xdt: extended datetime operations
# Using a plugin
import polars_xdt
df.with_columns(
pl.col("timestamp").xdt.to_julian_date().alias("julian"),
pl.col("date_col").xdt.is_workday().alias("is_business_day"),
)
๐ Behind the Question: Extending columnar engines with compiled code.
โ The Red Flags: Using map_elements() for performance-critical operations instead of writing a Rust plugin.
Q228: How do you benchmark Polars vs Pandas for a given workload?โ
Technical Answer:
Use identical data and operations, measure wall-clock time with time.perf_counter(), and track memory with tracemalloc or /proc/self/status. Key factors: data size, operation type (filter/join/groupby), number of columns, and string vs numeric columns. Polars typically wins at >100K rows.
Code/Syntax Justification:
import time
import tracemalloc
import pandas as pd
import polars as pl
import numpy as np
n = 10_000_000
data = {"a": np.random.rand(n), "b": np.random.randint(0, 1000, n), "c": np.random.choice(["x", "y", "z"], n)}
# Pandas benchmark
tracemalloc.start()
t0 = time.perf_counter()
pdf = pd.DataFrame(data)
result_pd = pdf.groupby("c").agg({"a": "mean", "b": "sum"})
t_pd = time.perf_counter() - t0
mem_pd = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
# Polars benchmark
tracemalloc.start()
t0 = time.perf_counter()
plf = pl.DataFrame(data)
result_pl = plf.group_by("c").agg([pl.col("a").mean(), pl.col("b").sum()])
t_pl = time.perf_counter() - t0
mem_pl = tracemalloc.get_traced_memory()[1]
tracemalloc.stop()
print(f"Pandas: {t_pd:.3f}s, {mem_pd/1e6:.1f}MB")
print(f"Polars: {t_pl:.3f}s, {mem_pl/1e6:.1f}MB")
๐ Behind the Question: Data-driven library selection methodology. โ The Red Flags: Benchmarking on small datasets or not measuring memory alongside time.
๐ Section 8: Cross-Library Patterns & Production Pipelinesโ
Q229: When should you use Pandas vs Polars vs NumPy for a given task?โ
Technical Answer: NumPy: homogeneous numeric arrays, linear algebra, signal processing, image manipulation. Pandas: interactive analysis, time-series with business calendar, extensive ecosystem integration (sklearn, statsmodels, plotting). Polars: large-scale ETL, Parquet-centric pipelines, multi-core processing, memory-constrained environments.
Code/Syntax Justification:
# NumPy: matrix operations, signal processing
eigenvalues = np.linalg.eigvalsh(covariance_matrix)
fft_result = np.fft.rfft(signal_data)
# Pandas: interactive exploration, rich ecosystem
df.describe()
df.plot(kind="scatter", x="age", y="salary")
statsmodels.api.OLS(y, X).fit()
# Polars: large-scale processing
(
pl.scan_parquet("s3://bucket/data/**/*.parquet")
.filter(pl.col("date") >= "2024-01-01")
.group_by("region")
.agg(pl.col("revenue").sum())
.collect(streaming=True)
)
๐ Behind the Question: Pragmatic library selection based on workload characteristics. โ The Red Flags: Using one library for everything or choosing based on familiarity rather than fit.
Q230: How do you convert between Pandas, Polars, and NumPy efficiently?โ
Technical Answer:
PolarsโPandas uses Arrow as the zero-copy interchange format via to_pandas() and pl.from_pandas(). NumPyโPandas uses .to_numpy() and pd.DataFrame(array). PolarsโNumPy goes through .to_numpy() or pl.from_numpy(). Avoid repeated conversions in tight loops.
Code/Syntax Justification:
# Polars โ Pandas (zero-copy via Arrow when possible)
pdf = plf.to_pandas(use_pyarrow_extension_types=True)
# Pandas โ Polars
plf = pl.from_pandas(pdf)
# Polars โ NumPy
arr = plf.select("numeric_col").to_numpy().flatten()
# NumPy โ Polars
plf = pl.from_numpy(arr, schema={"col1": pl.Float64, "col2": pl.Int32})
# Pandas โ NumPy
arr = pdf[["col1", "col2"]].to_numpy()
# NumPy โ Pandas
pdf = pd.DataFrame(arr, columns=["col1", "col2"])
# Arrow as universal interchange
import pyarrow as pa
arrow_table = plf.to_arrow()
pdf = arrow_table.to_pandas()
plf = pl.from_arrow(arrow_table)
๐ Behind the Question: Zero-copy data interchange between libraries. โ The Red Flags: Converting via CSV/JSON instead of Arrow for interop.
Q231: How do you build a production data pipeline with mixed Pandas/Polars/NumPy?โ
Technical Answer: Use Polars for I/O and heavy transformations, NumPy for numerical computations, and Pandas only where ecosystem libraries require it (e.g., plotting, statsmodels). Convert at boundaries, not within tight loops. Structure as a DAG of typed transformation functions.
Code/Syntax Justification:
from typing import Protocol
import polars as pl
import numpy as np
class DataPipeline:
def extract(self, path: str) -> pl.LazyFrame:
return pl.scan_parquet(path)
def transform(self, lf: pl.LazyFrame) -> pl.DataFrame:
return (
lf
.filter(pl.col("quality") > 0.5)
.with_columns([
pl.col("features").list.to_array(10).alias("feature_array"),
])
.collect()
)
def compute_scores(self, df: pl.DataFrame) -> np.ndarray:
"""NumPy for heavy numerical computation."""
X = df.select("feature_array").to_numpy()
scores = np.linalg.norm(X - X.mean(axis=0), axis=1)
return scores
def enrich(self, df: pl.DataFrame, scores: np.ndarray) -> pl.DataFrame:
return df.with_columns(pl.Series("score", scores))
def load(self, df: pl.DataFrame, path: str) -> None:
df.write_parquet(path, compression="zstd")
๐ Behind the Question: Multi-library pipeline architecture. โ The Red Flags: Random conversions between libraries without a clear boundary strategy.
Q232: How do you handle schema evolution in Parquet-based data pipelines?โ
Technical Answer:
Schema evolution means adding/removing/renaming columns over time. Parquet supports schema merging when reading. Use allow_missing_columns=True in Polars, explicit schema enforcement with cast(), and versioned schemas stored alongside data.
Code/Syntax Justification:
# Read with schema evolution (missing columns become null)
df = pl.read_parquet(
"data/v1/*.parquet",
allow_missing_columns=True,
)
# Enforce target schema
target_schema = {
"id": pl.Int64,
"name": pl.Utf8,
"amount": pl.Float64,
"new_field": pl.Utf8, # Added in v2
}
def enforce_schema(df: pl.DataFrame, schema: dict) -> pl.DataFrame:
for col, dtype in schema.items():
if col not in df.columns:
df = df.with_columns(pl.lit(None).cast(dtype).alias(col))
else:
df = df.with_columns(pl.col(col).cast(dtype))
return df.select(sorted(schema.keys()))
๐ Behind the Question: Data format evolution in long-lived pipelines. โ The Red Flags: Assuming all Parquet files have identical schemas across partitions and time.
Q233: How do you implement efficient feature engineering with Polars for ML?โ
Technical Answer:
Use lazy mode for feature computation, expressions for declarative transformations, over() for entity-level features, and group_by().agg() for aggregated features. Write features to Parquet for offline training and reuse expressions for online serving.
Code/Syntax Justification:
def build_user_features(events: pl.LazyFrame) -> pl.LazyFrame:
return events.group_by("user_id").agg([
# Recency
pl.col("timestamp").max().alias("last_event"),
# Frequency
pl.col("event_id").count().alias("event_count"),
pl.col("event_id").n_unique().alias("unique_events"),
# Monetary
pl.col("amount").sum().alias("total_amount"),
pl.col("amount").mean().alias("avg_amount"),
pl.col("amount").std().alias("std_amount"),
# Behavioral
pl.col("category").n_unique().alias("category_diversity"),
pl.col("device").mode().first().alias("primary_device"),
# Temporal
pl.col("timestamp").diff().dt.total_hours().mean().alias("avg_hours_between"),
(pl.col("timestamp").max() - pl.col("timestamp").min()).dt.total_days().alias("active_days"),
])
๐ Behind the Question: Declarative feature engineering at scale. โ The Red Flags: Computing features with Python loops or one-at-a-time column operations.
Q234: How do you implement data quality monitoring with Polars?โ
Technical Answer: Build expression-based data quality checks that run as part of the pipeline, tracking null rates, value distributions, schema compliance, and statistical anomalies. Store results as structured metrics for trending and alerting.
Code/Syntax Justification:
def compute_quality_metrics(df: pl.DataFrame, name: str) -> pl.DataFrame:
metrics = []
for col in df.columns:
col_metrics = df.select([
pl.lit(name).alias("dataset"),
pl.lit(col).alias("column"),
pl.col(col).null_count().alias("null_count"),
(pl.col(col).null_count() / pl.len()).alias("null_rate"),
pl.col(col).n_unique().alias("unique_count"),
])
metrics.append(col_metrics)
return pl.concat(metrics)
def check_expectations(df: pl.DataFrame) -> dict:
return {
"row_count": df.height,
"no_nulls_in_id": df.filter(pl.col("id").is_null()).height == 0,
"positive_amounts": df.filter(pl.col("amount") <= 0).height == 0,
"valid_dates": df.filter(pl.col("date") > datetime.now()).height == 0,
}
๐ Behind the Question: Automated data quality enforcement in production. โ The Red Flags: No data quality checks between pipeline stages.
Q235: How do you use Apache Arrow as the interchange format between libraries?โ
Technical Answer: Arrow defines a language-independent columnar memory format. Polars, Pandas (2.0+), DuckDB, and many ML libraries natively support Arrow. Using Arrow as the interchange format enables zero-copy transfers between libraries without serialization overhead.
Code/Syntax Justification:
import pyarrow as pa
import pyarrow.parquet as pq
import polars as pl
import pandas as pd
# Arrow as universal interchange
arrow_table = pa.table({
"id": pa.array([1, 2, 3]),
"name": pa.array(["a", "b", "c"]),
"value": pa.array([1.0, 2.0, 3.0]),
})
# Zero-copy conversions
polars_df = pl.from_arrow(arrow_table)
pandas_df = arrow_table.to_pandas(types_mapper=pd.ArrowDtype)
# DuckDB integration via Arrow
import duckdb
result = duckdb.sql("SELECT * FROM arrow_table WHERE value > 1").arrow()
polars_result = pl.from_arrow(result)
# IPC (Feather) for fast serialization
pq.write_table(arrow_table, "data.parquet")
with pa.ipc.new_file("data.arrow", arrow_table.schema) as writer:
writer.write_table(arrow_table)
๐ Behind the Question: Zero-copy data interoperability across the Python data ecosystem. โ The Red Flags: Serializing to CSV/JSON for library interop when Arrow is available.
Q236: How do you implement incremental processing with Polars?โ
Technical Answer:
Track watermarks (last processed timestamp or ID) to read only new data. Use Parquet partition pruning, scan_parquet() with filters, and sink_parquet() with append mode. Store state externally (file, database) for crash recovery.
Code/Syntax Justification:
import json
from pathlib import Path
class IncrementalProcessor:
def __init__(self, state_path: str):
self.state_path = Path(state_path)
self.state = self._load_state()
def _load_state(self) -> dict:
if self.state_path.exists():
return json.loads(self.state_path.read_text())
return {"last_processed_id": 0}
def process(self, source_path: str, output_path: str):
new_data = (
pl.scan_parquet(source_path)
.filter(pl.col("id") > self.state["last_processed_id"])
.sort("id")
.collect()
)
if new_data.height == 0:
return
# Transform
result = self._transform(new_data)
# Append to output
result.write_parquet(f"{output_path}/batch_{new_data['id'].max()}.parquet")
# Update state
self.state["last_processed_id"] = int(new_data["id"].max())
self.state_path.write_text(json.dumps(self.state))
๐ Behind the Question: Efficient batch processing patterns for growing datasets. โ The Red Flags: Reprocessing entire datasets on every run without incremental logic.
Q237: How do you handle large-scale string deduplication across datasets?โ
Technical Answer: Use hash-based deduplication for exact matches, MinHash/LSH for near-duplicate detection, and phonetic algorithms for fuzzy name matching. Polars' vectorized string operations handle the hashing and comparison at scale.
Code/Syntax Justification:
import hashlib
# Exact deduplication by hash
df = df.with_columns(
pl.concat_str(["name", "email", "phone"], separator="|")
.hash()
.alias("record_hash")
).unique(subset=["record_hash"])
# Fuzzy deduplication with normalized keys
df = df.with_columns(
pl.col("name")
.str.to_lowercase()
.str.strip_chars()
.str.replace_all(r"\s+", " ")
.alias("name_normalized")
).unique(subset=["name_normalized", "email"])
# Cross-dataset dedup
combined = pl.concat([df1.with_columns(pl.lit("source1").alias("src")),
df2.with_columns(pl.lit("source2").alias("src"))])
deduped = combined.unique(subset=["email"], keep="first")
๐ Behind the Question: Data deduplication at scale. โ The Red Flags: Nested-loop comparison for deduplication instead of hash-based approaches.
Q238: How do you implement a DataFrame-based configuration system for data pipelines?โ
Technical Answer: Store pipeline configuration as DataFrames โ column mappings, validation rules, transformation parameters. This makes configs queryable, version-controlled as Parquet, and programmatically composable. Use Polars for config joins against data.
Code/Syntax Justification:
# Column mapping configuration
column_config = pl.DataFrame({
"source_col": ["customer_name", "order_amt", "order_dt"],
"target_col": ["name", "amount", "date"],
"dtype": ["Utf8", "Float64", "Date"],
"nullable": [False, False, True],
"transform": ["strip_upper", "round_2", "parse_date"],
})
def apply_column_mapping(df: pl.DataFrame, config: pl.DataFrame) -> pl.DataFrame:
expressions = []
for row in config.iter_rows(named=True):
expr = pl.col(row["source_col"])
if row["transform"] == "strip_upper":
expr = expr.str.strip_chars().str.to_uppercase()
elif row["transform"] == "round_2":
expr = expr.round(2)
elif row["transform"] == "parse_date":
expr = expr.str.to_date("%Y-%m-%d")
expressions.append(expr.alias(row["target_col"]))
return df.select(expressions)
๐ Behind the Question: Configuration-driven data pipelines for maintainability. โ The Red Flags: Hardcoding column names and transformations directly in pipeline code.
Q239: How do you profile Polars query performance?โ
Technical Answer:
Use explain() to view the optimized query plan, profile() to get per-node execution times, and POLARS_VERBOSE=1 environment variable for detailed engine logs. Compare lazy vs eager timing to measure optimization impact.
Code/Syntax Justification:
import os
os.environ["POLARS_VERBOSE"] = "1"
# View optimized logical plan
lazy_q = (
pl.scan_parquet("data.parquet")
.filter(pl.col("a") > 10)
.select(["a", "b"])
.group_by("b")
.agg(pl.col("a").sum())
)
print(lazy_q.explain(optimized=True))
# Profile: returns (result, timing_df)
result, timing = lazy_q.profile()
print(timing) # Per-node execution time
# Manual timing comparison
import time
t0 = time.perf_counter()
eager_result = pl.read_parquet("data.parquet").filter(pl.col("a") > 10).select(["a", "b"])
t_eager = time.perf_counter() - t0
t0 = time.perf_counter()
lazy_result = pl.scan_parquet("data.parquet").filter(pl.col("a") > 10).select(["a", "b"]).collect()
t_lazy = time.perf_counter() - t0
print(f"Eager: {t_eager:.3f}s, Lazy: {t_lazy:.3f}s")
๐ Behind the Question: Query performance analysis and optimization. โ The Red Flags: Not examining the query plan or profiling before optimizing.
Q240: How do you handle memory pressure in large-scale Polars pipelines?โ
Technical Answer:
Use streaming mode (collect(streaming=True)) for queries exceeding RAM, sink_parquet() for streaming writes, partition large datasets for parallel chunk processing, and monitor memory with estimated_size(). Reduce column cardinality with categoricals and appropriate dtypes.
Code/Syntax Justification:
# Streaming collection for large queries
result = (
pl.scan_parquet("huge/*.parquet")
.filter(pl.col("date") >= "2024-01-01")
.group_by("category")
.agg(pl.col("amount").sum())
.collect(streaming=True)
)
# Check estimated memory
df = pl.read_parquet("data.parquet")
print(f"Estimated size: {df.estimated_size('mb'):.1f} MB")
# Reduce memory with dtypes
df = df.with_columns([
pl.col("id").cast(pl.Int32),
pl.col("status").cast(pl.Categorical),
pl.col("amount").cast(pl.Float32),
])
print(f"After optimization: {df.estimated_size('mb'):.1f} MB")
# Partition processing for bounded memory
partitions = df.partition_by("region")
results = [process_partition(p) for p in partitions]
final = pl.concat(results)
๐ Behind the Question: Memory management for data-intensive workloads. โ The Red Flags: Collecting entire lazy frames eagerly without checking estimated size.
Q241: How do you test data transformation functions effectively?โ
Technical Answer:
Use small fixture DataFrames with known edge cases, assert on schema (dtypes + column names), values (assert_frame_equal), and shape. Test null handling, empty DataFrames, single-row frames, and boundary conditions separately.
Code/Syntax Justification:
import pytest
from polars.testing import assert_frame_equal
def test_revenue_calculation():
input_df = pl.DataFrame({
"price": [10.0, 0.0, None, 5.0],
"quantity": [2, 3, 1, None],
})
result = calculate_revenue(input_df)
expected = pl.DataFrame({
"price": [10.0, 0.0, None, 5.0],
"quantity": [2, 3, 1, None],
"revenue": [20.0, 0.0, None, None],
})
assert_frame_equal(result, expected)
def test_empty_dataframe():
empty = pl.DataFrame(schema={"price": pl.Float64, "quantity": pl.Int64})
result = calculate_revenue(empty)
assert result.height == 0
assert "revenue" in result.columns
def test_schema_preservation():
result = transform(sample_df)
assert result.schema == {"id": pl.Int64, "name": pl.Utf8, "score": pl.Float64}
๐ Behind the Question: Testing methodology for data transformation code. โ The Red Flags: Not testing with null values, empty DataFrames, or edge cases.
Q242: How do you implement Pandas-style MultiIndex functionality in Polars?โ
Technical Answer:
Polars intentionally does not have MultiIndex. Instead, use multiple group-by columns, pivot()/unpivot() for reshaping, and struct columns for hierarchical keys. This is simpler and avoids the complexity/performance issues of Pandas MultiIndex.
Code/Syntax Justification:
# Pandas MultiIndex equivalent in Polars: just use multiple columns
result = (
df
.group_by(["region", "product", "quarter"])
.agg(pl.col("revenue").sum())
.sort(["region", "product", "quarter"])
)
# Cross-section query (Pandas xs() equivalent)
east_data = result.filter(pl.col("region") == "East")
# Pivot (unstack equivalent)
pivoted = result.pivot(
on="quarter",
index=["region", "product"],
values="revenue",
)
# Unpivot (stack equivalent)
melted = pivoted.unpivot(
index=["region", "product"],
variable_name="quarter",
value_name="revenue",
)
๐ Behind the Question: Translating Pandas MultiIndex patterns to Polars idioms. โ The Red Flags: Trying to force MultiIndex semantics into Polars instead of using flat group-by.
Q243: How do you implement efficient A/B test analysis with these libraries?โ
Technical Answer: Use Polars for data preparation and metric computation, NumPy/SciPy for statistical tests, and Pandas for visualization integration. Compute group-level metrics with expressions, then apply hypothesis tests on aggregated results.
Code/Syntax Justification:
import polars as pl
import numpy as np
from scipy import stats
# Compute per-variant metrics in Polars
metrics = (
events
.group_by("variant")
.agg([
pl.col("converted").mean().alias("conversion_rate"),
pl.col("converted").sum().alias("conversions"),
pl.col("converted").count().alias("total_users"),
pl.col("revenue").mean().alias("avg_revenue"),
pl.col("revenue").std().alias("std_revenue"),
])
)
# Extract for statistical test
control = events.filter(pl.col("variant") == "control")["revenue"].to_numpy()
treatment = events.filter(pl.col("variant") == "treatment")["revenue"].to_numpy()
# T-test
t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)
# Bootstrap confidence interval
n_bootstrap = 10_000
rng = np.random.default_rng(42)
diffs = []
for _ in range(n_bootstrap):
c_sample = rng.choice(control, size=len(control), replace=True)
t_sample = rng.choice(treatment, size=len(treatment), replace=True)
diffs.append(t_sample.mean() - c_sample.mean())
ci_95 = np.percentile(diffs, [2.5, 97.5])
print(f"95% CI for difference: [{ci_95[0]:.4f}, {ci_95[1]:.4f}]")
๐ Behind the Question: Statistical analysis workflows combining all three libraries. โ The Red Flags: Not using proper statistical tests or computing metrics one-at-a-time.
Q244: How do you handle geospatial data with Pandas and NumPy?โ
Technical Answer: Use GeoPandas (extends Pandas with geometry columns) for vector spatial data, or NumPy for raster calculations. Spatial joins, distance calculations, and projections require specialized libraries like Shapely and pyproj.
Code/Syntax Justification:
import geopandas as gpd
from shapely.geometry import Point
import numpy as np
# Create GeoDataFrame
gdf = gpd.GeoDataFrame(
df,
geometry=gpd.points_from_xy(df["longitude"], df["latitude"]),
crs="EPSG:4326",
)
# Spatial join: find points within polygons
regions = gpd.read_file("regions.geojson")
joined = gpd.sjoin(gdf, regions, how="left", predicate="within")
# Distance calculation with NumPy (Haversine)
def haversine_np(lat1, lon1, lat2, lon2):
R = 6371 # Earth radius in km
lat1, lon1, lat2, lon2 = map(np.radians, [lat1, lon1, lat2, lon2])
dlat = lat2 - lat1
dlon = lon2 - lon1
a = np.sin(dlat/2)**2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon/2)**2
return R * 2 * np.arcsin(np.sqrt(a))
๐ Behind the Question: Spatial data processing in the Python data stack. โ The Red Flags: Computing distances with Python loops instead of vectorized Haversine.
Q245: How do you implement data versioning and lineage tracking?โ
Technical Answer: Store metadata (schema version, row count, hash, timestamp, source files) alongside each output. Use content-addressable naming, Parquet metadata fields, and external catalogs. Track transformations as a DAG for reproducibility.
Code/Syntax Justification:
import hashlib
from datetime import datetime
def write_versioned(df: pl.DataFrame, base_path: str, version: str):
"""Write with lineage metadata."""
content_hash = hashlib.sha256(
df.to_pandas().to_json().encode()
).hexdigest()[:16]
metadata = {
"version": version,
"row_count": str(df.height),
"columns": ",".join(df.columns),
"content_hash": content_hash,
"created_at": datetime.utcnow().isoformat(),
"schema_hash": hashlib.md5(str(df.schema).encode()).hexdigest(),
}
path = f"{base_path}/v{version}/{content_hash}.parquet"
df.write_parquet(path)
# Write metadata sidecar
import json
with open(f"{path}.meta.json", "w") as f:
json.dump(metadata, f, indent=2)
return path
๐ Behind the Question: Data governance and reproducibility in ML/analytics pipelines. โ The Red Flags: Overwriting output files without any versioning or metadata tracking.
Q246: How do you implement efficient pivot tables across large datasets?โ
Technical Answer:
Use pivot() for wide-format reshaping (Polars) or pivot_table() for aggregated pivots (Pandas). For very large datasets, pre-aggregate with group_by before pivoting to reduce the pivot's input size. Sparse pivots can use SparseDtype in Pandas.
Code/Syntax Justification:
# Polars pivot
pivoted = (
df
.group_by(["region", "quarter"])
.agg(pl.col("revenue").sum())
.pivot(on="quarter", index="region", values="revenue")
)
# Pandas pivot_table with multiple aggregations
pivot = pd.pivot_table(
df,
values="revenue",
index=["region", "product"],
columns="quarter",
aggfunc={"revenue": ["sum", "mean", "count"]},
fill_value=0,
margins=True,
)
# Pre-aggregate for large datasets
pre_agg = df.group_by(["region", "quarter"]).agg(
pl.col("revenue").sum().alias("total_revenue")
)
wide = pre_agg.pivot(on="quarter", index="region", values="total_revenue")
๐ Behind the Question: Reshaping patterns for reporting and analysis. โ The Red Flags: Pivoting raw data with millions of rows without pre-aggregation.
Q247: How do you implement parallel data processing with Polars and multiprocessing?โ
Technical Answer:
Polars already parallelizes internally via Rayon (Rust thread pool). For pipeline-level parallelism, use concurrent.futures to process independent partitions. Avoid multiprocessing with large DataFrames due to serialization overhead โ use file-based interchange instead.
Code/Syntax Justification:
from concurrent.futures import ProcessPoolExecutor
from pathlib import Path
def process_partition(path: str) -> str:
"""Process a single partition file."""
df = pl.read_parquet(path)
result = df.group_by("category").agg(pl.col("amount").sum())
output_path = path.replace("raw/", "processed/")
result.write_parquet(output_path)
return output_path
# File-based parallel processing (no serialization overhead)
partition_files = list(Path("data/raw/").glob("*.parquet"))
with ProcessPoolExecutor(max_workers=4) as executor:
output_paths = list(executor.map(process_partition, [str(p) for p in partition_files]))
# Combine results
final = pl.concat([pl.read_parquet(p) for p in output_paths])
๐ Behind the Question: Pipeline-level parallelism beyond single-query parallelism. โ The Red Flags: Passing large DataFrames between processes (serialization bottleneck).
Q248: What are the key differences between Polars 0.x and 1.x API?โ
Technical Answer:
Polars 1.0 stabilized the API with breaking changes: apply() renamed to map_elements()/map_batches(), groupby() renamed to group_by(), melt() renamed to unpivot(), date range functions moved to pl.date_range(), and stricter type checking. The lazy API is now the recommended default.
Code/Syntax Justification:
# Old (0.x) โ New (1.x) API changes
# groupby โ group_by
# OLD: df.groupby("col").agg(...)
df.group_by("col").agg(...) # NEW
# apply โ map_elements / map_batches
# OLD: df.select(pl.col("x").apply(func))
df.select(pl.col("x").map_elements(func)) # NEW (element-wise)
df.select(pl.col("x").map_batches(func)) # NEW (batch)
# melt โ unpivot
# OLD: df.melt(id_vars=..., value_vars=...)
df.unpivot(index=..., on=...) # NEW
# Date range
# OLD: pl.date_range(start, end, "1d")
pl.date_range(start, end, "1d", eager=True) # NEW (explicit eager/lazy)
# Strict typing
# OLD: pl.col("x").apply(lambda v: str(v))
pl.col("x").map_elements(lambda v: str(v), return_dtype=pl.Utf8) # NEW
๐ Behind the Question: API migration knowledge for production codebases. โ The Red Flags: Using deprecated 0.x API names in new Polars 1.x code.
๐ญ Section 9: Production Data Engineering Patternsโ
Q249: How do you implement idempotent data pipeline stages?โ
Technical Answer: Idempotent stages produce the same output regardless of how many times they run. Write outputs to deterministic paths based on input identifiers, use atomic file operations (write-then-rename), and include deduplication logic. This enables safe retries after failures.
Code/Syntax Justification:
import hashlib
from pathlib import Path
import polars as pl
def idempotent_transform(input_path: str, output_dir: str) -> str:
"""Same input always produces same output path."""
# Deterministic output path from input
input_hash = hashlib.md5(input_path.encode()).hexdigest()[:12]
output_path = Path(output_dir) / f"result_{input_hash}.parquet"
if output_path.exists():
return str(output_path) # Already processed
# Process
df = pl.read_parquet(input_path)
result = df.filter(pl.col("valid") == True).unique(subset=["id"])
# Atomic write: write to temp, then rename
tmp_path = output_path.with_suffix(".tmp.parquet")
result.write_parquet(str(tmp_path))
tmp_path.rename(output_path)
return str(output_path)
๐ Behind the Question: Pipeline reliability and fault tolerance. โ The Red Flags: Pipelines that produce different results or corrupt data on retry.
Q250: How do you implement data contracts between pipeline stages?โ
Technical Answer: Data contracts define the expected schema, value ranges, null policies, and cardinality between producer and consumer stages. Enforce contracts at stage boundaries with runtime validation. Use Polars schema inspection and assertion expressions.
Code/Syntax Justification:
from dataclasses import dataclass
import polars as pl
@dataclass
class DataContract:
name: str
schema: dict[str, pl.DataType]
non_nullable: list[str]
unique_keys: list[str]
def validate(self, df: pl.DataFrame) -> list[str]:
errors = []
# Schema check
for col, dtype in self.schema.items():
if col not in df.columns:
errors.append(f"Missing column: {col}")
elif df[col].dtype != dtype:
errors.append(f"{col}: expected {dtype}, got {df[col].dtype}")
# Null check
for col in self.non_nullable:
if col in df.columns and df[col].null_count() > 0:
errors.append(f"{col} has {df[col].null_count()} nulls")
# Uniqueness check
if self.unique_keys:
dupes = df.height - df.unique(subset=self.unique_keys).height
if dupes > 0:
errors.append(f"{dupes} duplicate rows on {self.unique_keys}")
return errors
orders_contract = DataContract(
name="orders",
schema={"order_id": pl.Int64, "user_id": pl.Int64, "amount": pl.Float64},
non_nullable=["order_id", "user_id"],
unique_keys=["order_id"],
)
๐ Behind the Question: Interface reliability between pipeline components. โ The Red Flags: No schema validation between producer and consumer stages.
Q251: How do you implement efficient data sampling strategies?โ
Technical Answer:
Simple random sampling uses sample(). Stratified sampling maintains group proportions. Reservoir sampling handles streaming data. Systematic sampling picks every k-th row. Choose based on whether you need representativeness, speed, or streaming capability.
Code/Syntax Justification:
# Simple random sample
sample = df.sample(n=1000, seed=42)
sample_frac = df.sample(fraction=0.01, seed=42)
# Stratified sampling in Polars
def stratified_sample(df: pl.DataFrame, group_col: str, n_per_group: int) -> pl.DataFrame:
return (
df
.with_columns(pl.arange(0, pl.len()).shuffle(seed=42).over(group_col).alias("_rank"))
.filter(pl.col("_rank") < n_per_group)
.drop("_rank")
)
balanced = stratified_sample(df, "category", n_per_group=100)
# Reservoir sampling for streaming (NumPy)
def reservoir_sample(stream, k: int, rng=None):
rng = rng or np.random.default_rng(42)
reservoir = []
for i, item in enumerate(stream):
if i < k:
reservoir.append(item)
else:
j = rng.integers(0, i + 1)
if j < k:
reservoir[j] = item
return reservoir
๐ Behind the Question: Statistical sampling for development, testing, and exploration.
โ The Red Flags: Using head() instead of proper random sampling for analysis subsets.
Q252: How do you implement change data capture (CDC) processing with DataFrames?โ
Technical Answer:
CDC processes capture inserts, updates, and deletes from source systems. With DataFrames, compare snapshots using hash-based diff, track operation types, and apply changes incrementally. Use anti_join for deletes, semi_join for matches, and hash columns for change detection.
Code/Syntax Justification:
def detect_changes(current: pl.DataFrame, previous: pl.DataFrame, key_cols: list[str]) -> dict:
"""Detect inserts, updates, and deletes between snapshots."""
# New rows (in current but not previous)
inserts = current.join(previous, on=key_cols, how="anti")
# Deleted rows (in previous but not current)
deletes = previous.join(current, on=key_cols, how="anti")
# Potential updates (matching keys)
matched = current.join(previous, on=key_cols, how="inner", suffix="_prev")
# Detect actual changes via hash comparison
value_cols = [c for c in current.columns if c not in key_cols]
hash_current = matched.select(key_cols + [
pl.concat_str(value_cols, separator="|").hash().alias("_hash_curr")
])
hash_prev = matched.select(key_cols + [
pl.concat_str([f"{c}_prev" for c in value_cols], separator="|").hash().alias("_hash_prev")
])
updates = current.join(
hash_current.join(hash_prev, on=key_cols)
.filter(pl.col("_hash_curr") != pl.col("_hash_prev"))
.select(key_cols),
on=key_cols,
)
return {"inserts": inserts, "updates": updates, "deletes": deletes}
๐ Behind the Question: Incremental data synchronization patterns. โ The Red Flags: Full table replacement instead of incremental CDC processing.
Q253: How do you implement outlier detection with NumPy and Pandas?โ
Technical Answer: Common methods: Z-score (assumes normality), IQR (robust to skew), Modified Z-score (uses median absolute deviation), and Isolation Forest (ML-based). Choose based on data distribution and whether outliers are errors or legitimate extreme values.
Code/Syntax Justification:
import numpy as np
import pandas as pd
# Z-score method
z_scores = np.abs((df["value"] - df["value"].mean()) / df["value"].std())
outliers_z = df[z_scores > 3]
# IQR method (robust)
Q1 = df["value"].quantile(0.25)
Q3 = df["value"].quantile(0.75)
IQR = Q3 - Q1
outliers_iqr = df[(df["value"] < Q1 - 1.5 * IQR) | (df["value"] > Q3 + 1.5 * IQR)]
# Modified Z-score (Median Absolute Deviation)
median = np.median(df["value"])
mad = np.median(np.abs(df["value"] - median))
modified_z = 0.6745 * (df["value"] - median) / (mad + 1e-10)
outliers_mad = df[np.abs(modified_z) > 3.5]
# Per-group outlier detection
df["is_outlier"] = df.groupby("category")["value"].transform(
lambda x: np.abs((x - x.mean()) / (x.std() + 1e-10)) > 3
)
๐ Behind the Question: Statistical anomaly detection in data quality pipelines. โ The Red Flags: Using only Z-score for skewed distributions or removing outliers without investigation.
Q254: How do you implement efficient data deduplication with fuzzy matching?โ
Technical Answer:
Exact dedup uses hash or unique keys. Fuzzy dedup requires blocking (reduce comparison pairs), similarity scoring (edit distance, TF-IDF cosine), and threshold-based linking. Libraries like recordlinkage and rapidfuzz integrate with Pandas for production fuzzy matching.
Code/Syntax Justification:
from rapidfuzz import fuzz, process
import pandas as pd
def fuzzy_dedup(df: pd.DataFrame, col: str, threshold: int = 85) -> pd.DataFrame:
"""Deduplicate using fuzzy string matching with blocking."""
# Blocking: group by first 3 characters to reduce comparisons
df["_block"] = df[col].str[:3].str.lower()
keep_mask = pd.Series(True, index=df.index)
for block, group in df.groupby("_block"):
if len(group) < 2:
continue
values = group[col].tolist()
indices = group.index.tolist()
for i in range(len(values)):
if not keep_mask[indices[i]]:
continue
for j in range(i + 1, len(values)):
if not keep_mask[indices[j]]:
continue
if fuzz.ratio(values[i], values[j]) >= threshold:
keep_mask[indices[j]] = False # Mark duplicate
return df[keep_mask].drop(columns=["_block"])
๐ Behind the Question: Real-world deduplication where exact matching fails. โ The Red Flags: Only using exact match dedup when data has typos, abbreviations, or formatting variations.
Q255: How do you implement time-series resampling and interpolation?โ
Technical Answer:
Resampling changes time frequency: downsampling aggregates (5min โ 1h), upsampling creates gaps that need interpolation. Use resample() in Pandas, group_by_dynamic() in Polars. Interpolation methods include linear, polynomial, spline, and forward/backward fill.
Code/Syntax Justification:
# Pandas resampling
ts = pd.DataFrame({
"timestamp": pd.date_range("2024-01-01", periods=1000, freq="5min"),
"value": np.random.rand(1000),
})
ts = ts.set_index("timestamp")
# Downsample: 5min โ 1h
hourly = ts.resample("1h").agg({"value": ["mean", "min", "max", "count"]})
# Upsample: 1h โ 1min with interpolation
minutely = ts.resample("1min").interpolate(method="cubic")
# Polars equivalent
ts_pl = pl.DataFrame({
"timestamp": pl.date_range(datetime(2024, 1, 1), datetime(2024, 1, 5), "5m", eager=True),
"value": np.random.rand(1152),
})
hourly_pl = ts_pl.group_by_dynamic("timestamp", every="1h").agg([
pl.col("value").mean().alias("avg"),
pl.col("value").min().alias("min"),
pl.col("value").max().alias("max"),
])
๐ Behind the Question: Time-series frequency conversion patterns. โ The Red Flags: Manually binning timestamps instead of using resample/group_by_dynamic.
Q256: How do you implement efficient one-hot encoding for high-cardinality features?โ
Technical Answer:
Use sparse representation for high cardinality to avoid memory explosion. pd.get_dummies(sparse=True) creates sparse DataFrame. For ML, use sklearn.preprocessing.OneHotEncoder with sparse_output=True. Consider target encoding or feature hashing for very high cardinality.
Code/Syntax Justification:
# Pandas sparse one-hot
dummies = pd.get_dummies(df["category"], sparse=True, prefix="cat")
print(f"Dense would be: {len(df) * dummies.shape[1] * 8 / 1e6:.0f} MB")
print(f"Sparse is: {dummies.memory_usage(deep=True).sum() / 1e6:.1f} MB")
# sklearn for ML pipelines
from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_extraction import FeatureHasher
# Sparse one-hot
ohe = OneHotEncoder(sparse_output=True, handle_unknown="ignore")
X_encoded = ohe.fit_transform(df[["category"]])
# Feature hashing for very high cardinality (no vocabulary needed)
hasher = FeatureHasher(n_features=256, input_type="string")
X_hashed = hasher.fit_transform(df["category"].apply(lambda x: [x]))
# Target encoding (handles high cardinality without dimension explosion)
from sklearn.preprocessing import TargetEncoder
te = TargetEncoder(smooth="auto")
df["cat_encoded"] = te.fit_transform(df[["category"]], df["target"])
๐ Behind the Question: Feature encoding strategies for different cardinality levels. โ The Red Flags: Dense one-hot encoding a column with 10,000+ unique values.
Q257: How do you implement efficient data binning and discretization?โ
Technical Answer:
pd.cut() bins into equal-width intervals. pd.qcut() bins into equal-frequency quantiles. Custom bins use explicit edges. For numerical stability, use np.digitize() with sorted bins. Polars uses cut() expressions.
Code/Syntax Justification:
# Equal-width bins
df["price_bin"] = pd.cut(df["price"], bins=10, labels=False)
# Equal-frequency quantile bins
df["price_quartile"] = pd.qcut(df["price"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])
# Custom bins with labels
edges = [0, 10, 50, 100, 500, float("inf")]
labels = ["micro", "small", "medium", "large", "enterprise"]
df["size_tier"] = pd.cut(df["revenue"], bins=edges, labels=labels)
# NumPy digitize for fast bin lookup
bins = np.array([0, 10, 50, 100, 500])
df["bin_idx"] = np.digitize(df["revenue"].values, bins)
# Polars cut
df_pl = df_pl.with_columns(
pl.col("price").cut([10, 50, 100, 500]).alias("price_tier")
)
๐ Behind the Question: Continuous-to-categorical transformation for analytics and ML. โ The Red Flags: Manually implementing binning logic with if/else chains.
Q258: How do you implement efficient DataFrame serialization for caching?โ
Technical Answer: Parquet is best for disk caching (compressed, columnar, typed). Feather/Arrow IPC is best for inter-process or same-machine caching (zero-copy deserialization). Pickle is fast but fragile across versions. Never use CSV for caching.
Code/Syntax Justification:
import time
# Benchmark serialization formats
df = pl.DataFrame({"a": np.random.rand(1_000_000), "b": np.random.randint(0, 100, 1_000_000)})
# Parquet: best compression, good read speed
t0 = time.perf_counter()
df.write_parquet("cache.parquet", compression="zstd")
t_write = time.perf_counter() - t0
t0 = time.perf_counter()
_ = pl.read_parquet("cache.parquet")
t_read = time.perf_counter() - t0
print(f"Parquet: write={t_write:.3f}s, read={t_read:.3f}s")
# Arrow IPC: fastest deserialization
t0 = time.perf_counter()
df.write_ipc("cache.arrow")
t_write = time.perf_counter() - t0
t0 = time.perf_counter()
_ = pl.read_ipc("cache.arrow")
t_read = time.perf_counter() - t0
print(f"Arrow IPC: write={t_write:.3f}s, read={t_read:.3f}s")
# Memory-mapped reading for repeated access
df_mmap = pl.read_ipc("cache.arrow", memory_map=True)
๐ Behind the Question: Data caching strategies for pipeline performance. โ The Red Flags: Using CSV or pickle for DataFrame caching in production.
Q259: How do you implement efficient data aggregation across multiple time granularities?โ
Technical Answer:
Pre-compute aggregations at multiple levels (minute, hour, day, week, month) in a single pass using group_by with truncated timestamps. Store each level separately for fast query serving. This is the core of OLAP cube pre-computation.
Code/Syntax Justification:
# Multi-granularity aggregation in one pass
def compute_time_aggregations(df: pl.LazyFrame) -> dict[str, pl.DataFrame]:
base_agg = [
pl.col("revenue").sum().alias("total_revenue"),
pl.col("order_id").n_unique().alias("order_count"),
pl.col("user_id").n_unique().alias("user_count"),
]
return {
"hourly": df.with_columns(
pl.col("timestamp").dt.truncate("1h").alias("period")
).group_by("period").agg(base_agg).collect(),
"daily": df.with_columns(
pl.col("timestamp").dt.truncate("1d").alias("period")
).group_by("period").agg(base_agg).collect(),
"weekly": df.with_columns(
pl.col("timestamp").dt.truncate("1w").alias("period")
).group_by("period").agg(base_agg).collect(),
"monthly": df.with_columns(
pl.col("timestamp").dt.truncate("1mo").alias("period")
).group_by("period").agg(base_agg).collect(),
}
๐ Behind the Question: Multi-resolution time-series aggregation for dashboards and reporting. โ The Red Flags: Computing each time granularity independently from raw data.
Q260: How do you implement data partitioning strategies for large datasets?โ
Technical Answer: Partition by time (daily/monthly directories), by category (region, tenant), or by hash (evenly distributed). Time partitioning enables efficient pruning for range queries. Hive-style partitioning is the standard for Parquet-based data lakes.
Code/Syntax Justification:
import pyarrow as pa
import pyarrow.parquet as pq
# Write hive-partitioned Parquet
table = df.to_arrow()
pq.write_to_dataset(
table,
root_path="data/events/",
partition_cols=["year", "month"],
existing_data_behavior="overwrite_or_ignore",
)
# Read with partition pruning (only scans matching directories)
df = pl.scan_parquet(
"data/events/**/*.parquet",
hive_partitioning=True,
).filter(
(pl.col("year") == 2024) & (pl.col("month") >= 6)
).collect()
# Hash partitioning for even distribution
n_partitions = 16
df = df.with_columns(
(pl.col("user_id").hash() % n_partitions).alias("partition")
)
for i in range(n_partitions):
partition_df = df.filter(pl.col("partition") == i).drop("partition")
partition_df.write_parquet(f"data/partitioned/part_{i:04d}.parquet")
๐ Behind the Question: Storage-level optimization for query performance. โ The Red Flags: Storing all data in a single file without partitioning.
Q261: How do you implement efficient data joins with skewed key distributions?โ
Technical Answer: Skewed joins occur when a few keys dominate (e.g., one user has 90% of orders). Solutions: salting (add random suffix to hot keys, join, then aggregate), broadcast join for small tables, or pre-filtering hot keys for separate processing.
Code/Syntax Justification:
# Detect skew
key_counts = df.group_by("user_id").agg(pl.len().alias("count"))
hot_keys = key_counts.filter(pl.col("count") > key_counts["count"].quantile(0.99))
print(f"Hot keys: {hot_keys.height}, covering {hot_keys['count'].sum()} rows")
# Salt-based solution for skewed joins
n_salts = 10
# Salt the large (skewed) table
large_salted = large_df.with_columns(
(pl.arange(0, pl.len()) % n_salts).alias("salt")
)
# Replicate the small table for each salt
small_replicated = pl.concat([
small_df.with_columns(pl.lit(i).alias("salt"))
for i in range(n_salts)
])
# Join on key + salt (distributes hot keys)
result = large_salted.join(small_replicated, on=["user_id", "salt"]).drop("salt")
๐ Behind the Question: Handling data skew in distributed and large-scale joins. โ The Red Flags: Not detecting key skew before large joins, leading to memory spikes.
Q262: How do you implement sliding window feature computation for ML?โ
Technical Answer:
Create features from historical windows: rolling aggregations (mean, std, min, max), lag features, and trend indicators. In Polars, use over() with sorted data for per-entity rolling features. Ensure no data leakage by using only past data.
Code/Syntax Justification:
def create_window_features(df: pl.DataFrame, entity_col: str, time_col: str, value_col: str) -> pl.DataFrame:
return df.sort([entity_col, time_col]).with_columns([
# Lag features
pl.col(value_col).shift(1).over(entity_col).alias(f"{value_col}_lag1"),
pl.col(value_col).shift(7).over(entity_col).alias(f"{value_col}_lag7"),
# Rolling statistics
pl.col(value_col).rolling_mean(window_size=7).over(entity_col).alias(f"{value_col}_ma7"),
pl.col(value_col).rolling_std(window_size=7).over(entity_col).alias(f"{value_col}_std7"),
pl.col(value_col).rolling_min(window_size=7).over(entity_col).alias(f"{value_col}_min7"),
pl.col(value_col).rolling_max(window_size=7).over(entity_col).alias(f"{value_col}_max7"),
# Trend: current vs rolling mean
(pl.col(value_col) / pl.col(value_col).rolling_mean(window_size=7).over(entity_col)).alias(f"{value_col}_trend7"),
# Rate of change
(pl.col(value_col) - pl.col(value_col).shift(1).over(entity_col)).alias(f"{value_col}_diff1"),
])
๐ Behind the Question: Time-aware feature engineering for predictive models. โ The Red Flags: Including future data in window features (data leakage).
Q263: How do you implement efficient percentile/quantile calculations at scale?โ
Technical Answer:
Exact percentiles require sorting (O(n log n)). Approximate methods like T-Digest and GK sketches provide O(1) memory estimates. For grouped percentiles, use quantile() in aggregations. NumPy's percentile() and quantile() support multiple quantiles in one pass.
Code/Syntax Justification:
# Multiple quantiles in one pass
quantiles = np.quantile(data, [0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99])
# Per-group quantiles in Polars
percentiles = df.group_by("category").agg([
pl.col("value").quantile(0.5).alias("p50"),
pl.col("value").quantile(0.9).alias("p90"),
pl.col("value").quantile(0.95).alias("p95"),
pl.col("value").quantile(0.99).alias("p99"),
])
# Pandas describe for quick distribution summary
summary = df.groupby("category")["value"].describe(percentiles=[0.1, 0.5, 0.9, 0.99])
# Approximate quantiles for streaming (T-Digest)
from tdigest import TDigest
digest = TDigest()
for chunk in stream:
digest.update(chunk)
print(f"P99: {digest.percentile(99)}")
๐ Behind the Question: Distribution analysis at scale. โ The Red Flags: Computing percentiles with sorted arrays for streaming/approximate use cases.
Q264: How do you implement data encryption and masking in DataFrames?โ
Technical Answer: Mask PII columns with hashing (irreversible), encryption (reversible with key), or tokenization (mapping to random IDs). Apply masking as early as possible in the pipeline. Use deterministic hashing for join-preserving anonymization.
Code/Syntax Justification:
import hashlib
# Deterministic hashing (preserves joins, irreversible)
def hash_pii(series: pl.Series, salt: str = "secret_salt") -> pl.Series:
return series.map_elements(
lambda x: hashlib.sha256(f"{salt}:{x}".encode()).hexdigest()[:16] if x else None,
return_dtype=pl.Utf8,
)
df = df.with_columns([
hash_pii(df["email"]).alias("email_hash"),
hash_pii(df["phone"]).alias("phone_hash"),
])
# Column-level masking
def mask_email(email: str) -> str:
if not email or "@" not in email:
return email
user, domain = email.split("@", 1)
return f"{user[0]}***@{domain}"
df = df.with_columns(
pl.col("email").map_elements(mask_email, return_dtype=pl.Utf8).alias("email_masked")
)
๐ Behind the Question: PII handling and data privacy compliance. โ The Red Flags: Storing PII in analytics pipelines without masking or encryption.
Q265: How do you implement efficient cumulative and running calculations?โ
Technical Answer:
Use cumsum(), cumprod(), cummax(), cummin() for running aggregations. For running distinct counts or percentiles, use window expressions with expanding logic. NumPy's accumulate on ufuncs handles custom running operations.
Code/Syntax Justification:
# Basic cumulative operations
df = df.with_columns([
pl.col("revenue").cum_sum().alias("cumulative_revenue"),
pl.col("revenue").cum_max().alias("running_max"),
(pl.col("revenue").cum_sum() / pl.arange(1, pl.len() + 1)).alias("running_avg"),
])
# Running distinct count (Polars)
df = df.sort("timestamp").with_columns(
pl.col("user_id").cum_count().alias("cumulative_users")
)
# NumPy cumulative with custom ufunc
import numpy as np
values = np.array([10, -3, 5, -8, 12, -2])
running_max = np.maximum.accumulate(values) # [10, 10, 10, 10, 12, 12]
running_min = np.minimum.accumulate(values) # [10, -3, -3, -8, -8, -8]
cumulative_sum = np.add.accumulate(values) # [10, 7, 12, 4, 16, 14]
# Drawdown from peak
peak = np.maximum.accumulate(values)
drawdown = (values - peak) / peak
๐ Behind the Question: Running statistics for financial and operational analytics. โ The Red Flags: Using explicit loops for cumulative calculations.
Q266: How do you implement correlation analysis across large feature sets?โ
Technical Answer:
Use DataFrame.corr() for pairwise Pearson correlation, np.corrcoef() for the correlation matrix, and Spearman/Kendall for non-linear relationships. For large feature sets, compute only upper-triangle pairs and filter by threshold.
Code/Syntax Justification:
# Full correlation matrix
corr_matrix = df.select(pl.col(pl.NUMERIC_DTYPES)).to_pandas().corr()
# Find highly correlated pairs
def find_high_correlations(df: pd.DataFrame, threshold: float = 0.8):
corr = df.corr().abs()
upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(bool))
high_corr = [(col, row, upper.loc[row, col])
for col in upper.columns
for row in upper.index
if upper.loc[row, col] > threshold]
return sorted(high_corr, key=lambda x: -x[2])
# NumPy for large matrices
X = df.select(pl.col(pl.NUMERIC_DTYPES)).to_numpy()
corr = np.corrcoef(X, rowvar=False)
# Spearman rank correlation
from scipy.stats import spearmanr
rho, p_values = spearmanr(X)
๐ Behind the Question: Feature selection and multicollinearity detection. โ The Red Flags: Not checking for highly correlated features before model training.
Q267: How do you implement efficient grouped ranking and top-k operations?โ
Technical Answer:
Use rank() with over() in Polars for per-group ranking, then filter to top-k. Supports different ranking methods: ordinal, min, max, dense, average. For large groups, arg_sort() with slicing is more efficient than full ranking.
Code/Syntax Justification:
# Top 3 products per category by revenue
top_3 = (
df
.with_columns(
pl.col("revenue")
.rank(descending=True, method="ordinal")
.over("category")
.alias("rank")
)
.filter(pl.col("rank") <= 3)
.sort(["category", "rank"])
)
# Pandas equivalent with groupby + nlargest
top_3_pd = df.groupby("category").apply(
lambda g: g.nlargest(3, "revenue")
).reset_index(drop=True)
# Dense ranking (no gaps)
df = df.with_columns(
pl.col("score").rank(method="dense", descending=True).over("team").alias("dense_rank")
)
# Percentile rank
df = df.with_columns(
(pl.col("salary").rank().over("dept") / pl.col("salary").count().over("dept")).alias("pct_rank")
)
๐ Behind the Question: Analytical ranking for leaderboards, reporting, and feature engineering. โ The Red Flags: Sorting entire groups when only top-k is needed.
Q268: How do you implement data imputation strategies beyond simple fill?โ
Technical Answer: Simple: mean/median/mode fill. Better: group-based fill (mean per category), KNN imputation (fill from nearest neighbors), iterative imputation (MICE), or time-series interpolation. The choice depends on missing mechanism: MCAR, MAR, or MNAR.
Code/Syntax Justification:
# Group-based imputation
df = df.with_columns(
pl.col("salary")
.fill_null(pl.col("salary").mean().over("department"))
.alias("salary_imputed")
)
# Forward/backward fill for time-series
df = df.sort("timestamp").with_columns([
pl.col("value").forward_fill().alias("value_ffill"),
pl.col("value").interpolate().alias("value_interp"),
])
# sklearn iterative imputation
from sklearn.impute import IterativeImputer, KNNImputer
# KNN imputation
imputer = KNNImputer(n_neighbors=5)
X_imputed = imputer.fit_transform(X_with_nans)
# MICE (Multiple Imputation by Chained Equations)
mice = IterativeImputer(max_iter=10, random_state=42)
X_mice = mice.fit_transform(X_with_nans)
# Flag imputed values
df = df.with_columns(
pl.col("value").is_null().alias("was_imputed")
)
๐ Behind the Question: Statistically sound missing data handling. โ The Red Flags: Always using mean fill without considering the missing data mechanism.
Q269: How do you implement efficient string tokenization for NLP preprocessing?โ
Technical Answer:
Use vectorized string operations for basic tokenization (split, regex), CountVectorizer/TfidfVectorizer for bag-of-words, and Hugging Face tokenizers for subword tokenization. Process in batches with Polars for large corpora.
Code/Syntax Justification:
# Basic tokenization with Polars
df = df.with_columns([
pl.col("text").str.to_lowercase().alias("text_clean"),
pl.col("text").str.split(" ").alias("tokens"),
pl.col("text").str.split(" ").list.len().alias("word_count"),
pl.col("text").str.replace_all(r"[^\w\s]", "").str.split(" ").alias("clean_tokens"),
])
# sklearn vectorization
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=10000, ngram_range=(1, 2), stop_words="english")
X_tfidf = tfidf.fit_transform(df["text"].to_list())
# NumPy sparse matrix operations
print(f"Shape: {X_tfidf.shape}, Non-zero: {X_tfidf.nnz}, Density: {X_tfidf.nnz / np.prod(X_tfidf.shape):.4%}")
๐ Behind the Question: Text preprocessing for ML and search applications. โ The Red Flags: Processing text one document at a time instead of vectorized batch operations.
Q270: How do you implement efficient pivot + unpivot for ETL transformations?โ
Technical Answer: Pivot converts long-to-wide format; unpivot (melt) converts wide-to-long. In ETL, use unpivot to normalize source data, then pivot for reporting outputs. Both operations should specify exact columns to avoid schema drift.
Code/Syntax Justification:
# Wide to long (unpivot/melt) for normalization
wide = pl.DataFrame({
"id": [1, 2],
"jan_sales": [100, 200],
"feb_sales": [110, 190],
"mar_sales": [120, 210],
})
long = wide.unpivot(
index=["id"],
on=["jan_sales", "feb_sales", "mar_sales"],
variable_name="month",
value_name="sales",
).with_columns(
pl.col("month").str.replace("_sales", "")
)
# Long to wide (pivot) for reporting
report = long.pivot(on="month", index="id", values="sales")
# Pandas melt with multiple value vars
pd_long = pd.melt(
wide_df,
id_vars=["product", "region"],
value_vars=["q1_revenue", "q2_revenue", "q3_revenue", "q4_revenue"],
var_name="quarter",
value_name="revenue",
)
๐ Behind the Question: Data reshaping for normalization and denormalization in pipelines. โ The Red Flags: Manually transposing data with loops instead of using pivot/unpivot.
Q271: How do you implement efficient batch scoring with DataFrames?โ
Technical Answer:
Load the model once, prepare features as a DataFrame, convert to NumPy for prediction, and attach scores back. For large datasets, process in chunks to control memory. Use Polars lazy mode for feature preparation and .to_numpy() at the scoring boundary.
Code/Syntax Justification:
import joblib
import polars as pl
import numpy as np
def batch_score(df: pl.DataFrame, model_path: str, feature_cols: list[str], chunk_size: int = 100_000) -> pl.DataFrame:
model = joblib.load(model_path)
scores = []
for i in range(0, df.height, chunk_size):
chunk = df.slice(i, chunk_size)
X = chunk.select(feature_cols).to_numpy()
preds = model.predict_proba(X)[:, 1]
scores.append(preds)
all_scores = np.concatenate(scores)
return df.with_columns(pl.Series("score", all_scores))
# Usage
scored_df = batch_score(
df=pl.read_parquet("features.parquet"),
model_path="model.joblib",
feature_cols=["feat_1", "feat_2", "feat_3"],
chunk_size=50_000,
)
scored_df.write_parquet("scored_output.parquet")
๐ Behind the Question: ML model inference at scale with DataFrames. โ The Red Flags: Scoring one row at a time or loading the model inside the loop.
Q272: How do you implement data lineage tracking with metadata columns?โ
Technical Answer: Add metadata columns at each pipeline stage: source file, processing timestamp, stage name, and row-level audit fields. This creates a traceable record from raw input to final output without external systems.
Code/Syntax Justification:
from datetime import datetime
def add_lineage(df: pl.DataFrame, stage: str, source: str) -> pl.DataFrame:
return df.with_columns([
pl.lit(source).alias("_source"),
pl.lit(stage).alias("_stage"),
pl.lit(datetime.utcnow().isoformat()).alias("_processed_at"),
pl.arange(0, pl.len()).alias("_row_id"),
])
# Pipeline with lineage
raw = pl.read_parquet("input.parquet")
raw = add_lineage(raw, "extract", "s3://bucket/input.parquet")
cleaned = raw.filter(pl.col("amount") > 0).drop_nulls(subset=["user_id"])
cleaned = add_lineage(cleaned, "clean", "raw")
enriched = cleaned.join(lookup_df, on="user_id")
enriched = add_lineage(enriched, "enrich", "cleaned+lookup")
enriched.write_parquet("output.parquet")
๐ Behind the Question: Audit trail and debugging capability in data pipelines. โ The Red Flags: No ability to trace output rows back to their source data.
Q273: How do you implement efficient column-wise and row-wise operations in NumPy?โ
Technical Answer:
NumPy operations default to element-wise. Use axis=0 for column-wise (across rows), axis=1 for row-wise (across columns). keepdims=True preserves shape for broadcasting. Understanding axis semantics is critical for correct array programming.
Code/Syntax Justification:
data = np.random.rand(1000, 50)
# Column-wise: one result per column (reduce rows)
col_means = data.mean(axis=0) # Shape: (50,)
col_stds = data.std(axis=0) # Shape: (50,)
# Row-wise: one result per row (reduce columns)
row_sums = data.sum(axis=1) # Shape: (1000,)
row_maxes = data.max(axis=1) # Shape: (1000,)
# Keep dimensions for broadcasting
col_means_kd = data.mean(axis=0, keepdims=True) # Shape: (1, 50)
centered = data - col_means_kd # Shape: (1000, 50) โ broadcasted
# Normalize each column (z-score)
normalized = (data - data.mean(axis=0)) / (data.std(axis=0) + 1e-8)
# Normalize each row (L2 norm)
row_norms = np.linalg.norm(data, axis=1, keepdims=True)
unit_vectors = data / (row_norms + 1e-8)
๐ Behind the Question: Axis semantics โ the most common source of NumPy bugs. โ The Red Flags: Confusing axis=0 (column-wise) with axis=1 (row-wise).
Q274: How do you implement efficient matrix operations for recommendation systems?โ
Technical Answer:
Collaborative filtering uses matrix factorization (SVD) on user-item interaction matrices. Cosine similarity between item/user vectors drives recommendations. Sparse matrices (scipy.sparse) are essential since interaction matrices are >99% zeros.
Code/Syntax Justification:
from scipy.sparse import csr_matrix
from scipy.sparse.linalg import svds
from sklearn.metrics.pairwise import cosine_similarity
# Build sparse user-item matrix
rows = interactions["user_idx"].to_numpy()
cols = interactions["item_idx"].to_numpy()
values = interactions["rating"].to_numpy()
matrix = csr_matrix((values, (rows, cols)), shape=(n_users, n_items))
# SVD for dimensionality reduction
U, sigma, Vt = svds(matrix, k=50)
user_factors = U * sigma # User embeddings
item_factors = Vt.T # Item embeddings
# Item-item similarity
item_sim = cosine_similarity(item_factors)
# Predict rating for user u, item i
def predict(u, i):
return np.dot(user_factors[u], item_factors[i])
# Top-k recommendations for a user
def recommend(u, k=10):
scores = user_factors[u] @ item_factors.T
seen = set(matrix[u].indices)
unseen = [i for i in range(n_items) if i not in seen]
top_k = sorted(unseen, key=lambda i: -scores[i])[:k]
return top_k
๐ Behind the Question: Core ML algorithm implementation with NumPy/SciPy. โ The Red Flags: Using dense matrices for sparse user-item interactions.
Q275: How do you implement DuckDB integration for SQL queries over Polars DataFrames?โ
Technical Answer: DuckDB can query Polars DataFrames directly via Arrow zero-copy interop. This combines DuckDB's SQL optimizer with Polars' in-memory representation for complex analytical queries that are easier to express in SQL.
Code/Syntax Justification:
import duckdb
import polars as pl
orders = pl.DataFrame({
"order_id": [1, 2, 3, 4],
"user_id": [10, 20, 10, 30],
"amount": [100, 200, 150, 300],
"date": ["2024-01-01", "2024-01-15", "2024-02-01", "2024-02-15"],
})
# Query Polars DataFrame with DuckDB SQL
result = duckdb.sql("""
SELECT
user_id,
COUNT(*) AS order_count,
SUM(amount) AS total_spent,
AVG(amount) AS avg_order,
MAX(amount) - MIN(amount) AS spend_range
FROM orders
GROUP BY user_id
HAVING COUNT(*) >= 2
ORDER BY total_spent DESC
""").pl() # Returns Polars DataFrame
# Complex window functions
ranked = duckdb.sql("""
SELECT *,
ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY amount DESC) AS rn,
LAG(amount) OVER (PARTITION BY user_id ORDER BY date) AS prev_amount
FROM orders
""").pl()
๐ Behind the Question: SQL + DataFrame interoperability for analytical workflows. โ The Red Flags: Converting to Pandas just to run SQL queries when DuckDB works directly with Polars.
Q276: How do you implement efficient data type inference and conversion?โ
Technical Answer:
Auto-inference (infer_objects, convert_dtypes) catches common patterns but can be wrong for ambiguous data. Explicit dtype specification is more reliable. Use pd.to_numeric(), pd.to_datetime() with errors="coerce" for safe conversion.
Code/Syntax Justification:
# Pandas safe type conversion
df["amount"] = pd.to_numeric(df["amount"], errors="coerce") # Invalid โ NaN
df["date"] = pd.to_datetime(df["date"], errors="coerce", format="mixed")
# Convert all columns to best possible types
df = df.convert_dtypes() # Uses nullable dtypes (Int64, StringDtype, etc.)
# Polars type casting
df_pl = df_pl.with_columns([
pl.col("amount").cast(pl.Float64, strict=False), # Invalid โ null
pl.col("date").str.to_date("%Y-%m-%d", strict=False),
pl.col("id").cast(pl.Int32),
])
# Bulk dtype optimization
def optimize_dtypes(df: pl.DataFrame) -> pl.DataFrame:
for col in df.columns:
if df[col].dtype == pl.Int64:
min_val, max_val = df[col].min(), df[col].max()
if min_val >= 0 and max_val < 256:
df = df.with_columns(pl.col(col).cast(pl.UInt8))
elif -128 <= min_val and max_val < 128:
df = df.with_columns(pl.col(col).cast(pl.Int8))
elif -32768 <= min_val and max_val < 32768:
df = df.with_columns(pl.col(col).cast(pl.Int16))
return df
๐ Behind the Question: Type system management for data quality and performance. โ The Red Flags: Not handling conversion errors or using default dtypes for all columns.
Q277: How do you implement efficient multi-table aggregation pipelines?โ
Technical Answer: Join tables progressively from the most selective filter, aggregate at each level to reduce intermediate sizes, and use lazy execution to let the optimizer reorder. Pre-aggregate fact tables before joining to dimension tables when possible.
Code/Syntax Justification:
# Lazy multi-table pipeline (optimizer reorders)
result = (
pl.scan_parquet("orders.parquet")
.filter(pl.col("date") >= "2024-01-01") # Filter early
.join(
pl.scan_parquet("users.parquet").select(["user_id", "region", "tier"]),
on="user_id",
how="left",
)
.join(
pl.scan_parquet("products.parquet").select(["product_id", "category"]),
on="product_id",
how="left",
)
.group_by(["region", "tier", "category"])
.agg([
pl.col("amount").sum().alias("revenue"),
pl.col("order_id").n_unique().alias("orders"),
pl.col("user_id").n_unique().alias("customers"),
])
.collect()
)
๐ Behind the Question: Multi-table analytical query design. โ The Red Flags: Joining full tables before filtering or aggregating.
Q278: How do you build a complete data analysis report programmatically?โ
Technical Answer: Combine Polars/Pandas for data processing, Matplotlib/Plotly for visualizations, and Jinja2 for HTML templating. Output to HTML or PDF with charts embedded as base64 images. Automate with scheduled runs and email delivery.
Code/Syntax Justification:
import polars as pl
import matplotlib.pyplot as plt
import base64
from io import BytesIO
from jinja2 import Template
def fig_to_base64(fig) -> str:
buf = BytesIO()
fig.savefig(buf, format="png", dpi=100, bbox_inches="tight")
buf.seek(0)
return base64.b64encode(buf.read()).decode()
# Compute metrics
metrics = df.group_by("category").agg([
pl.col("revenue").sum().alias("total_revenue"),
pl.col("orders").sum().alias("total_orders"),
])
# Create chart
fig, ax = plt.subplots(figsize=(10, 6))
ax.bar(metrics["category"].to_list(), metrics["total_revenue"].to_list())
ax.set_title("Revenue by Category")
chart_b64 = fig_to_base64(fig)
plt.close()
# Render HTML report
template = Template("""
<h1>Weekly Revenue Report</h1>
<img src="data:image/png;base64,{{ chart }}">
<table>{% for row in data %}<tr><td>{{ row.category }}</td><td>${{ row.total_revenue:,.0f }}</td></tr>{% endfor %}</table>
""")
html = template.render(chart=chart_b64, data=metrics.iter_rows(named=True))
Path("report.html").write_text(html)
๐ Behind the Question: End-to-end automated reporting workflows. โ The Red Flags: Manual copy-paste from notebooks for recurring reports.
๐งช Section 10: Testing, Debugging & Edge Casesโ
Q279: How do you debug silent data corruption in Pandas pipelines?โ
Technical Answer:
Silent corruption happens when dtype coercion loses precision (int64โfloat64 during merge with NaN), string columns absorb mixed types, or index misalignment causes wrong joins. Insert assertions at stage boundaries and use pd.testing.assert_frame_equal() to compare expected vs actual.
Code/Syntax Justification:
import pandas as pd
import numpy as np
# Detect dtype changes across pipeline stages
def check_schema_drift(before: pd.DataFrame, after: pd.DataFrame, label: str):
for col in before.columns:
if col in after.columns and before[col].dtype != after[col].dtype:
print(f"[{label}] {col}: {before[col].dtype} โ {after[col].dtype}")
# Detect silent NaN introduction
def check_nan_introduction(before: pd.DataFrame, after: pd.DataFrame, label: str):
for col in before.columns:
if col in after.columns:
before_nulls = before[col].isna().sum()
after_nulls = after[col].isna().sum()
if after_nulls > before_nulls:
print(f"[{label}] {col}: {before_nulls} โ {after_nulls} nulls (+{after_nulls - before_nulls})")
# Pipeline with debug checkpoints
df_raw = pd.read_csv("data.csv")
check_schema_drift(df_raw, df_raw, "load")
df_merged = df_raw.merge(lookup, on="key", how="left")
check_nan_introduction(df_raw, df_merged, "merge")
check_schema_drift(df_raw, df_merged, "merge")
๐ Behind the Question: Diagnosing data quality issues in complex pipelines. โ The Red Flags: No assertions or checkpoints between pipeline stages.
Q280: How do you write property-based tests for DataFrame transformations?โ
Technical Answer: Property-based testing with Hypothesis generates random DataFrames and checks invariants: output schema matches expected, row count relationships hold, no unexpected nulls appear, and derived columns satisfy mathematical properties. This catches edge cases unit tests miss.
Code/Syntax Justification:
from hypothesis import given, settings
from hypothesis import strategies as st
from hypothesis.extra.pandas import data_frames, column
import pandas as pd
# Generate random DataFrames with controlled schemas
@given(
df=data_frames([
column("price", dtype=float, elements=st.floats(min_value=0, max_value=10000, allow_nan=False)),
column("quantity", dtype=int, elements=st.integers(min_value=0, max_value=100)),
], index=st.just(pd.RangeIndex(10)))
)
@settings(max_examples=200)
def test_revenue_is_non_negative(df):
result = df.assign(revenue=df["price"] * df["quantity"])
assert (result["revenue"] >= 0).all()
assert len(result) == len(df) # Row count preserved
@given(
df=data_frames([
column("value", dtype=float, elements=st.floats(allow_nan=True, allow_infinity=False)),
])
)
def test_fill_null_removes_all_nulls(df):
result = df.fillna(0)
assert result["value"].isna().sum() == 0
๐ Behind the Question: Exhaustive testing without manually writing edge cases. โ The Red Flags: Only testing with handcrafted example data that doesn't cover edge cases.
Q281: How do you handle the SettingWithCopyWarning in Pandas?โ
Technical Answer:
This warning fires when modifying a slice that might be a view or a copy. Fix by using .loc[] for explicit assignment, .copy() when you intentionally want a copy, or method chaining with .assign(). In Pandas 3.0, Copy-on-Write will eliminate this entirely.
Code/Syntax Justification:
# BAD: ambiguous โ view or copy?
subset = df[df["age"] > 25]
subset["status"] = "senior" # SettingWithCopyWarning!
# GOOD: explicit loc assignment
df.loc[df["age"] > 25, "status"] = "senior"
# GOOD: explicit copy when you want independence
subset = df[df["age"] > 25].copy()
subset["status"] = "senior" # No warning โ it's a copy
# GOOD: method chaining with assign
result = df.query("age > 25").assign(status="senior")
# Enable Copy-on-Write (Pandas 2.0+, default in 3.0)
pd.options.mode.copy_on_write = True
๐ Behind the Question: Understanding Pandas' copy semantics and avoiding silent bugs. โ The Red Flags: Ignoring SettingWithCopyWarning or suppressing it without understanding the cause.
Q282: How do you handle mixed-type columns that Pandas infers as object dtype?โ
Technical Answer:
Object dtype columns arise from mixed integers/strings, mixed numeric formats, or partial parsing. Use pd.to_numeric(errors="coerce") to force numeric, infer_objects() to re-infer, or explicit type mapping. Investigate the non-conforming values first.
Code/Syntax Justification:
# Diagnose mixed types
def diagnose_mixed_types(series: pd.Series) -> dict:
type_counts = series.dropna().apply(type).value_counts()
return {
"dtype": str(series.dtype),
"type_distribution": type_counts.to_dict(),
"sample_non_numeric": series[pd.to_numeric(series, errors="coerce").isna()].head(5).tolist(),
}
# Force numeric, coerce failures to NaN
df["amount"] = pd.to_numeric(df["amount"], errors="coerce")
# Clean then convert
df["amount"] = (
df["amount"]
.astype(str)
.str.replace(r"[$,]", "", regex=True)
.str.strip()
.pipe(pd.to_numeric, errors="coerce")
)
# Polars strict casting surfaces errors immediately
df_pl.with_columns(pl.col("amount").cast(pl.Float64, strict=True)) # Raises on invalid
๐ Behind the Question: Data cleaning for real-world messy inputs. โ The Red Flags: Leaving object dtype columns without investigation.
Q283: How do you handle integer columns with NaN values in Pandas?โ
Technical Answer:
Standard NumPy integers can't hold NaN, so Pandas upcasts to float64 when NaN is present. Use Pandas nullable integer types (Int64, Int32, etc.) to keep integer semantics with proper NA support. This preserves precision for IDs, counts, and foreign keys.
Code/Syntax Justification:
# Problem: NaN forces float conversion
df = pd.DataFrame({"id": [1, 2, None, 4]})
print(df["id"].dtype) # float64 โ lost integer precision!
# Solution: nullable integer type
df = pd.DataFrame({"id": pd.array([1, 2, None, 4], dtype="Int64")})
print(df["id"].dtype) # Int64 โ nullable integer
print(df["id"][2]) # <NA> โ not NaN
# Convert existing float-with-nans to nullable integer
df["id"] = df["id"].astype("Int64")
# Read CSV with nullable integers
df = pd.read_csv("data.csv", dtype={"user_id": "Int64", "order_count": "Int32"})
# Polars handles this natively โ integers are always nullable
df_pl = pl.DataFrame({"id": [1, 2, None, 4]})
print(df_pl["id"].dtype) # Int64 with null support built-in
๐ Behind the Question: Type system pitfalls in Pandas for ID/count columns. โ The Red Flags: Allowing foreign key columns to become float64 due to NaN.
Q284: How do you handle extremely large string columns efficiently?โ
Technical Answer:
Large string columns dominate memory. Strategies: convert to categorical if cardinality is low, use Arrow-backed strings (string[pyarrow]), truncate unnecessary length, or store as dictionary-encoded Parquet. Profile memory per column to find the worst offenders.
Code/Syntax Justification:
# Profile memory per column
mem_usage = df.memory_usage(deep=True).sort_values(ascending=False)
print(mem_usage.head(10))
# Convert low-cardinality strings to categorical
for col in df.select_dtypes("object").columns:
nunique = df[col].nunique()
total = len(df)
if nunique / total < 0.05: # Less than 5% unique
df[col] = df[col].astype("category")
print(f" {col}: {nunique} unique values โ category")
# Arrow-backed strings (Pandas 2.0+)
df["description"] = df["description"].astype("string[pyarrow]")
# Truncate overly long strings
df["notes"] = df["notes"].str[:500] # Keep first 500 chars
# Polars: strings are Arrow-backed by default โ already efficient
๐ Behind the Question: Memory optimization for text-heavy datasets. โ The Red Flags: Not profiling per-column memory usage for DataFrames with string columns.
Q285: How do you handle date parsing edge cases in real-world data?โ
Technical Answer:
Real data has mixed formats ("2024-01-15", "01/15/2024", "Jan 15, 2024"), timezone inconsistencies, and invalid dates. Use pd.to_datetime() with format="mixed", dayfirst/yearfirst flags, and errors="coerce" for partial parsing. Always validate parsed dates are in expected range.
Code/Syntax Justification:
# Mixed format parsing
df["date"] = pd.to_datetime(df["date_str"], format="mixed", dayfirst=False)
# Multiple known formats with fallback
def parse_date_robust(series: pd.Series) -> pd.Series:
formats = ["%Y-%m-%d", "%m/%d/%Y", "%d-%b-%Y", "%Y-%m-%dT%H:%M:%S"]
result = pd.Series(pd.NaT, index=series.index)
for fmt in formats:
mask = result.isna() & series.notna()
result[mask] = pd.to_datetime(series[mask], format=fmt, errors="coerce")
return result
# Validate parsed dates
df["date"] = pd.to_datetime(df["date_str"], errors="coerce")
invalid_dates = df[df["date"].isna() & df["date_str"].notna()]
print(f"Failed to parse: {len(invalid_dates)} rows")
# Range validation
future_dates = df[df["date"] > pd.Timestamp.now()]
ancient_dates = df[df["date"] < pd.Timestamp("2000-01-01")]
๐ Behind the Question: Robust date parsing for messy real-world data. โ The Red Flags: Using a single format string and silently dropping unparseable dates.
Q286: How do you handle memory errors when working with large DataFrames?โ
Technical Answer:
Strategies: (1) read only needed columns with usecols, (2) specify minimal dtypes upfront, (3) use chunked reading with chunksize, (4) switch to Polars lazy/streaming mode, (5) use memory-mapped files, (6) sample for exploration then process in chunks for production.
Code/Syntax Justification:
import sys
# Check memory before loading
file_size_gb = Path("large.csv").stat().st_size / 1e9
available_ram = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 1e9
print(f"File: {file_size_gb:.1f} GB, RAM: {available_ram:.1f} GB")
# Strategy 1: minimal columns and dtypes
df = pd.read_csv("large.csv",
usecols=["id", "amount", "date"],
dtype={"id": "int32", "amount": "float32"},
parse_dates=["date"],
)
# Strategy 2: chunked processing
result_chunks = []
for chunk in pd.read_csv("large.csv", chunksize=100_000):
processed = chunk.query("amount > 100").groupby("category")["amount"].sum()
result_chunks.append(processed)
result = pd.concat(result_chunks).groupby(level=0).sum()
# Strategy 3: Polars streaming (best for large files)
result = (
pl.scan_csv("large.csv")
.filter(pl.col("amount") > 100)
.group_by("category")
.agg(pl.col("amount").sum())
.collect(streaming=True)
)
๐ Behind the Question: Handling datasets larger than available RAM.
โ The Red Flags: Calling pd.read_csv() on a file larger than available RAM without any memory strategy.
Q287: How do you debug incorrect merge/join results?โ
Technical Answer:
Common join bugs: duplicate keys causing row multiplication, mismatched dtypes (int vs string), trailing whitespace in string keys, and NaN/null key values (which never match). Use indicator=True to trace which table contributed each row.
Code/Syntax Justification:
# Pre-join diagnostics
def diagnose_join(left, right, on):
print(f"Left: {len(left)} rows, Right: {len(right)} rows")
print(f"Left key nulls: {left[on].isna().sum()}")
print(f"Right key nulls: {right[on].isna().sum()}")
print(f"Left key dtype: {left[on].dtype}, Right key dtype: {right[on].dtype}")
print(f"Left unique keys: {left[on].nunique()}, Right unique keys: {right[on].nunique()}")
print(f"Left duplicates: {left[on].duplicated().sum()}")
print(f"Right duplicates: {right[on].duplicated().sum()}")
# Join with indicator
merged = left.merge(right, on="key", how="outer", indicator=True)
print(merged["_merge"].value_counts())
# both โ matched
# left_only โ in left but not right
# right_only โ in right but not left
# Detect row multiplication
expected_rows = len(left)
actual_rows = len(merged)
if actual_rows > expected_rows * 1.1:
print(f"WARNING: Row multiplication! {expected_rows} โ {actual_rows}")
dup_keys = right[right[on].duplicated(keep=False)][on].unique()
print(f"Duplicate keys in right table: {dup_keys[:10]}")
๐ Behind the Question: Join debugging methodology. โ The Red Flags: Not checking row counts before and after joins.
Q288: How do you handle Pandas' index alignment behavior?โ
Technical Answer:
Pandas automatically aligns on index during operations between Series/DataFrames. Misaligned indices produce NaN for non-matching positions. This is a common source of silent bugs. Use .values or .to_numpy() to bypass alignment, or reset_index() to ensure matching.
Code/Syntax Justification:
# Silent bug: index misalignment
s1 = pd.Series([10, 20, 30], index=[0, 1, 2])
s2 = pd.Series([1, 2, 3], index=[1, 2, 3])
result = s1 + s2 # NaN at index 0 and 3!
# 0 NaN
# 1 21.0
# 2 32.0
# 3 NaN
# Fix: reset index or use values
result_correct = s1.values + s2.values # [11, 22, 33] โ positional
# Fix: explicit alignment
s1_aligned, s2_aligned = s1.align(s2, fill_value=0)
# Common trap: filtered DataFrame operations
df_a = df[df["status"] == "active"]
df_b = df[df["status"] == "inactive"]
# df_a["score"] + df_b["score"] โ Will align on original index, not position!
# Polars: no index alignment โ purely positional
# This is simpler but requires you to ensure row correspondence yourself
๐ Behind the Question: Understanding index-based alignment โ Pandas' most surprising behavior. โ The Red Flags: Not resetting index before positional operations on filtered DataFrames.
Q289: How do you test DataFrame equality with tolerance for floating-point values?โ
Technical Answer:
Use pd.testing.assert_frame_equal() with check_exact=False and atol/rtol parameters. For Polars, use polars.testing.assert_frame_equal(). Always compare schemas separately from values. Handle NaN equality explicitly.
Code/Syntax Justification:
import pandas as pd
from polars.testing import assert_frame_equal as pl_assert_equal
# Pandas: approximate equality
pd.testing.assert_frame_equal(
actual, expected,
check_exact=False,
atol=1e-6,
rtol=1e-5,
check_dtype=True,
check_names=True,
)
# Polars: approximate equality
pl_assert_equal(actual_pl, expected_pl, atol=1e-6, rtol=1e-5)
# Custom comparison for complex cases
def compare_dataframes(actual, expected, tolerance=1e-6):
assert actual.shape == expected.shape, f"Shape mismatch: {actual.shape} vs {expected.shape}"
assert list(actual.columns) == list(expected.columns), "Column mismatch"
for col in actual.columns:
if pd.api.types.is_numeric_dtype(actual[col]):
np.testing.assert_allclose(
actual[col].to_numpy(),
expected[col].to_numpy(),
atol=tolerance,
equal_nan=True,
)
else:
pd.testing.assert_series_equal(actual[col], expected[col])
๐ Behind the Question: Correct testing methodology for numerical DataFrames.
โ The Red Flags: Using == for floating-point DataFrame comparison or not handling NaN in comparisons.
Q290: How do you handle chained assignment and the Pandas Copy-on-Write mode?โ
Technical Answer:
Copy-on-Write (CoW), default in Pandas 3.0, makes all indexing operations return views that are lazily copied only on mutation. This eliminates SettingWithCopyWarning, reduces memory usage, and makes the copy semantics deterministic. Enable early with pd.options.mode.copy_on_write = True.
Code/Syntax Justification:
# Enable Copy-on-Write (recommended for Pandas 2.x)
pd.options.mode.copy_on_write = True
# With CoW: indexing always returns a view, copy happens on write
df2 = df[["a", "b"]] # View โ no memory allocated
df2["a"] = 0 # NOW a copy is made โ df is unchanged
# Without CoW: behavior was unpredictable
# Sometimes a view (mutating df), sometimes a copy (not mutating df)
# CoW-compatible patterns
# GOOD: use loc for in-place modification
df.loc[df["a"] > 5, "b"] = 0
# GOOD: use assign for new columns
df = df.assign(c=df["a"] + df["b"])
# GOOD: method chaining (always safe)
result = df.query("a > 5").assign(d=lambda x: x["a"] * 2)
๐ Behind the Question: Modern Pandas memory model and migration to CoW. โ The Red Flags: Not enabling CoW in Pandas 2.x or relying on implicit copy/view behavior.
Q291: How do you benchmark and compare different data processing approaches?โ
Technical Answer:
Use timeit for micro-benchmarks, memory_profiler for memory, and line_profiler for hot-path identification. Compare the same logical operation across Pandas, Polars, and NumPy with identical data. Report median time (not mean) and include memory alongside speed.
Code/Syntax Justification:
import timeit
import tracemalloc
from dataclasses import dataclass
@dataclass
class BenchResult:
name: str
time_ms: float
peak_memory_mb: float
def benchmark(name: str, func, n_runs: int = 5) -> BenchResult:
# Warm up
func()
# Time
times = timeit.repeat(func, number=1, repeat=n_runs)
median_time = sorted(times)[n_runs // 2]
# Memory
tracemalloc.start()
func()
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
return BenchResult(name, median_time * 1000, peak / 1e6)
# Compare approaches
data = np.random.rand(1_000_000)
results = [
benchmark("numpy_sort", lambda: np.sort(data.copy())),
benchmark("pandas_sort", lambda: pd.Series(data).sort_values()),
benchmark("polars_sort", lambda: pl.Series(data).sort()),
]
for r in results:
print(f"{r.name}: {r.time_ms:.1f}ms, {r.peak_memory_mb:.1f}MB")
๐ Behind the Question: Rigorous performance comparison methodology.
โ The Red Flags: Benchmarking with time.time() on a single run without memory measurement.
Q292: How do you handle unicode and encoding issues in text data?โ
Technical Answer:
Specify encoding explicitly when reading files. Common encodings: UTF-8 (default), Latin-1 (ISO-8859-1), Windows-1252. Use errors="replace" or errors="ignore" for malformed bytes. Normalize unicode with unicodedata.normalize() for consistent comparison.
Code/Syntax Justification:
import unicodedata
# Read with explicit encoding
df = pd.read_csv("data.csv", encoding="utf-8")
# Fallback for unknown encoding
try:
df = pd.read_csv("data.csv", encoding="utf-8")
except UnicodeDecodeError:
df = pd.read_csv("data.csv", encoding="latin-1")
# Auto-detect encoding
import chardet
with open("data.csv", "rb") as f:
result = chardet.detect(f.read(100_000))
print(f"Encoding: {result['encoding']}, Confidence: {result['confidence']}")
# Normalize unicode for consistent comparison
def normalize_text(text: str) -> str:
return unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode()
df["name_clean"] = df["name"].apply(normalize_text)
# Polars: specify encoding
df_pl = pl.read_csv("data.csv", encoding="utf8-lossy")
๐ Behind the Question: Text encoding issues in international datasets. โ The Red Flags: Assuming all files are UTF-8 or ignoring encoding errors silently.
Q293: How do you implement snapshot testing for data pipeline outputs?โ
Technical Answer: Snapshot testing saves a known-good output and compares future runs against it. Useful for regression testing complex transformations. Store snapshots as Parquet (preserves dtypes) with a hash-based naming scheme. Update snapshots explicitly when the transformation intentionally changes.
Code/Syntax Justification:
from pathlib import Path
import polars as pl
SNAPSHOT_DIR = Path("tests/snapshots")
def assert_snapshot(df: pl.DataFrame, name: str, update: bool = False):
path = SNAPSHOT_DIR / f"{name}.parquet"
if update or not path.exists():
SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
df.write_parquet(str(path))
print(f"Snapshot {'updated' if path.exists() else 'created'}: {name}")
return
expected = pl.read_parquet(str(path))
try:
from polars.testing import assert_frame_equal
assert_frame_equal(df, expected, check_exact=False, atol=1e-6)
except AssertionError as e:
print(f"Snapshot mismatch for {name}!")
# Show diff
if df.shape != expected.shape:
print(f" Shape: {df.shape} vs {expected.shape}")
raise
# Usage in tests
def test_feature_pipeline():
result = run_feature_pipeline(sample_data)
assert_snapshot(result, "feature_pipeline_output")
๐ Behind the Question: Regression testing for data transformations. โ The Red Flags: No automated tests for pipeline output correctness.
Q294: How do you handle numeric precision issues in financial calculations?โ
Technical Answer:
Floating-point arithmetic produces rounding errors. For financial calculations, use Python Decimal, integer arithmetic in cents/basis points, or pandas.Decimal extension. Never compare financial values with ==; use tolerance-based comparison.
Code/Syntax Justification:
from decimal import Decimal, ROUND_HALF_UP
# Problem: floating-point precision
print(0.1 + 0.2) # 0.30000000000000004
print(0.1 + 0.2 == 0.3) # False!
# Solution 1: Integer arithmetic (cents)
price_cents = 1999 # $19.99
tax_cents = round(price_cents * 0.08) # 8% tax
total_cents = price_cents + tax_cents
# Solution 2: Decimal for exact arithmetic
price = Decimal("19.99")
tax = (price * Decimal("0.08")).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP)
total = price + tax # Decimal('21.59') โ exact
# Solution 3: Round after operations, not during
df["total"] = (df["price"] * df["quantity"]).round(2)
# Polars: round at end
df_pl = df_pl.with_columns(
(pl.col("price") * pl.col("qty")).round(2).alias("total")
)
๐ Behind the Question: Financial computation correctness.
โ The Red Flags: Using float64 for currency and comparing with ==.
Q295: How do you implement DataFrame schema validation with Pandera?โ
Technical Answer: Pandera defines DataFrame schemas as Python classes with type annotations, value constraints, and custom validators. It integrates with Pandas, Polars, and pytest for runtime schema enforcement and test-time validation.
Code/Syntax Justification:
import pandera as pa
from pandera import Column, Check, DataFrameSchema
# Schema definition
order_schema = DataFrameSchema({
"order_id": Column(int, Check.gt(0), unique=True, nullable=False),
"user_id": Column(int, Check.gt(0), nullable=False),
"amount": Column(float, Check.in_range(0, 1_000_000), nullable=False),
"status": Column(str, Check.isin(["pending", "shipped", "delivered", "cancelled"])),
"created_at": Column("datetime64[ns]", nullable=False),
})
# Validate
try:
validated = order_schema.validate(df)
except pa.errors.SchemaError as e:
print(f"Validation failed: {e}")
# Decorator-based validation
@pa.check_input(order_schema)
@pa.check_output(order_schema.add_columns({"total": Column(float, Check.ge(0))}))
def process_orders(df: pd.DataFrame) -> pd.DataFrame:
return df.assign(total=df["amount"] * 1.08)
๐ Behind the Question: Declarative schema validation for data quality. โ The Red Flags: No schema validation at pipeline boundaries.
Q296: How do you implement efficient error handling in batch data processing?โ
Technical Answer:
Separate valid and invalid records instead of failing the entire batch. Log errors with context (row index, column, value, error type). Use errors="coerce" for type conversion, filter invalid rows to an error table, and process valid rows normally.
Code/Syntax Justification:
def robust_transform(df: pl.DataFrame) -> tuple[pl.DataFrame, pl.DataFrame]:
"""Returns (valid_df, error_df) tuple."""
# Attempt transformations, mark failures
result = df.with_columns([
pl.col("amount").cast(pl.Float64, strict=False).alias("amount_parsed"),
pl.col("date").str.to_date("%Y-%m-%d", strict=False).alias("date_parsed"),
])
# Separate valid from invalid
valid_mask = (
result["amount_parsed"].is_not_null() &
result["date_parsed"].is_not_null()
)
valid = result.filter(valid_mask).drop(["amount", "date"]).rename({
"amount_parsed": "amount",
"date_parsed": "date",
})
errors = result.filter(~valid_mask).with_columns([
pl.when(pl.col("amount_parsed").is_null())
.then(pl.lit("invalid_amount"))
.when(pl.col("date_parsed").is_null())
.then(pl.lit("invalid_date"))
.otherwise(pl.lit("unknown"))
.alias("error_reason")
])
return valid, errors
๐ Behind the Question: Fault-tolerant data processing patterns. โ The Red Flags: Failing entire batches on single-row errors or silently dropping invalid data.
Q297: How do you handle large categorical columns with unseen values at inference time?โ
Technical Answer:
At training time, build a category map. At inference time, map known categories and assign an "unknown" bucket for unseen values. Use set_categories() in Pandas or Enum type in Polars with a fallback. This prevents crashes on new data.
Code/Syntax Justification:
# Training: build category map
known_categories = df_train["product_type"].unique().tolist()
# Inference: handle unseen values
def safe_categorize(series: pd.Series, known: list[str]) -> pd.Series:
result = series.where(series.isin(known), other="__unknown__")
return result.astype(pd.CategoricalDtype(categories=known + ["__unknown__"]))
df_test["product_type"] = safe_categorize(df_test["product_type"], known_categories)
# Polars Enum for strict categories
product_enum = pl.Enum(known_categories)
# Strict mode raises on unknown
# Lenient: cast unknown to null
df_test = df_test.with_columns(
pl.col("product_type").cast(product_enum, strict=False)
)
# Null values are unseen categories
unseen = df_test.filter(pl.col("product_type").is_null())
๐ Behind the Question: Production ML robustness for categorical features. โ The Red Flags: Training pipelines that crash on unseen categorical values in production.
Q298: How do you implement data-aware logging for pipeline debugging?โ
Technical Answer: Log DataFrame statistics at each stage: shape, null counts, value distributions, and sample rows. Use structured logging with stage identifiers. In production, log to a metrics system for monitoring data pipeline health.
Code/Syntax Justification:
import logging
import json
logger = logging.getLogger("pipeline")
def log_stage(df: pl.DataFrame, stage: str) -> pl.DataFrame:
"""Log DataFrame stats without modifying it."""
stats = {
"stage": stage,
"rows": df.height,
"cols": df.width,
"null_counts": {col: df[col].null_count() for col in df.columns if df[col].null_count() > 0},
"memory_mb": round(df.estimated_size("mb"), 2),
}
logger.info(json.dumps(stats))
return df
# Pipeline with logging
df = pl.read_parquet("input.parquet")
df = log_stage(df, "extract")
df = df.filter(pl.col("status") == "active")
df = log_stage(df, "filter_active")
df = df.join(lookup, on="key", how="left")
df = log_stage(df, "join_lookup")
df = df.group_by("category").agg(pl.col("amount").sum())
df = log_stage(df, "aggregate")
๐ Behind the Question: Observability in data pipelines. โ The Red Flags: No logging of DataFrame statistics between pipeline stages.
Q299: How do you handle daylight saving time transitions in time-series data?โ
Technical Answer:
DST transitions create duplicate timestamps (fall back) and missing timestamps (spring forward). Store in UTC, convert to local only for display. Use tz_localize(ambiguous="NaT") or nonexistent="shift_forward" to handle edge cases explicitly.
Code/Syntax Justification:
# Spring forward: 2:00 AM doesn't exist (jumps to 3:00 AM)
try:
pd.Timestamp("2024-03-10 02:30:00").tz_localize("US/Eastern")
except Exception as e:
print(f"NonExistentTimeError: {e}")
# Handle nonexistent times
ts = pd.Timestamp("2024-03-10 02:30:00")
ts_safe = ts.tz_localize("US/Eastern", nonexistent="shift_forward")
# Fall back: 1:00 AM exists twice
ts_ambiguous = pd.Timestamp("2024-11-03 01:30:00")
ts_first = ts_ambiguous.tz_localize("US/Eastern", ambiguous=True) # EDT
ts_second = ts_ambiguous.tz_localize("US/Eastern", ambiguous=False) # EST
# Best practice: store as UTC, convert on display
df["event_time_utc"] = pd.to_datetime(df["event_time"]).dt.tz_localize("UTC")
df["event_time_local"] = df["event_time_utc"].dt.tz_convert("US/Eastern")
# Polars: explicit DST handling
df_pl = df_pl.with_columns(
pl.col("timestamp").dt.replace_time_zone("UTC").dt.convert_time_zone("US/Eastern")
)
๐ Behind the Question: Correct temporal data handling for global applications. โ The Red Flags: Storing local times without UTC conversion or ignoring DST edge cases.
Q300: How do you implement efficient array manipulation for image processing with NumPy?โ
Technical Answer: Images are 3D arrays (height ร width ร channels) or 4D batches. Use slicing for cropping, transpose for channel reordering, and vectorized operations for pixel-level transforms. Avoid Python loops over pixels โ use broadcasting and fancy indexing.
Code/Syntax Justification:
# Image as NumPy array: (H, W, C)
image = np.random.randint(0, 256, (480, 640, 3), dtype=np.uint8)
# Crop
cropped = image[100:400, 200:500]
# Resize (simple nearest-neighbor)
scale = 0.5
h, w = image.shape[:2]
indices_y = (np.arange(int(h * scale)) / scale).astype(int)
indices_x = (np.arange(int(w * scale)) / scale).astype(int)
resized = image[np.ix_(indices_y, indices_x)]
# Grayscale conversion (vectorized)
gray = np.dot(image[..., :3], [0.2989, 0.5870, 0.1140]).astype(np.uint8)
# Normalize to [0, 1] float32 for ML
normalized = image.astype(np.float32) / 255.0
# Batch operations (B, H, W, C)
batch = np.random.rand(32, 224, 224, 3).astype(np.float32)
batch_normalized = (batch - batch.mean(axis=(1, 2), keepdims=True)) / (batch.std(axis=(1, 2), keepdims=True) + 1e-8)
# Channel reorder: HWC โ CHW (for PyTorch)
chw = np.transpose(image, (2, 0, 1))
๐ Behind the Question: NumPy for computer vision preprocessing. โ The Red Flags: Iterating over pixels with Python loops instead of vectorized array operations.
Q301: How do you implement efficient sparse matrix operations for graph and network data?โ
Technical Answer:
Use scipy.sparse for adjacency matrices, transition matrices, and feature matrices. CSR format is efficient for row slicing and matrix-vector products. CSC is better for column slicing. COO is best for construction from edge lists.
Code/Syntax Justification:
from scipy.sparse import csr_matrix, coo_matrix
from scipy.sparse.linalg import eigsh
import numpy as np
# Build adjacency matrix from edge list
edges = np.array([[0, 1], [1, 2], [2, 3], [0, 3], [1, 3]])
n_nodes = 4
row, col = edges[:, 0], edges[:, 1]
# COO for construction (fast)
adj_coo = coo_matrix((np.ones(len(edges)), (row, col)), shape=(n_nodes, n_nodes))
# CSR for computation (fast row access, matmul)
adj = adj_coo.tocsr()
adj = adj + adj.T # Make symmetric (undirected)
# Degree matrix
degrees = np.array(adj.sum(axis=1)).flatten()
# PageRank-style random walk
transition = csr_matrix(adj / degrees[:, None])
pagerank = np.ones(n_nodes) / n_nodes
for _ in range(100):
pagerank = 0.85 * transition.T @ pagerank + 0.15 / n_nodes
# Spectral analysis
eigenvalues, eigenvectors = eigsh(adj.astype(float), k=3)
๐ Behind the Question: Graph algorithms with sparse linear algebra. โ The Red Flags: Using dense matrices for sparse graph data.
Q302: How do you implement data-driven alerting based on statistical process control?โ
Technical Answer: Use control charts (moving average ยฑ k ร standard deviation) to detect when metrics go out of bounds. Implement Shewhart rules (single point beyond 3ฯ), Western Electric rules (multiple points near 2ฯ), and CUSUM for detecting small persistent shifts.
Code/Syntax Justification:
def detect_anomalies(series: pl.Series, window: int = 30, sigma: float = 3.0) -> pl.DataFrame:
df = pl.DataFrame({"value": series})
result = df.with_columns([
pl.col("value").rolling_mean(window_size=window).alias("rolling_mean"),
pl.col("value").rolling_std(window_size=window).alias("rolling_std"),
]).with_columns([
(pl.col("value") > pl.col("rolling_mean") + sigma * pl.col("rolling_std")).alias("upper_breach"),
(pl.col("value") < pl.col("rolling_mean") - sigma * pl.col("rolling_std")).alias("lower_breach"),
((pl.col("value") - pl.col("rolling_mean")) / pl.col("rolling_std")).alias("z_score"),
])
return result
# CUSUM for small shift detection
def cusum(values: np.ndarray, target: float, threshold: float = 5.0, drift: float = 0.5):
s_pos = np.zeros(len(values))
s_neg = np.zeros(len(values))
for i in range(1, len(values)):
s_pos[i] = max(0, s_pos[i-1] + values[i] - target - drift)
s_neg[i] = min(0, s_neg[i-1] + values[i] - target + drift)
alarms = (s_pos > threshold) | (s_neg < -threshold)
return alarms
๐ Behind the Question: Statistical monitoring for operational metrics. โ The Red Flags: Using fixed thresholds instead of statistical process control for alerting.
Q303: What is the most critical performance principle when choosing between Pandas, NumPy, and Polars?โ
Technical Answer:
The most critical principle is match the abstraction level to your data shape and operation. NumPy: use when data is a homogeneous numeric array and operations are mathematical. Pandas: use for labeled heterogeneous tabular data with rich ecosystem needs. Polars: use when processing speed and memory efficiency are primary constraints on tabular data. Within each library, prefer native vectorized operations over Python-level iteration. The performance hierarchy is: native expression > vectorized function > map_batches > map_elements > Python loop.
Code/Syntax Justification:
# Performance hierarchy (fastest โ slowest):
# 1. Native expression (Rust/C speed)
df.with_columns((pl.col("a") + pl.col("b")).alias("c")) # Polars
df["c"] = df["a"] + df["b"] # Pandas
result = a + b # NumPy
# 2. Vectorized function
df.with_columns(pl.col("a").log().alias("log_a")) # Polars
df["log_a"] = np.log(df["a"]) # Pandas + NumPy
# 3. map_batches (Python function on full column)
df.with_columns(pl.col("a").map_batches(custom_vectorized_fn))
# 4. map_elements (Python function per element โ SLOW)
df.with_columns(pl.col("a").map_elements(scalar_fn))
# 5. Python loop (NEVER for large data)
for i in range(len(df)): # DON'T DO THIS
process(df[i])
๐ Behind the Question: The fundamental performance mental model for data engineering. โ The Red Flags: Choosing a library or approach without understanding the performance hierarchy.
Senior / Principal โ Real World Data Science Scenariosโ
Q304: Pandas memory blowup on large CSVโ
Level: Senior
Scenario: A Pandas script reads a 10GB CSV. The machine has 16GB RAM. The process is OOM-killed. The script worked on a 1GB sample.
Impact: Job fails. Data pipeline breaks. Can't process production-sized files.
Root Cause: Pandas loads the entire CSV into memory as a DataFrame. Data types default to object (strings) and float64, using 5-10x more memory than the raw file size.
Simple explanation: You try to fit a 10-gallon water tank into a 2-gallon bucket. It overflows (OOM).
Technical Answer:
- Use chunked reading:
pd.read_csv(..., chunksize=100_000)and process in batches. - Specify
dtypeto avoidobject/float64bloat (e.g.,categoryfor low-cardinality strings,int32instead ofint64). - Use Polars or DuckDB for larger-than-memory (lazy evaluation, streaming, out-of-core).
- Use PyArrow backend for Parquet instead of CSV.
- Drop unused columns early (
usecols).
Code/Syntax Justification:
# Chunked reading
for chunk in pd.read_csv("large.csv", chunksize=100_000, usecols=["id", "amount"], dtype={"amount": "float32"}):
process(chunk)
# Polars (lazy, streaming)
import polars as pl
df = pl.scan_csv("large.csv").select(["id", "amount"]).collect(streaming=True)
Workflow Diagram:
10GB CSV into Pandas DataFrame: Chunked / Polars:
load entire file into RAM read 100k rows at a time
| |
v v
16GB RAM exceeded -> OOM memory bounded
๐ Behind the Question: Memory management, chunked reading, dtype optimization, Polars/DuckDB for large data.
โ The Red Flags: read_csv without chunksize or dtype; loading entire large file; Pandas for >RAM data.
Q305: Slow Pandas operation โ row-by-row Python loopsโ
Level: Principal
Scenario: A Pandas operation on 5 million rows takes 10 minutes. Profiling shows 99% time in a Python for loop over df.iterrows().
Impact: Data pipeline SLA missed. Dashboard stale. Can't scale to more data.
Root Cause: iterrows() / row-by-row Python loops defeat vectorization. Pandas is optimized for vectorized (C/NumPy) operations, not Python loops.
Simple explanation: Adding up a column by tapping each cell on a calculator one at a time instead of using SUM.
Technical Answer:
- Vectorize: use built-in Pandas/NumPy operations (
df['a'] + df['b'],df.groupby().agg()). - Use
df.apply()only as last resort; prefer vectorized. - For row-wise logic that can't vectorize, use Numba (
@njit) or Polars. - Profile with
line_profilerto find the hot loop before optimizing.
Code/Syntax Justification:
# Slow: row-by-row
for _, row in df.iterrows():
df.at[_, "total"] = row["price"] * row["qty"]
# Fast: vectorized
df["total"] = df["price"] * df["qty"]
Workflow Diagram:
iterrows() on 5M rows: Vectorized:
Python loop per row Single C/NumPy operation over entire column
| |
v v
10 minutes < 1 second
๐ Behind the Question: Vectorization, iterrows avoidance, profiling, Numba/Polars alternatives.
โ The Red Flags: iterrows() on large DataFrames; row-by-row Python loops; no profiling before optimizing.