Apache Spark interview questions

27 questions from the Apache Spark: Big Data Processing with DataFrames course, each with the correct answer and an explanation. Try to answer before you open the answer.

Spark Basics

1. What is Spark mainly used for?

  1. Designing web pages
  2. Storing data permanently like a database
  3. Parallel processing of large datasets across many cores or machines
  4. Managing passwords
Show answer

Answer: Parallel processing of large datasets across many cores or machines

Spark is a compute engine that reads from and writes to separate storage.

Lesson: What Spark Is

2. What triggers a Spark job?

  1. An action such as count() or write
  2. Creating a DataFrame variable
  3. Importing pyspark
  4. Typing a comment
Show answer

Answer: An action such as count() or write

Transformations only build a plan; an action makes Spark actually execute it.

Lesson: Architecture: Driver, Executors, Jobs, Stages, Tasks

3. What is the entry point for DataFrames and SQL in PySpark?

  1. pandas.read()
  2. SparkPlug
  3. DataBase()
  4. SparkSession
Show answer

Answer: SparkSession

SparkSession is created first and used to read data and run SQL.

Lesson: Running Spark and Creating a SparkSession

4. Why prefer DataFrames over raw RDDs?

  1. RDDs do not exist anymore
  2. The Catalyst optimizer can plan DataFrame operations efficiently
  3. DataFrames never use memory
  4. RDDs cannot run in parallel
Show answer

Answer: The Catalyst optimizer can plan DataFrame operations efficiently

Named columns and declarative operations let Spark optimise the whole query.

Lesson: RDDs, DataFrames and Spark SQL

The DataFrame API

5. Why does `df.filter(...)` not modify `df` itself?

  1. Filters are ignored
  2. DataFrames are immutable; each operation returns a new DataFrame
  3. It deletes the data
  4. Spark forbids filters
Show answer

Answer: DataFrames are immutable; each operation returns a new DataFrame

Assign the result to a variable to keep the transformed DataFrame.

Lesson: Selecting, Filtering and Columns

6. What does `sum` do with null values in a group?

  1. Doubles the result
  2. Treats them as 1
  3. Fails always
  4. Ignores them (and returns NULL only if all values are null)
Show answer

Answer: Ignores them (and returns NULL only if all values are null)

Aggregate functions skip nulls, which can hide missing data, so check null counts.

Lesson: Aggregations and Null Handling

7. Which join keeps every row of the left DataFrame, with nulls where there is no match?

  1. left outer join
  2. inner join
  3. left anti join
  4. cross join without keys
Show answer

Answer: left outer join

A left outer join preserves unmatched left rows and fills right columns with null.

Lesson: Joins

8. What is the key difference between groupBy and a window function?

  1. There is no difference
  2. groupBy keeps every row, windows collapse them
  3. A window function keeps every input row
  4. Windows only work on strings
Show answer

Answer: A window function keeps every input row

groupBy collapses each group to one row; window functions add a computed column to all rows.

Lesson: Window Functions

Spark SQL and Files

9. What does `spark.sql("SELECT ...")` return?

  1. A pandas file
  2. A string
  3. A DataFrame
  4. Nothing
Show answer

Answer: A DataFrame

SQL results are DataFrames and can continue into further transformations.

Lesson: Spark SQL and Temporary Views

10. Why is Parquet usually better than CSV for analytics?

  1. It is columnar, compressed and stores the schema, so Spark reads less data
  2. It can be opened in a text editor
  3. It has no types
  4. It is slower to read
Show answer

Answer: It is columnar, compressed and stores the schema, so Spark reads less data

Columnar layout allows column pruning and predicate pushdown.

Lesson: File Formats: CSV, JSON, Parquet

11. Which is a good partition column for event data?

  1. the full timestamp to the millisecond
  2. user_id with millions of values
  3. a random UUID
  4. event_date
Show answer

Answer: event_date

A low-cardinality, frequently filtered column gives useful pruning without a flood of tiny files.

Lesson: Partitioned Output and Partition Pruning

How Spark Executes Your Code

13. Which operation normally causes a shuffle?

  1. select of two columns
  2. groupBy aggregation
  3. filter on a column
  4. withColumn with a constant
Show answer

Answer: groupBy aggregation

Grouping needs all rows of a key together, so data must move between partitions.

Lesson: Narrow and Wide Transformations: the Shuffle

15. Which call reduces partitions without a full shuffle?

  1. coalesce(n)
  2. repartition(n)
  3. groupBy()
  4. distinct()
Show answer

Answer: coalesce(n)

coalesce merges neighbouring partitions locally, whereas repartition reshuffles all data.

Lesson: Partitions: repartition, coalesce and Skew

Performance Tuning

16. When is caching worthwhile?

  1. When the DataFrame is used once
  2. For every DataFrame always
  3. When an expensive DataFrame is reused several times
  4. Never
Show answer

Answer: When an expensive DataFrame is reused several times

Caching avoids recomputation, which only helps when the data is reused.

Lesson: Caching and Persistence

17. What is the main benefit of a broadcast join?

  1. It works for any table size
  2. It sorts both tables
  3. The large table does not need to be shuffled
  4. It deletes duplicates
Show answer

Answer: The large table does not need to be shuffled

Shipping the small table everywhere replaces an expensive shuffle with a cheap local lookup.

Lesson: Join Strategies and Broadcast Joins

18. Why avoid thousands of tiny files?

  1. Each file adds open-and-task overhead, which slows reads
  2. Tiny files are illegal
  3. They use too little disk
  4. Spark cannot read them at all
Show answer

Answer: Each file adds open-and-task overhead, which slows reads

Per-file overhead dominates when files are small, so compact them into larger ones.

Lesson: Data Layout, Small Files and Configuration

Structured Streaming

19. What does a checkpoint location store?

  1. Only the final output files
  2. Your passwords
  3. The Spark binaries
  4. Query progress and state so a restart can resume
Show answer

Answer: Query progress and state so a restart can resume

Offsets and state in the checkpoint let a restarted query continue where it stopped.

Lesson: The Structured Streaming Model

20. What does a watermark let Spark do?

  1. Make events arrive faster
  2. Drop old window state and bound memory while tolerating limited lateness
  3. Encrypt the stream
  4. Disable windows
Show answer

Answer: Drop old window state and bound memory while tolerating limited lateness

It defines how late data may be, so old state can be cleaned up.

Lesson: Event-Time Windows and Watermarks

Production Spark

21. Why keep data in object storage and not on the cluster?

  1. Object storage is always faster than disks
  2. So clusters can be temporary and scaled independently of storage
  3. Spark cannot read local files
  4. It removes the need for backups
Show answer

Answer: So clusters can be temporary and scaled independently of storage

Separating compute from storage lets you start, resize and stop clusters without moving data.

Lesson: Cluster Managers and Platforms

22. Why write transformations as functions of DataFrames?

  1. It removes the need for tests
  2. Spark forbids anything else
  3. Functions run faster by definition
  4. They are easy to unit-test with small local data
Show answer

Answer: They are easy to unit-test with small local data

Pure functions separate logic from I/O, so tests need no cluster.

Lesson: Testing Spark Jobs

23. In the Spark UI a stage has one task running far longer than the rest. What does it suggest?

  1. Data skew
  2. A fast network
  3. The job is finished
  4. Too few tasks exist
Show answer

Answer: Data skew

A straggler task usually holds a hot key with far more data than the others.

Lesson: Monitoring with the Spark UI and Lakehouse Tables

Putting It Together

24. What makes re-running the same date safe in this design?

  1. Using random file names
  2. Appending to the table each time
  3. Overwriting only that date's partition produces the same result each time
  4. Skipping data-quality checks
Show answer

Answer: Overwriting only that date's partition produces the same result each time

Idempotent, partition-scoped writes make retries and backfills harmless.

Lesson: Case Study: A Daily Sales Pipeline

25. A job calls `df.count()` twice on an expensive DataFrame with no caching. What happens?

  1. Spark errors out
  2. The second call is free
  3. The whole lineage is computed twice
  4. The result is cached automatically
Show answer

Answer: The whole lineage is computed twice

Each action recomputes from the source unless the DataFrame is cached or persisted.

Lesson: Revision: Cheat Sheet and Self-Check

26. A join between a 2 TB table and a 5 MB lookup table is slow. What is the best first idea?

  1. Broadcast the 5 MB table
  2. Collect the 2 TB table to the driver
  3. Add more shuffle partitions only
  4. Convert both to CSV
Show answer

Answer: Broadcast the 5 MB table

Broadcasting the small side avoids shuffling the huge table.

Lesson: Revision: Cheat Sheet and Self-Check

27. Which statement about coalesce and repartition is correct?

  1. repartition never moves data
  2. They are identical
  3. coalesce can increase partitions with a full shuffle
  4. coalesce only reduces partitions without a full shuffle; repartition shuffles to an exact count
Show answer

Answer: coalesce only reduces partitions without a full shuffle; repartition shuffles to an exact count

coalesce is a cheap merge for shrinking; repartition redistributes all data evenly.

Lesson: Revision: Cheat Sheet and Self-Check

← Back to the course