PySpark Important Last-Minute Notes

If you are preparing for Data Engineering interviews, Spark projects, or need a quick PySpark revision, this post consolidates the most important PySpark concepts in one place.

Best for: Data Engineers, Big Data Developers, Azure/Databricks/Microsoft Fabric users, and anyone doing last-minute interview preparation.

---

What is PySpark?

PySpark is the Python API for Apache Spark, an open-source distributed computing framework used for large-scale data processing.

Why PySpark?

• Distributed, in-memory processing
• Faster than traditional batch systems
• Scales across clusters
• Supports SQL, Streaming, and Machine Learning

Common use cases

• ETL / ELT pipelines
• Big data analytics
• Machine learning at scale
• Real-time and batch processing

---

Spark Cluster Architecture

A Spark cluster typically consists of:

• Master Node: Manages resources and schedules tasks
• Worker Nodes: Execute tasks in parallel

This architecture enables efficient distributed processing on large datasets.

---

SparkSession – Entry Point to PySpark

SparkSession is the unified entry point to Spark features (SQL, streaming, ML, and DataFrame operations).

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("MySparkApp") \
    .getOrCreate()

Key points

• .appName() helps track multiple applications
• .getOrCreate() prevents duplicate sessions that waste memory

---

PySpark DataFrames

PySpark DataFrames are similar to Pandas, but distributed and optimized for big data processing.

Create a DataFrame

df = spark.read.csv("file.csv", header=True, inferSchema=True)
df.show()

Inspect schema

df.printSchema()

Count rows

df.count()

---

Basic Analytics Operations

Group and aggregate

df.groupBy("gender").agg({"salary_usd": "avg"})

Common DataFrame methods (PySpark vs SQL)

PySpark	SQL Equivalent
select()	SELECT
filter()	WHERE
groupBy()	GROUP BY
agg()	AGGREGATE

---

Filtering and Selecting Data

df.filter(df.age > 50).select("name", "age").show()

---

Handling Missing (Null) Values

Drop nulls

df.na.drop()
df.na.drop(subset=["column_name"])

Fill nulls

df.na.fill(value=0)
df.na.fill({"column1": 0, "column2": "Unknown"})

Filter out nulls using isNotNull()

df.filter(df["column_name"].isNotNull())

---

Column Operations

Create a new column

df = df.withColumn("new_column", df["existing_column"] * 2)

Rename a column

df = df.withColumnRenamed("old_column_name", "new_column_name")

Drop a column

df = df.drop("column_name")

---

Row Operations

Filter rows

df_filtered = df.filter(df["age"] > 30)

Group and aggregate

df_grouped = df.groupBy("category").agg({"value_column": "sum"})

---

Defining Schema Manually (Recommended)

Schema inference is convenient, but it can misinterpret column data types. Manual schema improves data quality and stability.

from pyspark.sql.types import StructType, StructField, IntegerType, StringType

schema = StructType([
    StructField("age", IntegerType()),
    StructField("education_num", IntegerType()),
    StructField("marital_status", StringType()),
    StructField("occupation", StringType()),
    StructField("income", StringType())
])

df = spark.read.csv("data.csv", schema=schema)
df.printSchema()

---

Joins in PySpark

Joins combine two DataFrames based on matching keys, similar to SQL joins.

Supported join types

• Inner Join
• Left (Outer) Join
• Right (Outer) Join
• Full Outer Join

df1.join(df2, df1["id"] == df2["id"], "inner")

---

Union Operation

Union stacks two DataFrames (they must have the same schema).

df_union = df1.union(df2)

---

Working with Complex Data Types

Arrays

from pyspark.sql.functions import lit
df = df.withColumn("array_col", lit([1, 2, 3]))

Maps

from pyspark.sql.functions import map, lit
df = df.withColumn("map_col", map(lit("k1"), lit("v1")))

Structs

Structs group related fields into one column (useful for nested data).

---

User Defined Functions (UDFs)

PySpark UDF

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def to_upper(s):
    return s.upper()

upper_udf = udf(to_upper, StringType())

df = df.withColumn("upper_name", upper_udf(df["name"]))

pandas UDF (Preferred for large datasets)

from pyspark.sql.functions import pandas_udf
import pandas as pd

@pandas_udf("float")
def square(s: pd.Series) -> pd.Series:
    return s * s

df = df.withColumn("squared", square(df["value"]))

---

PySpark UDF vs pandas UDF

Feature	PySpark UDF	pandas UDF
Performance	Slower	Faster
Best for data size	Small	Large
Execution	Row-wise	Vectorized

---

Final Takeaways

• PySpark is designed for distributed big data processing.
• DataFrames are optimized and support SQL-like operations.
• Manual schemas improve data quality and prevent type issues.
• pandas UDFs are typically faster for large datasets.
• PySpark is essential for Azure Databricks, Microsoft Fabric, and Spark-based interviews.

---

If this helped you: Bookmark this post for quick revision before interviews.