Apache Spark for Data Scientists – PySpark Big Data Guide 2026

When your data outgrows a single machine, Apache Spark is the answer. PySpark — Spark’s Python API — lets data scientists process terabytes across hundreds of machines using familiar DataFrame syntax. This guide covers everything from your first Spark job to MLlib machine learning and structured streaming.

Why Spark Over Pandas?

Pandas loads everything into RAM on one machine. Spark distributes data across a cluster and processes it in parallel. Spark is 10-100x faster than Hadoop MapReduce for iterative algorithms (like ML training) because it keeps data in memory across iterations. Use Pandas for data under ~10GB on a single machine. Switch to Spark when your data is larger, your pipeline is too slow, or you need real-time streaming processing.

Setting Up PySpark

pip install pyspark findspark

import findspark
findspark.init()

from pyspark.sql import SparkSession

spark = SparkSession.builder     .appName('DataExpertise-Tutorial')     .config('spark.sql.shuffle.partitions', '8')     .config('spark.driver.memory', '4g')     .getOrCreate()

print(spark.version)

DataFrames – The Core API

from pyspark.sql import functions as F
from pyspark.sql.types import *

# Load CSV — Spark infers schema automatically
df = spark.read.csv('sales_data.csv', header=True, inferSchema=True)
df.printSchema()
df.show(5)

# Transformations (lazy — nothing runs until an action is called)
result = (df
    .filter(F.col('revenue') > 1000)
    .withColumn('revenue_usd', F.col('revenue') * 0.012)
    .withColumn('month', F.month(F.col('sale_date')))
    .groupBy('region', 'month')
    .agg(
        F.sum('revenue_usd').alias('total_revenue'),
        F.count('*').alias('num_sales'),
        F.avg('revenue_usd').alias('avg_revenue')
    )
    .orderBy('total_revenue', ascending=False)
)

# Actions trigger execution
result.show(20)
result.write.parquet('output/revenue_by_region', mode='overwrite')

Spark SQL

# Register DataFrame as a temp view
df.createOrReplaceTempView('sales')

# Run SQL queries directly
top_regions = spark.sql('''
    SELECT region,
           SUM(revenue) AS total_revenue,
           COUNT(*) AS num_transactions
    FROM sales
    WHERE sale_date >= '2026-01-01'
    GROUP BY region
    HAVING SUM(revenue) > 100000
    ORDER BY total_revenue DESC
    LIMIT 10
''')
top_regions.show()

Machine Learning with MLlib

from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler, StandardScaler, StringIndexer
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Load and prepare data
df = spark.read.csv('churn_data.csv', header=True, inferSchema=True)

# Index string labels
label_indexer = StringIndexer(inputCol='churn', outputCol='label')

# Assemble features into a single vector column
feature_cols  = ['age', 'tenure', 'monthly_charges', 'num_products']
assembler     = VectorAssembler(inputCols=feature_cols,
                                outputCol='features_raw')
scaler        = StandardScaler(inputCol='features_raw',
                               outputCol='features')

# Random Forest
rf = RandomForestClassifier(numTrees=100, maxDepth=5,
                             featuresCol='features',
                             labelCol='label', seed=42)

# Pipeline
pipeline  = Pipeline(stages=[label_indexer, assembler, scaler, rf])
train, test = df.randomSplit([0.8, 0.2], seed=42)
pipeline_model = pipeline.fit(train)

# Evaluate
predictions = pipeline_model.transform(test)
evaluator   = MulticlassClassificationEvaluator(
    labelCol='label', predictionCol='prediction', metricName='accuracy')
print(f'Test Accuracy: {evaluator.evaluate(predictions):.4f}')

# Feature importance
rf_model = pipeline_model.stages[-1]
for feat, imp in sorted(zip(feature_cols, rf_model.featureImportances),
                         key=lambda x: -x[1]):
    print(f'{feat}: {imp:.4f}')

Structured Streaming

from pyspark.sql.functions import window, col

# Read from Kafka stream
stream_df = (spark.readStream
    .format('kafka')
    .option('kafka.bootstrap.servers', 'localhost:9092')
    .option('subscribe', 'transactions')
    .load())

# Parse JSON messages
from pyspark.sql.functions import from_json
schema = StructType([
    StructField('user_id', IntegerType()),
    StructField('amount', DoubleType()),
    StructField('timestamp', TimestampType())
])

parsed = stream_df.select(
    from_json(col('value').cast('string'), schema).alias('data')
).select('data.*')

# Aggregate in 5-minute windows
windowed = (parsed
    .withWatermark('timestamp', '10 minutes')
    .groupBy(window('timestamp', '5 minutes'), 'user_id')
    .agg(F.sum('amount').alias('total_spent')))

# Write to console (use parquet/kafka for production)
query = (windowed.writeStream
    .outputMode('update')
    .format('console')
    .trigger(processingTime='30 seconds')
    .start())
query.awaitTermination()

Performance Tips

Partition your data correctly — too few partitions underuses the cluster, too many creates overhead. A good rule is 2-4 partitions per CPU core. Cache DataFrames that are used multiple times with `df.cache()`. Use Parquet format over CSV — it is columnar, compressed, and 10x faster to read. Broadcast small lookup tables to avoid expensive shuffles: `spark.conf.set(“spark.sql.autoBroadcastJoinThreshold”, “10mb”)`.

Conclusion

PySpark is the essential tool for data scientists working with data that exceeds single-machine limits. The DataFrame API feels familiar to Pandas users while providing distributed processing power. MLlib brings the full ML pipeline to big data scale. And with managed services like Databricks and AWS EMR, you can spin up a 100-node cluster in minutes without managing infrastructure yourself.