# Spark Structured Streaming: A Principal Engineer's Guide

## 1\. First, what exactly is Spark Streaming?

Let's start with a simple question.

If a business generates 10 million events every day, do we really need to wait until midnight to process them?

Not necessarily.

In traditional batch processing, we collect data for a particular period and process it together.

For example:

```text
12:00 AM  -> Data collection starts
01:00 AM  -> More records arrive
02:00 AM  -> More records arrive
03:00 AM  -> Batch processing starts
03:30 AM  -> Data is available
```

This works perfectly well for daily reporting, financial reconciliation, and historical analysis.

But what if the business wants to identify suspicious transactions within seconds?

Waiting for the next scheduled batch isn't going to help.

This is where streaming comes in.

Instead of waiting for a large dataset, we continuously consume newly available events and process them incrementally.

For example:

```text
Order A arrives -> Process
Order B arrives -> Process
Order C arrives -> Process
Order D arrives -> Process
```

One important clarification: streaming does not always mean processing one record at a time.

Spark Structured Streaming uses micro-batch processing by default. It groups incoming records into small execution batches and processes them incrementally.

Before going further, let's distinguish the older Spark Streaming API from Structured Streaming.

The original Spark Streaming API used DStreams, where a continuous stream was represented as a sequence of RDDs.

Structured Streaming introduced a higher-level programming model built on Spark SQL and DataFrames.

Instead of managing individual RDD batches, we work with a streaming DataFrame that represents continuously arriving data.

For modern Spark development, Structured Streaming is generally the preferred API.

---

## 2\. How does Spark Structured Streaming actually work?

Let's take our financial transaction example.

Kafka is receiving thousands of transaction events. Spark needs to consume them, validate them, transform them, and write the results into Delta Lake.

What happens internally?

At a high level, the flow looks like this:

```text
          APACHE KAFKA
                |
                v
       Read Source Offsets
                |
                v
      SPARK STRUCTURED STREAMING
                |
                v
        Create Micro-batch
                |
                v
          Transform Data
                |
                v
       Stateful Operations
                |
                v
           Write Output
                |
                v
           DELTA LAKE
                |
                v
       Commit Progress
                |
                v
          Next Trigger
```

Let's understand the execution.

First, Spark connects to the source and identifies the available records.

If the source is Kafka, Spark tracks offsets for each topic partition.

For example:

```text
Topic: transactions

Partition 0: Offset 1000 -> 1100
Partition 1: Offset 2000 -> 2150
Partition 2: Offset 3000 -> 3050
```

These offsets tell Spark which records are available for the current execution.

Spark then creates an incremental execution plan, processes the available records, and writes the output to the configured sink.

After processing, it records progress in its checkpoint location.

The next execution picks up from the appropriate source position.

This is important because Spark isn't repeatedly loading the entire dataset. It processes new data incrementally.

You can start a streaming query with a familiar DataFrame API:

```python
orders = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", kafka_servers) \
    .option("subscribe", "orders") \
    .load()
```

Notice that we're using `readStream` instead of `read`.

That tells Spark we're working with a streaming source.

**Think of Structured Streaming as a continuously running query over an ever-growing table.**

That mental model makes the rest of the API much easier to understand.

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/b4411139-127d-4db9-954d-8a73384b5689.png align="center")

---

## 3\. Micro-batches: Why doesn't Spark process every record individually?

Let's say Kafka receives 100,000 events every minute.

Spark could theoretically process records individually, but distributed execution has overhead.

It needs to schedule tasks, distribute work across executors, coordinate processing, and manage progress.

Instead, Structured Streaming groups records into micro-batches.

For example:

```text
 Incoming Events
      |
      v
  Micro-batch 1
  10,000 records
      |
      v
  Micro-batch 2
  12,000 records
      |
      v
  Micro-batch 3
   8,000 records
      |
      v
  Micro-batch 4
  15,000 records
```

Each micro-batch is processed incrementally.

But how frequently should Spark execute these batches?

That's where triggers come in.

A processing-time trigger allows us to configure an execution interval:

```python
query = orders.writeStream \
    .format("delta") \
    .trigger(processingTime="10 seconds") \
    .option("checkpointLocation", checkpoint_path) \
    .start(target_path)
```

Here, Spark is configured to trigger processing every 10 seconds.

Another useful option is `AvailableNow`.

Imagine a file ingestion pipeline where data arrives throughout the day, but you don't need a continuously running cluster.

You can process all currently available data and stop:

```python
query = orders.writeStream \
    .format("delta") \
    .trigger(availableNow=True) \
    .option("checkpointLocation", checkpoint_path) \
    .start(target_path)

query.awaitTermination()
```

Spark processes the available backlog, potentially using multiple micro-batches, and terminates when the available data has been processed.

This is particularly useful for cost-conscious incremental ingestion.

One important point: a 10-second trigger does not guarantee that your data will be available within 10 seconds. If a micro-batch takes 30 seconds to execute, Spark cannot magically finish it in 10.

The actual latency depends on source delay, processing capacity, scheduling, and sink performance.

---

## 4\. Event Time vs. Processing Time: The problem with late events

Now comes an interesting problem.

A customer makes a payment at 10:00 AM.

But due to a network issue, the event reaches Kafka at 10:15 AM.

Should Spark count this transaction in the 10:00 window or the 10:15 window?

It depends on which timestamp you're using.

**Event time** represents when the event actually happened.

**Ingestion time** represents when the event entered the ingestion system.

**Processing time** represents when Spark processed the event.

Consider this:

```text
10:00 AM -> Transaction actually happened
10:11 AM -> Kafka received the event
10:12 AM -> Spark processed the event
```

These are three different timestamps.

For financial reporting, event time is often more meaningful because the business cares about when the transaction occurred, not when Spark happened to process it.

Suppose we calculate transaction totals every five minutes.

If we use processing time, a delayed 10:00 transaction might appear in the 10:10 window.

That's not necessarily what the business expects.

This is why event-time processing is important.

But now we have another problem.

How long should Spark wait for late-arriving events?

Should it wait forever?

That would be a terrible streaming system. We might as well ask the business team to wait until the next financial year.

This brings us to watermarks.

---

## 5\. Watermarks: How does Spark decide when to stop waiting?

A watermark defines how much event-time lateness a streaming query is designed to tolerate before old state can be cleaned up.

Let's say we're calculating five-minute transaction totals.

We configure a watermark of ten minutes:

```python
from pyspark.sql.functions import col, window, sum

result = orders \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        window(col("event_time"), "5 minutes")
    ) \
    .agg(
        sum("amount").alias("total_amount")
    )
```

Conceptually, if Spark has observed an event with an event time of 10:30 AM, the watermark would be approximately 10:20 AM.

That means Spark can start moving beyond older event-time windows and cleaning up their retained state.

The exact behavior depends on the query, available input data, and watermark coordination.

Now think about the trade-off.

If the business regularly receives events 20 minutes late but our watermark is only five minutes, valid late events may no longer be included in the relevant aggregation.

On the other hand, if we configure a watermark of several hours, Spark may need to retain much more state.

More state means more memory or disk usage, more checkpointing work, and potentially longer recovery times.

**A watermark is not just a Spark configuration. It's a business decision about how much lateness the system can tolerate.**

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/9882b823-219b-4919-aceb-15e2c13ac558.png align="center")

---

## 6\. Windows and State: Where streaming becomes interesting

Let's say the business wants transaction totals for every five minutes.

This is called a tumbling window.

```python
result = orders \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        window(col("event_time"), "5 minutes")
    ) \
    .agg(
        sum("amount").alias("total_amount")
    )
```

The windows look like this:

```text
10:00 - 10:05
10:05 - 10:10
10:10 - 10:15
```

Each event belongs to one window for this window definition.

But what if the business wants a rolling ten-minute total, updated every five minutes?

That's a sliding window.

```python
result = orders \
    .withWatermark("event_time", "10 minutes") \
    .groupBy(
        window(
            col("event_time"),
            "10 minutes",
            "5 minutes"
        )
    ) \
    .agg(
        sum("amount").alias("total_amount")
    )
```

Here, the windows overlap.

An event can contribute to more than one window.

There is also a session window, which groups events based on activity separated by an inactivity gap.

Imagine a customer browsing a website.

They view a product, add it to their cart, and proceed to checkout. Then they disappear for 30 minutes.

A session window can help group those related activities into one session.

```python
from pyspark.sql.functions import session_window

sessions = events \
    .withWatermark("event_time", "30 minutes") \
    .groupBy(
        "user_id",
        session_window("event_time", "10 minutes")
    ) \
    .count()
```

Now, here's something important.

Windowed aggregations and other stateful operations require Spark to remember information from previous events.

For example, if we calculate the number of orders per customer, Spark must maintain the running count for each customer.

This is stateful processing.

A simple filter such as `amount > 1000` is stateless. Spark can evaluate each event independently.

But aggregations, deduplication, and stream-stream joins require state.

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/69e60141-6b14-4333-b805-c2a1367d395c.png align="center")

Spark uses a state store to maintain this information. For large stateful workloads, RocksDB can be used as a disk-backed state-store implementation to reduce JVM heap pressure.

However, RocksDB does not make state management free.

If you're retaining millions of keys, you still need to understand state size, retention, checkpoint overhead, and recovery time.

This is where a seemingly small design decision can become a major infrastructure cost.

---

## 7\. Duplicates, Joins, and the Exactly-Once Question

Let's consider a payment event.

Spark processes transaction `TXN1001`.

The source system retries the same event, and it arrives again.

What happens?

Without appropriate deduplication, the transaction may be counted twice.

We can use an event identifier:

```python
deduplicated = orders \
    .withWatermark("event_time", "30 minutes") \
    .dropDuplicatesWithinWatermark(["event_id"])
```

The watermark bounds the deduplication horizon. It does not guarantee duplicate removal forever.

The event ID must also be a reliable business identifier. Combining customer ID, amount, and timestamp is not automatically a safe deduplication strategy.

Now imagine we want to enrich orders with customer information.

This is a stream-static join:

```python
enriched = orders_stream.join(
    customers,
    "customer_id",
    "left"
)
```

Orders are continuously arriving, while customers are read as a static DataFrame for that query.

Remember that changes to the underlying customer table are not automatically reflected in an already-created static DataFrame.

If both orders and payments are streaming, we have a stream-stream join.

That requires Spark to retain unmatched events while waiting for their corresponding records. Watermarks and time constraints become especially important.

And now we reach a question that every streaming engineer eventually encounters:

**Does Spark guarantee exactly-once processing?**

The answer needs some context.

Spark can provide exactly-once processing guarantees under its supported fault-tolerance model, using replayable sources, checkpointed progress, and compatible sinks.

But exactly-once processing and exactly-once delivery to an external system are not the same thing.

For example, the built-in Kafka sink provides at-least-once delivery. A failed query can cause records to be written again.

For Delta Lake, supported transactional streaming writes provide stronger exactly-once guarantees.

For custom external writes, you need to design idempotency or transactional handling yourself.

A simple Delta Lake merge illustrates the idea:

```sql
MERGE INTO target_payments AS target
USING incoming_payments AS source
ON target.event_id = source.event_id

WHEN NOT MATCHED THEN
  INSERT *
```

The objective is to ensure that replaying the same business event doesn't create another payment record.

But the complete implementation must also define how corrections and updates to existing events are handled.

**Exactly-once is not a magic switch. It is a property of the complete processing and delivery design.**

---

## 8\. Checkpointing: What happens when the cluster crashes?

Let's say your streaming query has processed 50 million records.

Suddenly, the cluster terminates.

Do we start again from the beginning?

Hopefully not.

Structured Streaming uses checkpointing to record the progress and state required for recovery.

```python
query = orders.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option(
        "checkpointLocation",
        "/mnt/checkpoints/orders"
    ) \
    .start("/mnt/delta/orders")
```

The checkpoint contains information such as source offsets, batch progress, commit information, and state-store data where applicable.

When the query restarts with a compatible checkpoint, Spark can recover its progress and continue processing.

There are three things I would insist on during a production design review.

First, every independent streaming query should have its own checkpoint location.

Second, never delete a production checkpoint simply because a query failed to restart. Understand the failure and recovery requirements first.

Third, changes to stateful operators, source configuration, and query structure must be evaluated for checkpoint compatibility.

A checkpoint is not just a temporary folder.

It is part of the streaming application's recovery contract.

---

## 9\. Kafka + Spark + Delta Lake: Putting everything together

Let's bring all these concepts into one architecture.

Imagine an e-commerce platform generating orders, payments, refunds, and customer activity events.

Kafka receives those events. Spark consumes them, applies transformations, and writes the data into Delta Lake.

I'd structure the Lakehouse around the familiar Bronze, Silver, and Gold layers.

**Bronze** preserves incoming records, including source metadata and ingestion timestamps.

**Silver** handles schema validation, deduplication, standardization, and data quality rules.

**Gold** contains business-ready datasets such as transaction summaries, customer metrics, and payment success rates.

```text
           APPLICATION EVENTS
                   |
                   v
              APACHE KAFKA
                   |
                   v
        SPARK STRUCTURED STREAMING
                   |
           Validation / Parsing
                   |
          Deduplication / Enrichment
                   |
                   v
              BRONZE DELTA
                   |
                   v
              SILVER DELTA
                   |
          Business Transformations
                   |
                   v
               GOLD DELTA
                   |
             +-----+-----+
             |           |
             v           v
           BI Tools   Data Products
```

Checkpoint storage supports recovery, while monitoring tracks source lag, processing duration, state-store growth, and data quality.

Unity Catalog can provide governance and access management across the platform.

One thing I'd avoid is putting every transformation into one enormous streaming query.

Separate responsibilities where it improves ownership, reliability, and maintainability. But don't create unnecessary streaming jobs and storage hops either.

The architecture should reflect actual business and operational requirements.

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/72609598-f723-465c-b683-cc3f6dbce3f7.png align="center")

---

## 10\. Real-time Mode: Is Spark moving beyond micro-batches?

Now let's look at an important newer development.

Spark introduced Real-time Mode in Spark 4.1.

The default micro-batch engine repeatedly schedules tasks to process bounded sets of records. This introduces scheduling overhead.

Real-time Mode takes a different approach.

It launches long-running tasks that process incoming records continuously rather than waiting for each micro-batch boundary.

Conceptually:

```text
MICRO-BATCH

Records Arrive
      |
Wait for Batch
      |
Schedule Tasks
      |
Process Records
      |
Commit Progress


REAL-TIME MODE

Records Arrive
      |
Long-running Tasks
      |
Process Records Continuously
      |
Commit Progress Periodically
```

The interesting part is that the Real-time trigger duration is primarily a checkpoint interval, not the time records must wait before processing.

For example, a five-minute trigger does not mean Spark waits five minutes before processing events.

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/3f2b30d5-0ca7-4a1e-a506-4524e5082ddb.png align="center")

In Spark 4.3, Real-time Mode supports stateless operations and a first set of stateful operations, including streaming aggregations and deduplication. Support is still more limited than the regular micro-batch engine, and stream-stream joins are not supported in the documented 4.3 implementation.

Here's the Python API shape:

```python
query = orders.writeStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", kafka_servers) \
    .option("topic", "processed_orders") \
    .option("checkpointLocation", checkpoint_path) \
    .outputMode("update") \
    .trigger(realTime="5 minutes") \
    .start()
```

This is an execution-mode example, not a drop-in replacement for every existing streaming query. Source, sink, output mode, and operator support must all be checked against the deployed Spark version.

For low-latency workloads, Real-time Mode is worth evaluating.

For complex stateful workloads that need broader operator support, micro-batch remains the more mature choice.

---

## 11\. When should you use Spark Streaming, and when should you avoid it?

This is probably the most important architectural question.

Just because a technology supports a use case doesn't mean it is the right choice.

I'd consider Structured Streaming for continuous Kafka ingestion, event processing, clickstream analytics, transaction monitoring, and incremental Lakehouse ingestion.

But if the business only needs a report every 30 minutes, a scheduled incremental job may be simpler and cheaper.

If the requirement is a few milliseconds of request-response latency, a dedicated online serving system may be more appropriate.

If the transformation is simple and data arrives only a few times a day, maintaining a continuously running Spark cluster may be unnecessary.

I would ask three questions before choosing the architecture:

**1\. What is the actual freshness requirement?**

Seconds, minutes, or milliseconds? Don't build a low-latency platform when the business only needs hourly updates.

**2\. How will the system recover?**

Can the source replay data? Are the sinks idempotent? How much state must be restored?

**3\. Who will operate it?**

Streaming needs monitoring, capacity planning, checkpoint management, incident response, and deployment discipline.

The cost of operating the platform matters just as much as the cost of building it.

---

## 12\. Final Thoughts: Streaming is easy to start. Hard to operate well.

There is a reason streaming tutorials often look deceptively simple.

Read from Kafka.

Apply a few transformations.

Write to Delta.

Done.

But production streaming is about understanding what happens when events arrive late, duplicates appear, the source changes its schema, state grows unexpectedly, or the cluster crashes.

These are not exceptional situations. They are normal conditions in distributed systems.

And this is where the Staff/Principal Engineer perspective becomes important.

The responsibility is not just to make the streaming query work.

It is to design a platform that can recover, scale, remain understandable, and produce trustworthy data even when the surrounding systems are not behaving perfectly.

![](https://cdn.hashnode.com/uploads/covers/69ef78e1330a1ad7f7f0ccf8/4267ef64-6823-4883-ab5b-95b4a429d1e9.png align="center")

My final thought:

> ***Don't introduce streaming because the business says it wants real-time data. Introduce it because you understand what freshness is worth, what it will cost, and how you will keep the system correct.***
>
> *Sometimes the right answer is a continuously running Spark query.*
>
> *Sometimes it is an AvailableNow job running every 15 minutes.*
>
> *And sometimes, it is simply a well-designed batch pipeline.*
>
> *The technology is important.*
>
> *But the engineering decision matters more.*
>
> ***Happy Engineering!***

#ApacheSpark #StructuredStreaming #PySpark #Databricks #DeltaLake #DataEngineering #SystemDesign