Real-Time Data Pipelines: When Streaming Is Worth the Complexity
Ask what decision changes with fresher data
The useful question is not 'do we want real-time data' — everyone says yes — but 'what would we do differently if this data were seconds old instead of an hour old, and what is that worth'. Fraud blocking, inventory availability during checkout, dynamic pricing, operational alerting and personalisation in-session all have clear answers. Executive dashboards, monthly reporting and most analytics do not. Half the streaming projects we review would have been better served by an incremental batch job running every few minutes.
The hard parts are ordering, duplicates and late data
Distributed streams deliver events out of order, occasionally more than once, and sometimes hours late after a mobile client reconnects. Every aggregation you build has to answer what happens when a transaction from Tuesday arrives on Thursday. That means event-time processing with watermarks, idempotent writes keyed on event ID, and a documented policy for how late is too late. These are solvable problems with well-understood patterns, but they are why a streaming pipeline takes three times as long to build correctly as its batch equivalent.
Change data capture beats polling your production database
For getting operational data into analytics quickly, reading the database transaction log is the pattern that scales — it captures every change including deletes, imposes minimal load on the source, and preserves ordering per table. Polling for updated rows misses deletes, hammers the primary, and quietly drifts out of sync. The setup cost is higher but it is the difference between a pipeline that stays correct and one that needs periodic full reloads to fix.
Schema changes will break you eventually
Someone renames a column in the source system on a Friday. In batch, you notice on Monday and backfill. In streaming, downstream consumers start failing immediately. A schema registry with compatibility rules, plus consumers written to tolerate unknown fields, turns that incident into a non-event. This is unglamorous infrastructure that nobody budgets for and everybody needs by month six.
A pragmatic middle path
Stream the few things that genuinely need it — the events driving real-time decisions — and batch everything else on a short schedule. Modern warehouses handle frequent incremental loads well enough that many 'real-time' requirements are satisfied by a five-minute cadence at a fraction of the operational burden. Keep both paths writing into the same tables so analysts do not need to know which is which.
Budget for operations, not just build
Streaming systems need on-call coverage, lag monitoring, replay procedures and someone who understands them at 3am. If your team is four people and this is their fifth priority, that reality should influence the architecture decision more than any technical merit. Choosing the boring option you can actually operate is a legitimate engineering decision.
Have a project that needs this kind of thinking?
Let's talk through what you're building — free consultation, no commitment.
Get in Touch