The Dataflow Model Revisited
summary
The original Dataflow Model paper correctly identified the need to handle unbounded, out-of-order data without waiting for completeness. However, it over-emphasized operational mechanics like windowing and triggers instead of leveraging database fundamentals like SQL and materialized views. The authors argue that the goal of streaming analytics should be the total removal of complexity for the end-user. True success in this analytics space comes from treating streams and tables as two perspectives of the same object with different access semantics.
key points
- The focus on triggers and windowing was an over-engineered solution to problems that users should never have encountered. These operational concerns should have been hidden from the user interface to simplify the system's behavior.
- Streaming and batch processing are essentially the same operation viewed through different perspectives. The most successful implementations have used incremental view maintenance and materialized views to hide the complexity of streaming from the user.
- The completeness principle has evolved into two distinct paths: watermarks for visible streams and snapshot-consistent refresh for invisible streams. Snapshot-consistent refresh is more widely adopted because it requires significantly less effort from the user.
who should read it
Data engineers and database researchers who want to understand the evolution of streaming analytics and the trade-offs between different processing models.
discussion positive
consensus
Most commenters agree that dedicated stream processing programming models have failed to gain mainstream adoption because few companies actually require ultra-low latency. The prevailing view is that SQL and relational database abstractions are more practical for the majority of analytics use cases, as they are easier to operate and evolve. Stateful streaming systems are seen as inherently more complex and expensive compared to batch systems or modern low-latency data warehouses.
top insight
The failure of stream processing as a standalone paradigm is attributed to a demand problem rather than a technical one, as most businesses do not need the extreme performance it provides. However, the dataflow model remains valuable for solving specific hard problems, such as determining time-period completeness and avoiding expensive recomputations. These strengths may eventually integrate back into batch systems as object storage reduces operational overhead.
Generated by an LLM from the article and thread. Check the sources before relying on it.