Streaming Egress (Buhler, Erl, Khattak)
How can processed data be exported in realtime from a Big Data platform to other systems?
A queue-based, publish-subscribe message-passing system is developed. The system is configured to copy data from a storage device and to forward the data to the downstream sinks in the form of events. The copying of the data is either based on a set interval, or it may get triggered as soon as data appears in the configured source location. Using a queue provides high availability, fault-tolerance, scalability and delivery assurance features and further enables exporting results to multiple downstream systems at a time.
The Streaming Egress pattern is generally applied together with the Streaming Source, Realtime Access Storage and High Velocity Realtime Processing patterns.
Functionality is added to the Big Data platform to enable exporting analysis results as soon as they get computed. A system is developed that continuously copies the computed result to the interested clients in realtime.
- Readings from a pressure sensor that arrive every five seconds are analyzed in realtime by a realtime processing engine.
- The analysis results are stored in a storage device.
- A publish-subscribe system copies the analysis results to a pressure monitoring application every ten seconds.
- The introduced latency with regards to the availability of the analysis results is very limited.
- Due to this, the engineer is able to make timely decisions.