Data stream oriented applications are typically dealing with huge volumes of data. Storing data and performing off-line processing on this huge data set can be costly, time consuming and impractical. Low Latency, High Performance Data Stream Processing: Systems Architecture, Algorithms And Implementation describes the author’s research results while designing and implementing an efficient data management system for online and offline processing of streaming data.
Major existing data stream processing engines, their internal architecture, and how they compare to Global Sensor Network (GSN) middleware are presented. In order to achieve high efficiency while processing large volumes of streaming data using window-based continuous queries, the author presents a set of optimization algorithms and techniques to intelligently group and process different types of continuous queries. An efficient query scheduling component that not only increases the performance at least by an order of magnitude, but also decreases the response time and memory requirements, is detailed here.
Techniques and algorithms to enable scalable delivery of streaming data for high data rate streams (e.g., Financial Ticks) are presented, as well.