WP4 - Distributed analytics over extremely large numbers of high-volume streams

Another technological package offered by the I-BiDaaS solution deals with the production of analytics over almost real-time streaming and heterogeneous data. WP4 is realized by three tasks.

 

  • Develop the I-BiDaaS Distributed Big Data analytics algorithms over extremely large numbers of high-volume streams;
  • Define, customize and integrate its complex event processing engine (Apama by SAG), and to customize the needed visualization tools (MashZone by SAG) in collaboration with advanced visualizations of WP2;
  • Develop machine-learning techniques deployed on respective edge nodes for handling incomplete data;
  • Develop both static and dynamic techniques to partition the incoming queries and execute to the GPU the analytics that will yield higher performance.

In the context of this WP three reports and three prototypes will be produced (more than one version per type). Most of the material will be publicly available (excluding reports that refer to commercial products).

This WP has produced 1 public and 1 confidential report according to the plan. This work carried out so far led to the design of the streaming component of the I-BiDaaS platform which allows to analyze a large number of input streams in a distributed fashion using a complex event processing engine. In addition, the offloading of compute-intense tasks to GPUs was designed and tested.