Elastic high-performance computing platform for real-time data analysis

Using the power of high-performance computing together with the flexibility of loosely coupled event-driven software architectures provides alot of benefits, especially when it comes to processing real-time data. This paper outlines the architecture of a general-purpose platform leveraging event-driven microservices architecture in combination with Event Sourcing and powerful High-Performance Computing core. The platform is aimed to software applications that process and analyze huge amount of data in a real-time or near-real-time fashion from a variety of sources, having as requirement downtime-less upgrade and scaling capabilities. The first-class citizens of this platform are applications in the domains of IoT, trading, meteorology and traffic control. The reference implementation of this platform used as a foundation for this research consists of two main components, the hardware based on Intel Xeon Phi Knights Corner family and Kubernetes as main container orchestration solution leveraging both Xeon processors and coprocessors for maximum performance. On the application level, the platform uses Apache Kafka as Event Sourcing mechanism that allows treating the applications as state machines, providing capability to perform “step back in time” or “multi-window event processing.” We present the architecture of the platform and initial experiments that demonstrate the feasibility of our approach.