Real Time Data Streaming with Epoch

Streams of data are becoming ubiquitous today – clickstreams, log streams, event streams, and more. Building a clickstream monitoring system, for example, where data is in the form of a continuous clickstream rather than discrete data sets, requires the use of continuous processing rather than ad-hoc, one-time queries.

In this blogpost we will explore how we can build a real-time monitoring system with Spring framework, Kafka, Storm, Redis, Node.Js and EpochJS ( https://fastly.github.io/epoch/ ).

We will have a producer producing the stream data to a Kafka topic and a consumer, Storm spout consuming the stream and storm bolts publishing those streams to redis. A simple node application subscribed to redis continuously push consumed stream to epoch using the open socket. Epoch creates a realtime view of the stream to the end user. So the architecture looks like -

Implementation code code be found here -

Spring-Kafka :- https://github.com/badalb/spring-kafka.git
Real Time Streaming :- https://github.com/badalb/epoch-realtime-data-stream.git