Real Time Data Streaming with Epoch

Streams of data are becoming ubiquitous today – clickstreams, log streams, event streams, and more. Building a clickstream monitoring system, for example, where data is in the form of a continuous clickstream rather than discrete data sets, requires the use of continuous processing rather than ad-hoc, one-time queries.

In this blogpost we will explore how we can build a real-time monitoring system with Spring framework, Kafka, Storm, Redis, Node.Js and EpochJS ( https://fastly.github.io/epoch/ ).

We will have a producer producing the stream data to a Kafka topic and a consumer, Storm spout consuming the stream and storm bolts publishing those streams to redis. A simple node application subscribed to redis continuously push consumed stream to epoch using the open socket. Epoch creates a realtime view of the stream to the end user. So the architecture looks like -

Implementation code code be found here -

Spring-Kafka :- https://github.com/badalb/spring-kafka.git
Real Time Streaming :- https://github.com/badalb/epoch-realtime-data-stream.git 

External Project Dependency in Gradle

Lets assume that you are working in a multi-module and distributed project and projects are organised like the structure below -

root1

       |__ project _1

       |__ project _2

root2

        |__ project _X

        |__ project _Y

Now we want to add dependency of project_1 of root1 in project_Y of root2, If you achieve that easily with -

1. Add the snippet below in settings.gradle file of project_Y

include ":project_1"

project(":project_1").projectDir = file("<path_to_project_1>")

2. Add dependency of project_1 in build.gradle file of project_Y

compile project(":project_1")