Spring Cloud Config : Externalise your configuration and avoid server restart !!! code example

Spring Cloud Config provides server and client-side support for externalised configuration in a distributed system. With the Config Server we have a central place to manage external properties for applications across all environments. The concepts on both client and server map identically to the Spring Environment and PropertySource abstractions, so they fit very well with Spring applications, but can be used with any application running in any language. As an application moves through the deployment pipeline from dev to test and into production we can manage the configuration between those environments and be certain that applications have everything they need to run when they migrate.

In the example below we will see how we can externalise and manage configuration from github repository and file system.

Spring Cloud Config Server

The Server provides an HTTP, resource-based API for external configuration. We can create/start a config server with @EnableConfigServer annotation at the Spring Boot application startup class.

Code:

@EnableConfigServer

@SpringBootApplication

public class ConfigServiceApplication {

    

    public static void main(String[] args) {

        SpringApplication.run(ConfigServiceApplication.class, args);

    }

}

application.properties file:

server.port=8888

# if you want to manage config files from local file system uncomment the property bellow 

#spring.cloud.config.server.git.uri=file://<your_local_system_directory>

# if you want to manage config files from github repository then uncomment the property below

# alongwith user name and password

#spring.cloud.config.server.git.uri=<github_url>

spring.cloud.config.server.git.searchPaths=<some-name>

#spring.cloud.config.server.git.username=

#spring.cloud.config.server.git.password=

finally in maven dependencies include 

<dependency>

    <groupId>org.springframework.cloud</groupId>

    <artifactId>spring-cloud-starter-config</artifactId>

    <version>1.2.0.RELEASE</version>

</dependency>

Lets assume that spring.cloud.config.server.git.searchPaths attribute value is sample-config and we are supporting default, dev and production environment, so your github repo/local file system will have three different files like

sample-config.properties

sample-config-dev.properties

sample-config-production.properties

Now start the application and access 

http://localhost:8888/sample-config/defaut

output: {"name":"sample-config","profiles":["defaut"],"label":"master","version":"f57f675e23c02a9e2f8422b01f07302d41d9774f","propertySources":[{"name":"<github-default-url>","source":{"message":"some message for default"}}]}

http://localhost:8888/sample-config/dev

{"name":"sample-config","profiles":["dev"],"label":"master","version":"f57f675e23c02a9e2f8422b01f07302d41d9774f","propertySources":[{"name":"<github-default-url>","source":{"message":"some message for dev"}}, ........... ]}

http://localhost:8888/sample-config/production

{"name":"sample-config","profiles":["production"],"label":"master","version":"f57f675e23c02a9e2f8422b01f07302d41d9774f","propertySources":[{"name":"<github-default-url>","source":{"message":"some message for production"}}, ........ ]}

we are assuming here that we have only one property with key message in the property files.

So our spring cloud config server is now ready to support environment specific configuration properties for the client.

we can also health check with http://localhost:8888/sample-config/health

Spring Cloud Config Client

A Spring Boot application can take immediate advantage of the Spring Config Server. Lets see how -

@SpringBootApplication

@RefreshScope

public class ConfigClientApplication {

    public static void main(String[] args) {

        SpringApplication.run(ConfigClientApplication.class, args);

    }

}

@RestController

class MessageRestController {

    @Value("${message:Hello default}")

    private String message;

    @RequestMapping("/message")

    String getMessage() {

        return this.message;

    }

}

So we have a REST endpoint /message which returns the value of the key message from property files.

application.properties file

server.port=8080

spring.application.name=sample-config # this is the name we used in config server

#URL for config server

spring.cloud.config.uri=http://localhost:8888

# Active profile

spring.profiles.active=dev

finally add the dependency below in pom.xml

<dependency>

<groupId>org.springframework.cloud</groupId>

<artifactId>spring-cloud-starter-config</artifactId>

<version>1.2.0.RELEASE</version>

</dependency>

Lets start the server and access http://localhost:8080/message  [the output will be some message for dev] 

If we change the spring.profiles.active value the output will be changed accordingly. 

@RefreshScope

If we change the value of any of the key (value of the property files attribute key for any environment) we don't even need to restart the Config Server and Config Client, Just send HTTP POST request to http://localhost:8080/refresh the updated values will be reflected automatically. 

So we can use Spring Cloud config server to - 

1. Externalise the configuration properties (property files for applications) either to file system or github repository.

2. We can get the support of different profiles like dev, production seamlessly.

3. If at runtime we need to change any configuration attribute value we don't even need to restart the server.

Source code is available at [https://github.com/badalb/spring-cloud-microservices-security]

JAVA In memory cache with time and size based eviction

In this blogpost we will see how we can create an in-memory cache with time and size based eviction. Time or time to live based eviction will check cache objects with last access time greater than a predefined value where as size based eviction will remove objects with LRU algorithm

Key for cache: 

public class CacheKey<T> {

private T key;

public CacheKey(T key) {

this.key = key;

}

public T getKey() {

return key;

}

public void setKey(T key) {

this.key = key;

}

}

Object as value for cache:

public class CachedObject<T> {

private long lastAccessedTime;

private T value;

public CachedObject(T value) {

this.lastAccessedTime = System.currentTimeMillis();

this.value = value;

}

public long getLastAccessedTime() {

return lastAccessedTime;

}

public void setLastAccessedTime(long lastAccessedTime) {

this.lastAccessedTime = lastAccessedTime;

}

public T getValue() {

return value;

}

public void setValue(T value) {

this.value = value;

}

LRU based cache storage :

public class LRUCache<K, V> {

private final int maxSize;

private ConcurrentHashMap<K, V> map;

private ConcurrentLinkedQueue<K> queue;

public LRUCache(final int maxSize) {

this.maxSize = maxSize;

map = new ConcurrentHashMap<K, V>(maxSize);

queue = new ConcurrentLinkedQueue<K>();

}

public void put(final K key, final V value) {

if (map.containsKey(key)) {

// remove the key from the FIFO queue

queue.remove(key);

}

while (queue.size() >= maxSize) {

K oldestKey = queue.poll();

System.out.println("Key to remove : " + oldestKey);

if (null != oldestKey) {

map.remove(oldestKey);

}

}

queue.add(key);

map.put(key, value);

}

public V get(final K key) {

if (map.containsKey(key)) {

// remove from queue and add it again in FIFO queue

queue.remove(key);

queue.add(key);

}

return map.get(key);

}

public void remove(final K key) {

if (map.containsKey(key)) {

// remove from queue and add it again in FIFO queue

queue.remove(key);

map.remove(key);

}

}

public ConcurrentHashMap<K, V> getMap() {

return map;

}

public void setMap(ConcurrentHashMap<K, V> map) {

this.map = map;

}

public ConcurrentLinkedQueue<K> getQueue() {

return queue;

}

public void setQueue(ConcurrentLinkedQueue<K> queue) {

this.queue = queue;

}

public int getMaxSize() {

return maxSize;

}

}

In memory cache implementation: 

public class InMemoryCache<K,V> {

private long timeToLive;

private LRUCache<CacheKey<?>, CachedObject<?>> cache;

public InMemoryCache(final long lifeTime, final long timerInterval, final int maxItems) {

this.timeToLive = lifeTime * 1000;

cache = new LRUCache<CacheKey<?>, CachedObject<?>>(maxItems);

if (timeToLive > 0 && timerInterval > 0) {

Thread t = new Thread(new Runnable() {

public void run() {

while (true) {

try {

Thread.sleep(timeToLive);

} catch (InterruptedException ex) {

}

cleanup();

}

}

private void cleanup() {

long now = System.currentTimeMillis();

List<CacheKey<?>> deleteKey = new ArrayList<CacheKey<?>>();

CacheKey<?> key = null;

CachedObject<?> cObject = null;

for (Map.Entry<CacheKey<?>, CachedObject<?>> entry : cache.getMap().entrySet()) {

key = entry.getKey();

cObject = entry.getValue();

if (cObject != null && (now > (timeToLive + cObject.getLastAccessedTime()))) {

deleteKey.add(key);

}

}

for (CacheKey<?> cacheKey : deleteKey) {

cache.remove(cacheKey);

Thread.yield();

}

}

});

t.setDaemon(true);

t.start();

}

}

public void put(CacheKey<?> key, CachedObject<?> value) {

cache.put(key, value);

}

public CachedObject<?> get(CacheKey<?> key) {

CachedObject<?> c = (CachedObject<?>) cache.get(key);

if (c == null)

return null;

else {

c.setLastAccessedTime(System.currentTimeMillis());

return c;

}

}

public void remove(CacheKey<?> key) {

cache.remove(key);

}

public int size() {

return cache.getMap().size();

}

}

So, in LRUCache storage implementation, once the predefined storage capacity is reached, it will check objects fetched earliest and remove that from storage. The key access order is maintained in a concurrent linked queue.

InMemoryCache keeps on running a background thread to check objects which has reached the time to live (idle time) and remove them from the cache.

Implement LRU cache in Java

In this blogpost we will see how we can create a LRU cache using JAVA collection APIs.

LRU (Least Recently Used) cache discards least recently used element from the cache when we need to free up some space from cache. This algorithm requires keeping track of what was used when, which is expensive if one wants to make sure the algorithm always discards the least recently used item. General implementations of this technique require keeping "age bits" for cache-lines and track the "Least Recently Used" cache-line based on age-bits. In such an implementation, every time a cache-line is used, the age of all other cache-lines changes.

We will use ConcurrentHashMap as cache storage and ConcurrentLinkedQueue to keep track of element access order from the cache. Lets see how the code looks like - 

import java.util.concurrent.ConcurrentHashMap;

import java.util.concurrent.ConcurrentLinkedQueue;

public class LRUCache<K, V> {

private final int maxSize;

private ConcurrentHashMap<K, V> map;

private ConcurrentLinkedQueue<K> queue;

public LRUCache(final int maxSize) {

this.maxSize = maxSize;

map = new ConcurrentHashMap<K, V>(maxSize);

queue = new ConcurrentLinkedQueue<K>();

}

public void put(final K key, final V value) {

if (map.containsKey(key)) {

// remove the key from the FIFO queue

queue.remove(key);

}

while (queue.size() >= maxSize) {

K oldestKey = queue.poll();

if (null != oldestKey) {

map.remove(oldestKey);

}

}

queue.add(key);

map.put(key, value);

}

public V get(final K key) {

if (map.containsKey(key)) {

// remove from queue and add it again in FIFO queue

queue.remove(key);

queue.add(key);

}

return map.get(key);

}

}

Lets see the main class 

public class ValidateLRUCache {

public static void main(String[] args) {

LRUCache<Integer, String> cache = new LRUCache<>(5);

cache.put(1, "A");

cache.put(2, "B");

cache.put(3, "C");

cache.put(4, "D");

cache.put(5, "E");

// key 5 moved ahead

System.out.println(cache.get(5));

// put new element to cache. this will evict key 1 from cache

cache.put(6, "F");

//this will print null

System.out.println(cache.get(1));

}

}

In this example we are using ConcurrentLinkedQueue to maintain the access order. The poll method retrieves and removes the head of this queue, or returns null if this queue is empty. So while putting elements to the cache storage if we exceed the size limit, least recently used keys will be removed from the queue and element for that key will be removed from the cache. Similarly if we fetch some element from cache we have to change the access order. 

Distributed SolrCloud setup with external ZooKeeper ensemble

We all know that Solr search performs better than database queries because of "inverse index" rather than database queries with a full table scan. Databases and Solr have complementary strengths and weaknesses though. In this blogpost we will set up a SolrCloud just like a production system.

SolrCloud or Solr Master Slave

SolrCloud and master-slave both address four particular issues:

• Sharding
• Near Real Time (NRT) search and incremental indexing
• Query distribution and load balancing
• High Availability (HA) 

If our application just reads data from Solr and need high availability on reading data from Solr then a simple one master to many slave hierarchy is more than sufficient. But if you are looking out for high availability on writing to Solr too, then SolrCloud is a right option

Is SolrCloud is better? Maintaining SolrCloud needs a good infrastructure and have to look out the availability of ZooKeepers, and nodes health, high performance disk for better replication speed etc. But, other than this we don't need to worry about Data consistency among nodes as this will be taken care by SolrCloud.

In which cases is better to coose SolrCloud? 

When we need high availability on Solr Writes as well as reads, we have to go for SolrCloud. Also, if we cannot afford bigger machines to have one single node, then we can split index to shards and keep it under smaller config machines.

In which cases is better to choose Solr Replication? 

When our application does not write in real time to SOLR, Replication is enough and no need to get complicated with SolrCloud. Also, its comparatively easy to setup Master Slave than SolrCloud

Sharding and Data consistency, automatic rebalancing of shards are better in SolrCloud. Query distribution and load balancing is automatic for SolrCloud, in sharded environment for master slave we need to use distributed query.

To setup a distributed SolrCloud we must have the following - 

1. At least 6 serves (3 for ZooKeeper cluster setup and 3 for SolrCloud setup)
2. Zookeper
3. Solr 6

For this blogpost we will be using a single system with three different ZooKeeper and three different Solr6 instances running on different port.

ZooKeeper Cloud Setup 

1. Download Apache ZooKeeper 3.4.6 is the version I am using
2. Create a directory, lets assume zk_cluster under $<home> .
3. Create three instances for ZooKeeper under zk_cluster, lets assume they are zookeeper-3.4.6_1,           zookeeper-3.4.6_2, zookeeper-3.4.6_3.
4. Create data and logs directory under zk_cluster directory.
5. Create ZooKeeper Server ID, basically this file reside in the ZooKeeper data directory.

At this point of time your ZooKeeper cluster will look like -

zk_cluster
|
    |-data
|---zookeeper-3.4.6_1
|--myid (content numeric 1)
|---zookeeper-3.4.6_2
|--myid (content numeric 2)
|---zookeeper-3.4.6_3
|--myid (content numeric 3)
|-log
|---zookeeper-3.4.6_1
|---zookeeper-3.4.6_2
|---zookeeper-3.4.6_3
|-zookeeper-3.4.6_1
|-zookeeper-3.4.6_2
|-zookeeper-3.4.6_3

6. Preparing ZooKeeper configuration called zoo.cfg at $<home>/zk_cluster/{zookeeper-3.4.6_1}/conf/zoo.cfg.  Here I will show you for Server 1. We have to perform same steps with appropriate values (clientPort, dataDir, dataLogDir) for respective ZooKeeper server.

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial synchronization phase can take
initLimit=10

# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5

# the directory where the snapshot is stored.
# Choose appropriately for your environment
dataDir=$<home>/zk-cluster/data/zookeeper-3.4.6_1

# the port at which the clients will connect
clientPort=2181 <change for other instances>

# the directory where transaction log is stored.
# this parameter provides dedicated log device for ZooKeeper
dataLogDir=$<home>/zk-cluster/logs/zookeeper-3.4.6_1

# ZooKeeper server and its port no.
# ZooKeeper ensemble should know about every other machine in the ensemble
# specify server id by creating 'myid' file in the dataDir
# use hostname instead of IP address for convenient maintenance
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890

7. Once zoo.cfg created for all the server then we can start the ZooKeeper Servers. ZooKeeper supports the following commands

• start
• start-foreground
• stop
• restart
• status
• upgrade
• print-cmd

SolrCloud Setp

Before we create the Solr instances, we'll need to create a configset in order to create a collection to shard and replicate across multiple instances.  Creating a configset is very specific to our  collection. We can use the pre-built configsets that come with Solr 6, they are located in solr-6.0.0/server/solr/configsets and we don't have to do anything.

A custom configset requires taking care of path and third party libraries defined in solrconfig.xml file.We also have to create/update the schema.xml as necessary to map data from the source to a Solr document

Lets assume we are creating a configset named solr_cloud_example simply copying the content of  basic_configs. Additional libraries and schema can be updated before we actually start creating indexes.

Uploading a configset to Zookeeper

This is relevant if we want to upload  configuration ahead of time instead of specifying the configuration to use in the "create" command or if we are using the Collections API to issue a "create" command via the REST interface.


To upload the configset, we have to use zkcli.sh which is in <BASE_INSTALL_DIR>/solr-6.0.0/server/scripts/cloud-scripts.  Lets go to that directory and issue the following command:

./zkcli.sh -zkhost localhost:2181,localhost:2182,localhost:2183 -cmd upconfig -confname < solr_cloud_example > -confdir <base_installation_dir>/solr-6.0.0/sever/solr/configsets/< solr_cloud_example >/conf

This will upload the config directory in ZooKeeper cluster we have setup earlier.

Creating Solr Instances

Under the <base_directory> create a directory <solr_cluster> , download and copy three solr 6 installations. So the directory structure looks like  -

<base_dir>/<solr_cluster>
                            |
                            |-- solr-6.0.0_1
                                    | -- server
                                           |--solr
                                                 |--configsets
                                                       |-- <solr_cloud_example>
                                   
                            |-- solr-6.0.0_2
                            |-- solr-6.0.0_3


Now we are ready with the setup.

Start SolrCloud

At this point of time we have all the setup ready, before we start solr instances make sure the zookeeper cluster is up and running.

Goto
<base_dir>/solr-cluster/solr-6.0.1 and execute
bin/solr start -cloud  -p 8983 -z localhost:2181,localhost:2182,localhost:2183 -noprompt

Goto
<base_dir>/solr-cluster/solr-6.0.2 and execute
bin/solr start -cloud  -p 8984 -z localhost:2181,localhost:2182,localhost:2183 -noprompt

Goto
<base_dir>/solr-cluster/solr-6.0.3 and execute
bin/solr start -cloud  -p 8985 -z localhost:2181,localhost:2182,localhost:2183 -noprompt

once all the instances are running just type
http://localhost:8983/solr/admin/collections?action=CREATE&name=test_solr_cloud&numShards=2&replicationFactor=2&maxShardsPerNode=2
&collection.configName= solr_cloud_example to create a collection named solr_cloud_example

Now go to http://localhost:8983/solr/#/~cloud  you will see the collection along with shards and replications in different nodes

Solr Master - Slave Configuration with DataImportHandler & Scheduling

In this post we will se how we can setup Solr Master - Slave replication setup as shown below -

For simplicity lets assume that we have two nodes node1 and node2. Node1 is the master node and Node2 is the slave node.

1. Install solr-5.3.1 on both Node1(master) and Node2(slave)
2. Create Solr core using the command
    $> bin/solr create [-c name] [-d confdir] [-n configName] [-shards #] [-replicationFactor #] [-p           port]
on both Node1 and Node2

Lets assume the name of the core is test_core.

So in both the instance if we go to ${SOLR_HOME}/server/solr we will see test_core which have conf directory , core.properties file and data directory.

Now lets start with master slave configuration -

Master Setup 

If we navigate to conf directory within the test-core directory under /server/solr we will see solrconfig.xml file

Edit the file and add

<requestHandler name="/replication" class="solr.ReplicationHandler">

    <lst name="master">

         <str name="enable">${master.replication.enabled:false}</str>

         <str name="replicateAfter">commit</str>

         <str name="replicateAfter">optimize</str>

        <str name="replicateAfter">startup</str>

    </lst>

</requestHandler>

add master.replication.enabled=true in core.properties file located in /solr directory.

Slave Setup

If we navigate to conf directory within the test-core directory under /server/solr we will see solrconfig.xml file

Edit the file and add

<requestHandler name="/replication" class="solr.ReplicationHandler">

     <lst name="slave">

           <str name="enable">${slave.replication.enabled:false}</str>

           <str name="masterUrl">http://${masterserver}/solr/${solr.core.name}/replication</str>

          <str name="pollInterval">00:05:00</str></lst>

</requestHandler>

add 

slave.replication.enabled=true

masterserver=52.33.134.44:8983

solr.core.name=<core_name> (test_core)

in core.properties file located in /solr directory.

Thats it we are done with master slave configuration.

DataImportHandler

Using solr DataImportHandler we can create indexes in solr directly from data store like MySQL Oracle, Postgre SQL etc.

Lets continue with previous example to configure a data import handler
1.  Edit solrconfig.xml file under conf directory of your core and add -

<requestHandler name="/dataimport"                           class="org.apache.solr.handler.dataimport.DataImportHandler">
  <lst name="defaults">
      <str name="config">data-config.xml</str>
  </lst>
</requestHandler>

2. Create data-config.xml file within the conf directory with following content-

<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" url="" user="" password=""/>
    <document name="">
        <entity name="" query=""
deltaQuery="<some_date_condition> &gt; '${recommendation.last_index_time}';">
 <field column="" name="" />
            .
.
.
.
      <field column="allcash_total_annualized_return_growth" name="Allcash_total_annualized_return_growth" />
        </entity>
    </document>
</dataConfig>

3. Create corresponding filed mapping in managed-schema file for index creation.

4. Make sure you have the jar file for Driver class is available in lib directory or any other directory and you have mentioned that in solrconfig.xml file like

<lib dir="${solr.install.dir:../../../..}/contrib/extraction/lib" regex=".*\.jar" />

We are done with DataImportHandler configuration.

Scheduling: 

Solr by default don,t support scheduling for delta import.
Clone either of

1. https://github.com/badalb/solr-data-import-scheduler.git
2. https://github.com/mbonaci/solr-data-import-scheduler.git

Create a jar file and put that jar file in {SOLR_HOME}/ server/solr-webapp/ webapp/ WEB-INF / lib directory

3. Make sure, regardless of whether you have single or multi-core Solr, that you create dataimport.properties located in your solr.home/conf (NOT solr.home/core/conf) with the content like

 #  to sync or not to sync
#  1 - active; anything else - inactive
syncEnabled=1

#  which cores to schedule
#  in a multi-core environment you can decide which cores you want syncronized
#  leave empty or comment it out if using single-core deployment
syncCores=coreHr,coreEn

#  solr server name or IP address
#  [defaults to localhost if empty]
server=localhost

#  solr server port
#  [defaults to 80 if empty]
port=8080

#  application name/context
#  [defaults to current ServletContextListener's context (app) name]
webapp=solrTest_WEB

#  URL params [mandatory]
#  remainder of URL
params=/select?qt=/dataimport&command=delta-import&clean=false&commit=true

#  schedule interval
#  number of minutes between two runs
#  [defaults to 30 if empty]
interval=10

4. Add application listener to web.xml of solr web app ({SOLR_HOME}/ server/solr-webapp/WEB-INF/web.xml)

<listener>
  <listener-class>org.apache.solr.handler.dataimport.scheduler.ApplicationListener</listener-class>
</listener>

Restart Solr so that changes are reflected.

Happy searching .....

Integrating Tableau Desktop with Spark SQL

In this post we will see how we can integrate Tableau Desktop with Spark SQL. Tableau’s integration with Spark brings tremendous value to the Spark community – we can visually analyse data without writing a single line of Spark SQL code. That’s a big deal because creating a visual interface to our data expands the Spark technology beyond data scientists and data engineers to all business users. The Spark connector takes advantage of Tableau’s flexible connection architecture that gives customers the option to connect live and issue interactive queries, or use Tableau’s fast in-memory database engine.

Software requirements :-

We will be using the following softwares to do the integration -
1. Tableau Desktop-9-2-0
2. Hive 1.2.1
3. Spark 1.4.0 for Hadoop 2.6.0

We can skip the Hive and can directly work with Spark SQL. For this example we will use Hive, import Hive tables to Spark SQL and will Integrate them with Tableau SQL.

Hive Setup :-

1. Download and install Hive 1.2.1.
2. Download and copy mysql connector jar file to ${HIVE_HOME}/lib directory so hive will use           MySql metastore.
3. Start Hive ${HIVE_HOME}/bin $./hive
4. Create some table and insert data to that table

create table product(productid INT, productname STRING, proce FLOAT, category STRING) ROW FORMAT DELIMITED
        FIELDS TERMINATED BY ',';

INSERT INTO TABLE product VALUES(1,Book,25,Statonery);
INSERT INTO TABLE product VALUES(2,Pens,10,Stationery);
INSERT INTO TABLE product VALUES(3,Sugar,40.05,House Hold Item);
INSERT INTO TABLE product VALUES(4,Furniture,1200,Interiors);

Hive setup is complete now.

Spark Setup :-

1. Download and extract Spark 1.5.2 for Hadoop 2.6.0
2. Copy hive-site.xml from ${HIVE_HOME}/conf directory to ${SPARK_HOME}/conf directory
3. Replace all "s" from time values like 0s to 0 or <xyz>ms to <xyz> else it might give us Number         Format Exception
4. Define  SPARK MASTER IP export SPARK_MASTER_IP=<host_ip_addr>  in spark-env.sh file  (without this thrift server will not work) located at ${SPARK_HOME}/conf directory

5. Start spark master and slave

1. ${SPARK_HOME}/sbin $./start-master.sh 
2. ${SPARK_HOME}/sbin $./start-slaves.sh 

6. Goto http://localhost:8080/   and check that worker has started

Now time to start Thrift server -

7.  ${SPARK_HOME}/sbin $ ././start-thriftserver.sh --master spark://<spark_host_ip>:<port> --driver-class-path ../lib/mysql-connector-java-5.1.34.jar  --hiveconf hive.server2.thrift.bind.host localhost --hiveconf hive.server2.thrift.port 10001

It will start thrift server on 10001 port

8. Go to http://localhost:8080/  and check spark sql application has started

Now go to Tableau Desktop

1. Select Spark Sql.
2. Enter host as localhost, enter thrift server port from step here its 10001
3. Select type as SparkThriftServer, Authentication as User Name 
4. Keep rest of the fields empty and click on OK

You are done!!! Happy report building using Tableau-Spark.

Vagrant - Puppet Java development environment setup

Vagrant:-

Vagrant is an open-source (MIT) tool for building and managing virtualised development environments

Simply put, Vagrant makes it really easy to work with virtual machines. According to the Vagrant docs:

"If you’re a designer, Vagrant will automatically set everything up that is required for that web app in order for you to focus on doing what you do best: design. Once a developer configures Vagrant, you don’t need to worry about how to get that app running ever again. No more bothering other developers to help you fix your environment so you can test designs. Just check out the code, vagrant up, and start designing."

Puppet:-

Puppet is a configuration management tool that is extremely powerful in deploying, configuring, managing, maintaining, a server machine.

Librarian Puppet:-

Librarian-puppet is a project by the amazing Tim Sharpe to take Librarian, a general reimplementation of Bundler, and provide an implementation for the Puppet ecosystem. It has support for installing Puppet modules from the Puppet Forge as well as Github, and provides any number of other features like version locking of installed modules.

Simply, we can have a virtual box and vagrant setup and we can write shell scripts/batch files to install softwares based on the development environment.

If we use puppet and librarian puppet along with with virtual box we only need to concentrate on setting up vagrant, puppet, librarian puppet rest will be taken care of by puppet module itself.

Virtual Box Setup

Download virtual box from here [https://www.virtualbox.org/wiki/Downloads] for the environment you are working on. Once downloaded follow the instructions to install

If we want to work with vagrant we must have a virtual box.

Vagrant Setup

With virtual box installed we are ready to go ahead with vagrant installation. 

Download vagrant from here [https://www.vagrantup.com/downloads.html]. Once downloaded follow the instructions to install.

Puppet Setup

You can write a environment specific shell script/batch file to install puppet manually or shell script could be executed from Vargrant file itself while executing the command  $ vagrant up For simplicity lets assume we will manually execute the shell script/batch file to install puppet.

Librarian Puppet Setup

You can write a environment specific shell script/batch file to install librarian puppet manually or shell script could be executed from Vargrant file itself while executing the command  $ vagrant up For simplicity lets assume we will manually execute the shell script/batch file to install librarian puppet.

Now we have a virual box, vagrant, puppet, librarian puppet installed.

Lets create our First Instance 

$ mkdir my_first_instance

$ cd my_first_instance

$ vagrant init precise32  http://files.vagrantup.com/precise32.box

once successfully executed it will create Vagrantfile in the empty directory created above with some default settings. Now execute 

$vagrant up

Wait for few minutes, this will start the virtual box [ubuntu machine]. Now using ssh we can interact with the virtual box

$vagrant ssh 

[

Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-23-generic-pae i686)

 * Documentation:  https://help.ubuntu.com/

New release '14.04.3 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Welcome to your Vagrant-built virtual machine.

Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2

vagrant@precise32:~$  ]

$vagrant ssh exit  - command to exit virtual box

$vagrant suspend -  command to stop virtual machine

$vagrant destroy - command to remove the setup

Development Environment Setup

At this point we have a virtual box up and running, vagrant setup, puppet and librarian puppet installed.

$ cd my_first_instance

$ mkdir puppet

$ cd puppet

$ mkdir  manifests

$ mkdir modules

$ touch Puppetfile

$ cd manifests

$ touch default.pp

the Puppetfile will have the modules required for your development and default.pp file will have the dependencies.

Sample Puppetfile

[forge "http://forge.puppetlabs.com"

mod "puppetlabs/stdlib", "3.2.1"

mod "puppetlabs/apt", "1.5.0"

mod "puppetlabs/mysql", "2.2.3"

#mod "puppetlabs/rabbitmq", "5.0.0"

mod "thomasvandoren/redis", "0.10.0"

mod "jbussdieker/memcached"

mod "puppetlabs/git"

mod "tylerwalts/jdk_oracle"

mod "gini/gradle"]

Puppet modules could be found executing the command $ sudo puppet module search {mysql}

In default.pp file under manifests define the dependencies like -

# --- MySQL --- #

class { '::mysql::server':

 root_password => 'foo'

}

Once defined go to puppet directory under parent directory [$ cd my_first_instance/puppet] and execute 

$ sudo librarian-puppet install - this will install modules under puppet directory

$ cd ..

$ vagrant reload --provision

Once provisioning has been successfully completed it will install the software module to virtual box. We can now login to vm and start using it.

Sample virtual box setup scripts are available here [https://github.com/badalb/vagrant-java]