A step by step guide for a high availability Solr environment

Bogdan Martin

Senior System Engineer

Alexandru Ionescu

System Engineer

Jul 22, 2020

As discussed in our previous blog: Here’s everything you need to know about Solr Solr is an open-source enterprise-search platform from the Apache Lucene project. The platform provides full-text search, hit highlight, database integration, rich document handling, and many more features that significantly speed up your search responses.

In this step-by-step guide, we’ll walk you through installing Solr and Zookeeper in order to achieve high availability. We will kick off with the installation of Zookeeper in order to have the framework for our Solr instances.

Solr & Zookeeper on Linux

For this example, we’ll show how to install Solr and Zookeeper clusters on Linux using CentOS. This test environment features 3 Solr instances and 3 Zookeeper nodes.

We will use these IPs for the VMs:

Solr1 -> 33.33.33.10

Solr2 -> 33.33.33.11

Solr3 -> 33.33.33.12

Zookeeper1 -> 33.33.33.13

Zookeeper2 -> 33.33.33.14

Zookeeper3 -> 33.33.33.15

Test environment setup

In order to create a high availability scenario we have chosen the following setup, please see below. The diagram also features the sizing (CPU, RAM and Disk Size) for each node and instance.

Installing Zookeeper

Download the latest Zookeeper, extract it and install it.

cd /opt

wget https://downloads.apache.org/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7-bin.tar.gz

mkdir /var/lib/zookeeper

ln -s apache-zookeeper-3.5.7 zookeeper

cp zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg

[root@zookeeper1 opt]# cat zookeeper/conf/zoo.cfg

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/tmp/zookeeper

# the port at which the clients will connect

clientPort=2181

# the maximum number of client connections.

# increase this if you need to handle more clients

#maxClientCnxns=0

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

dataDir=/var/lib/zookeeper

server.1=33.33.33.13:2888:3888

server.2=33.33.33.14:2888:3888

server.3=33.33.33.15:2888:3888

On each zookeeper, you need to edit /var/lib/zookeeper/myid and add a single line containing only the text of that machine's id. This id must be unique for each Zookeeper instance.

##Zookeeper1

[root@zookeeper1 opt]# cat /var/lib/zookeeper/myid

[root@zookeeper1 opt]#

##Zookeeper2

[root@zookeeper2 opt]# cat /var/lib/zookeeper/myid

[root@zookeeper2 opt]#

##Zookeeper3

[root@zookeeper3 opt]# cat /var/lib/zookeeper/myid

[root@zookeeper3 opt]#

Remove the following from the zkEnv.sh file:

vim /opt/zookeeper/bin/zkEnv.sh

---- >>> remove these lines from the end of the file

# default heap for zookeeper server

ZK_SERVER_HEAP="${ZK_SERVER_HEAP:-1000}"

export SERVER_JVMFLAGS="-Xmx${ZK_SERVER_HEAP}m $SERVER_JVMFLAGS"

# default heap for zookeeper client

ZK_CLIENT_HEAP="${ZK_CLIENT_HEAP:-256}"

export CLIENT_JVMFLAGS="-Xmx${ZK_CLIENT_HEAP}m $CLIENT_JVMFLAGS"

Create a new Java environment file /opt/zookeeper/conf/java.env and add start parameters

[root@zookeeper1 opt]# cat /opt/zookeeper/conf/java.env

export JVMFLAGS="-Xmx2g -Xms2g"

export ZOO_LOG_DIR="/opt/zookeeper/logs"

Find the "start" function and add "-Dzookeeper.4lw.commands.whitelist=*". In our case, the start command would go like this:

nohup "$JAVA" $ZOO_DATADIR_AUTOCREATE "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" \

"-Dzookeeper.log.file=${ZOO_LOG_FILE}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" "-Dzookeeper.4lw.commands.whitelist=*" \

-XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='kill -9 %p' \

-cp "$CLASSPATH" $JVMFLAGS $ZOOMAIN "$ZOOCFG" > "$_ZOO_DAEMON_OUT" 2>&1 < /dev/null &

/opt/zookeeper/bin/zkServer.sh start

Check to see which one is the master and which ones are the followers. In our case, Zookeeper2 is the master.

[root@zookeeper2 ~]# echo mntr | nc 127.0.0.1 2181

zk_version  3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT

zk_avg_latency  0

zk_max_latency  0

zk_min_latency  0

zk_packets_received 1

zk_packets_sent 0

zk_num_alive_connections 1

zk_outstanding_requests 0

zk_server_state leader

zk_znode_count  5

zk_watch_count  0

zk_ephemerals_count 0

zk_approximate_data_size 182

zk_open_file_descriptor_count   59

zk_max_file_descriptor_count 4096

zk_followers 2

zk_synced_followers 2

zk_pending_syncs 0

zk_last_proposal_size   -1

zk_max_proposal_size -1

zk_min_proposal_size -1

[root@zookeeper2 ~]#

and the other ones are the followers:

[root@zookeeper1 ~]# echo mntr | nc 127.0.0.1 2181

zk_version  3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT

zk_avg_latency  0

zk_max_latency  0

zk_min_latency  0

zk_packets_received 1

zk_packets_sent 0

zk_num_alive_connections 1

zk_outstanding_requests 0

zk_server_state follower

zk_znode_count  5

zk_watch_count  0

zk_ephemerals_count 0

zk_approximate_data_size 182

zk_open_file_descriptor_count   57

zk_max_file_descriptor_count 4096

[root@zookeeper1 ~]#

Installing Solr

Now it’s time to install Solr by following these steps:

Download the latest Solr, extract it and install it as follows.

# --> Got to /opt and download Solr

cd /opt

wget https://archive.apache.org/dist/lucene/solr/8.5.0/solr-8.5.0.tgz

# --> Extract just the installation script from thee archive

tar xzf solr-8.5.0.tgz solr-8.5.0/bin/install_solr_service.sh --strip-components=2

# --> Execute the script against the archive with some parameters

# --> i -> Directory to extract the Solr installation archive; defaults to /opt/". The specified path must exist prior to using this script.

# --> d -> Directory for live / writable Solr files, such as logs, pid files, and index data; defaults to /var/solr

# --> u -> User to own the Solr files and run the Solr process as; defaults to solr. This script will create the specified user account if it does not exist.

# --> s -> Service name; defaults to solr

# --> p -> Port Solr should bind to; default is 8983

./install_solr_service.sh solr-8.5.0.tgz -i /opt -d /var/solr -u solr -s solr -p 8983

Extracting solr-8.5.0.tgz to /opt

Installing symlink /opt/solr -> /opt/solr-8.5.0 ...

Installing /etc/init.d/solr script ...

Installing /etc/default/solr.in.sh ...

Service solr installed.

Customize Solr startup configuration in /etc/default/solr.in.sh

*** [WARN] *** Your open file limit is currently 1024.

 It should be set to 65000 to avoid operational disruption.

 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh

*** [WARN] ***  Your Max Processes Limit is currently 3894.

 It should be set to 65000 to avoid operational disruption.

 If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to false in your profile or solr.in.sh

Warning: Available entropy is low. As a result, use of the UUIDField, SSL, or any other features that require

RNG might not work properly. To check for the amount of available entropy, use 'cat /proc/sys/kernel/random/entropy_avail'.

Waiting up to 180 seconds to see Solr running on port 8983 [-]

Started Solr server on port 8983 (pid=5500). Happy searching!

Found 1 Solr nodes:

Solr process 5500 running on port 8983

  "solr_home":"/var/solr/data",

  "version":"8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 - romseygeek - 2020-03-13 09:38:24",

  "startTime":"2020-03-30T15:48:15.498Z",

  "uptime":"0 days, 0 hours, 0 minutes, 13 seconds",

  "memory":"140.5 MB (%27.4) of 512 MB"}

[root@solr2 opt]# service solr status

Found 1 Solr nodes:

Solr process 5500 running on port 8983

  "solr_home":"/var/solr/data",

  "version":"8.5.0 7ac489bf7b97b61749b19fa2ee0dc46e74b8dc42 - romseygeek - 2020-03-13 09:38:24",

  "startTime":"2020-03-30T15:48:15.498Z",

  "uptime":"0 days, 0 hours, 0 minutes, 59 seconds",

  "memory":"145.5 MB (%28.4) of 512 MB"}

[root@solr1 opt]# cat /etc/default/solr.in.sh | grep SOLR_JAVA_MEM

SOLR_JAVA_MEM="-Xms2g -Xmx2g"

[root@solr1 opt]#

[root@solr1 ~]# cat /etc/default/solr.in.sh | grep ZK_HOST

ZK_HOST="33.33.33.13,33.33.33.14,33.33.33.15"

[root@solr1 ~]#

service solr start

Additional operational steps

Getting the health check of your collection (the IPs are from the Zookeeper ensemble)

/opt/solr/bin/solr healthcheck -c collection1 -z 33.33.33.13,33.33.33.14,33.33.33.15

  "collection":"collection1",

  "status":"healthy",

  "numDocs":0,

  "numShards":1,

  "shards":[{

   "shard":"shard1",

   "status":"healthy",

   "replicas":[

       "name":"core_node3",     "url":"http://33.33.33.12:8983/solr/collection1_shard1_replica_n1/",

       "numDocs":0,

       "status":"active",

       "uptime":"0 days, 1 hours, 32 minutes, 16 seconds",

       "memory":"552.9 MB (%27) of 2 GB"},

       "name":"core_node5",      "url":"http://33.33.33.10:8983/solr/collection1_shard1_replica_n2/",

       "numDocs":0,

       "status":"active",

       "uptime":"0 days, 1 hours, 34 minutes, 9 seconds",

       "memory":"502.5 MB (%24.5) of 2 GB",

      "leader":true},

       "name":"core_node6",

"url":"http://33.33.33.11:8983/solr/collection1_shard1_replica_n4/",

       "numDocs":0,

       "status":"active",

       "uptime":"0 days, 1 hours, 32 minutes, 57 seconds",

       "memory":"64.5 MB (%3.1) of 2 GB"}]}]}

Back-up via snapshot

A snapshot is a piece of metadata referring to the specific Lucene index commit. Solr guarantees it is preserved during the lifetime of the snapshot, in spite of subsequent index optimisations. This enables a Solr collection snapshot to provide a point-in-time, consistent state of index data even in the presence of concurrent index operations.

The snapshot creation is very fast since it just persists the snapshot metadata and does not copy the associated index files.

1. The IPs are from the Zookeeper ensemble.

2. The describe snapshot command can be issued on any of the Solr instances (not just on the master). The IPs are from the Zookeeper ensemble.

/opt/solr/server/scripts/cloud-scripts/snapshotscli.sh --describe my-first-snapshot -c collection1 -z 33.33.33.13,33.33.33.14,33.33.33.15

Name: my-first-snapshot

Status: Successful

Time of creation: Sun, 3 May 2020 08:37:18 EDT

Total number of cores with snapshot: 3

-----------------------------------

Core [name=collection1_shard1_replica_n1, leader=false, generation=1, indexDirPath=/var/solr/data/collection1_shard1_replica_n1/data/index/]

Core [name=collection1_shard1_replica_n4, leader=false, generation=2, indexDirPath=/var/solr/data/collection1_shard1_replica_n4/data/index/]

Core [name=collection1_shard1_replica_n2, leader=true, generation=1, indexDirPath=/var/solr/data/collection1_shard1_replica_n2/data/index/]

3. For a list of all the snapshots in a collection issue the command on any Solr instances (not just on the master). The IPs are from the Zookeeper ensemble.

/opt/solr/server/scripts/cloud-scripts/snapshotscli.sh --list -c collection1 -z 33.33.33.13,33.33.33.14,33.33.33.15

my-2nd-snapshot

my-first-snapshot

4. There is an option to export the created snapshot to the disk, for this access to a shared drive is needed(e.g.: a NFS share), all nodes should have the shared drive mounted in the same location/mount point and the “solr” user should have write permissions. Our Solr instances have the shared drive mounted under /mnt/nfs/var/nfs.

The command can be issued on any of the Solr instances (not just on the master). The IPs are from the Zookeeper ensemble.

[root@solr3 /]# ll /mnt/nfs/var/nfs

total 0

[root@solr3 /]#

[root@solr2 logs]# /opt/solr/server/scripts/cloud-scripts/snapshotscli.sh --export my-first-snapshot -c collection1 -d /mnt/nfs/var/nfs -z 33.33.33.13,33.33.33.14,33.33.33.15

Done. GoodBye!

[root@solr2 logs]#

[root@solr3 /]# ll /mnt/nfs/var/nfs

total 4

drwxrwxr-x 4 solr solr 4096 Jul  1 16:09 my-first-snapshot

[root@solr3 /]#

The command can be issued on any of the Solr instances (not just on the master). The IPs are from the Zookeeper ensemble.

[root@solr3 /]# ll /mnt/nfs/var/nfs

total 0

[root@solr3 /]#

[root@solr2 logs]# /opt/solr/server/scripts/cloud-scripts/snapshotscli.sh --export my-first-snapshot -c collection1 -d /mnt/nfs/var/nfs -z 33.33.33.13,33.33.33.14,33.33.33.15

Done. GoodBye!

[root@solr2 logs]#

[root@solr3 /]# ll /mnt/nfs/var/nfs

total 4

drwxrwxr-x 4 solr solr 4096 Jul  1 16:09 my-first-snapshot

[root@solr3 /]#

This concludes our installation guide for Zookeeper and Solr to achieve high availability. Bear in mind that there are many more options you can use in order to personalize your environment and make sure you have the perfect blend of functionality and performance.

Choosing the right Solr integration for your business

At Netcentric, our Solr experts have experience dealing with various integrations across multiple platforms. This enables us to offer a wide range of services that transform your Solr implementation in conjunction with the Adobe Stack. By making sure that all your needs are covered, we can add value to each line of business that features Solr and Adobe Stack instances.

Get in touch with our experts today to discover how we can help take your Solr implementation to the next level.

Developer Circle, Data Management

Bogdan Martin

Senior System Engineer

Alexandru Ionescu

System Engineer

More Developer Circle

How we manage complex client setups

Alexandru Abager

Support Engineer
Patterns to secure systems end-to-end in a large enterprise

Ashwin Sheshadri

Principal Multi-Experience Architect
Using Private Dependencies in Cloud Manager Builds
Georg Henzler

Principal Solution Architect

Georg has grown up with the web space from the very beginning and has helped enterprises to design their solution for it from the time when images were first added to their web sites. Today he shapes the way Netcentric delivers IT projects technically by continuously evolving the development process and automating infrastructure to the last slash. Amongst being a committer for Apache Sling (the web framework in AEM) he has contributed to many other open source projects and talks regularly at tech conferences. He loves Netcentric for its culture of "Share, discuss and bring ideas to life"!