This blog post is about the configuration of Apache Cassandra for running a cluster of 3 instances on a single host. Basically C* is not really build to run in a multi instances environment, but for many cases and reasons you might run a C* cluster on a single server. The principal reason is for tests cases.
Context
The objective is to have a Cassandra ring of 3 instances on 1 host. For that, we will use:
- apache-cassandra-3.0.9 (community version)
- Separated disks for a segregation between binaries, data and logs: binaries, data and logs will be in separated mount points
- Multiple IPs interfaces for each instance (virtual interfaces in our case)
- Instances name: csd1, csd2, csd3
- Cluster name: dbitest
Binary installation
Download the binaries from the Apache website, http://www.apache.org/dyn/closer.lua/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz and install it. In this post we are going to install the 3.0.9 release.
wget http://www.pirbot.com/mirrors/apache/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz
Extract it:
tar -zxvf apache-cassandra-3.0.9-bin.tar.gz
Place the software into the product directory:
cp –r apache-cassandra-3.0.9 /u00/app/cassandra/product/
Instances configuration
First, we will create the admin directory for each instance. The admin directory contains the following sub folders:
- /u00/app/cassandra/admin/
- /backup: soft link to the backup mount point
- /dump: soft link to the dump mount point
- /etc: Cassandra configuration files, such as Cassandra.yaml, logback.xml and Cassandra-env.sh.
- /log: soft link to the log file from the log mount point
- /pid: contain the pid of the C* process
To create these directories, use the following loop.
for i in 1 2 3 ; do mkdir -p /u00/app/cassandra/admin/csd$i && mkdir /u00/app/cassandra/admin/csd$i/backup && mkdir /u00/app/cassandra/admin/csd$i/dump && mkdir /u00/app/cassandra/admin/csd$i/etc && mkdir /u00/app/cassandra/admin/csd$i/log && mkdir /u00/app/cassandra/admin/csd$i/pid && mkdir /u00/app/cassandra/admin/csd$i/product; done
Next copy the C* configuration files into each admin/csd$i/etc directory. Every instance has its own configuration files.
for i in 1 2 3; do cp -r /u00/app/cassandra/admin/csd$i/product/apache-cassandra-3.0.9/conf/* /u00/app/cassandra/admin/csd$i/etc/; done
Now, create the three data directories for each instance.
for i in 1 2 3; do sudo mkdir -p /u01/cassandradata/csd$i && sudo chown -R cassandra:cassandra /u01/cassandradata/csd$i; done
Configure virtual network interfaces
You must create additional virtual network interfaces for each C* instance, to use a different listen_address and rpc_address for each instance.
sudo ifconfig enp0s3:0 192.168.56.105 sudo ifconfig enp0s3:1 192.168.56.106
Then, you have 3 interfaces (default + 2 virtuals), for your C* instances.
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.56.104 … enp0s3:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.56.105 … enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.56.106 … enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 … lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 …
As all the required files are in place, we can now change the 3 main configuration files, to set up the three instances (csd1, csd2 and csd3).
cassandra-env.sh:
Use a different “JMX_PORT” for each node. For instance: 7199 for csd1, 7299 for csd2 and 7399 for csd3.
JMX will bind to the local host IP by default, you can use the default.
cassandra.yaml
The central configuration file of Apache Cassandra. Change the following parameters:
- cluster_name: ‘dbitest’ All instances must have the same cluster name
- commitlog_directory: /u01/cassandradata/csd$i/commitlog
- data_file_directories:/u01/cassandradata/csd$i/data
- saved_caches_directory: /u01/cassandradata/csd$i/saved_caches
- listen_address: 192.168.56.104
- rpc_address: 192.168.56.104
- seeds: specify the list of IP of the seed nodes
logback.xml
The configuration file for C* logs. Make the following changes:
<file>${cassandra.logdir}/system.log</file> to <file>/u00/app/cassandra/admin/csd$i/log/system.log</file> <fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern> to <fileNamePattern>/u00/app/cassandra/admin/csd$i/log/system.log.%i.zip</fileNamePattern> <file>${cassandra.logdir}/debug.log</file> to <file>/u00/app/cassandra/admin/csd$i/debug.log</file> <fileNamePattern>${cassandra.logdir}/debug.log.%i.zip</fileNamePattern> to <fileNamePattern>/u00/app/cassandra/admin/csd1/log/debug.log.%i.zip</fileNamePattern>
Make the change for the three instances.
Starting the nodes
Before starting each node, you must dynamically set the environment variables for each of them.
$CASSANDRA_HOME=/u00/app/cassandra/product/apache-cassandra-3.0.9 $CASSANDRA_CONF=/u00/app/cassandra/admin/csd$i/etc/
After, setting the variables for each node you can start Apache Cassandra by command-line:
./$CASSANDRA_HOME/bin/cassandra –f
Execute the same command in a separated shell, for each node. Do not forget to set the environment variables $CASSANDRA_HOME and $CASSANDRA_CONF.
Cluster status
Verify the cluster status with nodetool utility. Check if all nodes are up and running.
[cassandra@test bin]$ ./nodetool status Datacenter: datacenter1 ======================= Status=Up/Down|/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.56.104 263.42 KB 256 64.5% cddee7f4-c0d5-4cba-9ddc-b773a08a9245 rack1 UN 192.168.56.105 212.19 KB 256 67.5% 19cd17d2-1aeb-48e5-9299-7a0282c2e92e rack1 UN 192.168.56.106 147.54 KB 256 68.0% f65d5b92-ed3b-4f68-b93d-72e6162eafca rack1
Enjoy 😉