This blog post is about the configuration of Apache Cassandra for running a cluster of 3 instances on a single host. Basically C* is not really build to run in a multi instances environment, but for many cases and reasons you might run a C* cluster on a single server. The principal reason is for tests cases.

Context

The objective is to have a Cassandra ring of 3 instances on 1 host. For that, we will use:

  • apache-cassandra-3.0.9 (community version)
  • Separated disks for a segregation between binaries, data and logs: binaries, data and logs will be in separated mount points
  • Multiple IPs interfaces for each instance (virtual interfaces in our case)
  • Instances name: csd1, csd2, csd3
  • Cluster name: dbitest

Binary installation

Download the binaries from the Apache website, http://www.apache.org/dyn/closer.lua/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz and install it. In this post we are going to install the 3.0.9 release.

wget http://www.pirbot.com/mirrors/apache/cassandra/3.0.9/apache-cassandra-3.0.9-bin.tar.gz

Extract it:

tar -zxvf apache-cassandra-3.0.9-bin.tar.gz

Place the software into the product directory:

cp –r apache-cassandra-3.0.9 /u00/app/cassandra/product/

 

Instances configuration

First, we will create the admin directory for each instance. The admin directory contains the following sub folders:

  • /u00/app/cassandra/admin/
    • /backup: soft link to the backup mount point
    • /dump: soft link to the dump mount point
    • /etc: Cassandra configuration files, such as Cassandra.yaml, logback.xml and Cassandra-env.sh.
    • /log: soft link to the log file from the log mount point
    • /pid: contain the pid of the C* process

To create these directories, use the following loop.

for i in 1 2 3 ;
do mkdir -p /u00/app/cassandra/admin/csd$i &&
mkdir /u00/app/cassandra/admin/csd$i/backup &&
mkdir /u00/app/cassandra/admin/csd$i/dump &&
mkdir /u00/app/cassandra/admin/csd$i/etc &&
mkdir /u00/app/cassandra/admin/csd$i/log &&
mkdir /u00/app/cassandra/admin/csd$i/pid &&
mkdir /u00/app/cassandra/admin/csd$i/product;
done

Next copy the C* configuration files into each admin/csd$i/etc directory. Every instance has its own configuration files.

for i in 1 2 3;
do cp -r /u00/app/cassandra/admin/csd$i/product/apache-cassandra-3.0.9/conf/* /u00/app/cassandra/admin/csd$i/etc/;
done

Now, create the three data directories for each instance.

for i in 1 2 3;
do
sudo mkdir -p /u01/cassandradata/csd$i &&
sudo chown -R cassandra:cassandra /u01/cassandradata/csd$i;
done

Configure virtual network interfaces

You must create additional virtual network interfaces for each C* instance, to use a different listen_address and rpc_address for each instance.

sudo ifconfig enp0s3:0 192.168.56.105
sudo ifconfig enp0s3:1 192.168.56.106

Then, you have 3 interfaces (default + 2 virtuals), for your C* instances.

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.104  …

enp0s3:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.105  …

enp0s3:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.106  …

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        …

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
…

As all the required files are in place, we can now change the 3 main configuration files, to set up the three instances (csd1, csd2 and csd3).

cassandra-env.sh:

Use a different “JMX_PORT” for each node. For instance: 7199 for csd1, 7299 for csd2 and 7399 for csd3.

JMX will bind to the local host IP by default, you can use the default.

cassandra.yaml

The central configuration file of Apache Cassandra. Change the following parameters:

  • cluster_name: ‘dbitest’ All instances must have the same cluster name
  • commitlog_directory: /u01/cassandradata/csd$i/commitlog
  • data_file_directories:/u01/cassandradata/csd$i/data
  • saved_caches_directory: /u01/cassandradata/csd$i/saved_caches
  • listen_address: 192.168.56.104
  • rpc_address: 192.168.56.104
  • seeds: specify the list of IP of the seed nodes

logback.xml

The configuration file for C* logs. Make the following changes:

<file>${cassandra.logdir}/system.log</file> to <file>/u00/app/cassandra/admin/csd$i/log/system.log</file>

<fileNamePattern>${cassandra.logdir}/system.log.%i.zip</fileNamePattern> to 
<fileNamePattern>/u00/app/cassandra/admin/csd$i/log/system.log.%i.zip</fileNamePattern>

<file>${cassandra.logdir}/debug.log</file> to <file>/u00/app/cassandra/admin/csd$i/debug.log</file>

<fileNamePattern>${cassandra.logdir}/debug.log.%i.zip</fileNamePattern> to
<fileNamePattern>/u00/app/cassandra/admin/csd1/log/debug.log.%i.zip</fileNamePattern>

Make the change for the three instances.

Starting the nodes

Before starting each node, you must dynamically set the environment variables for each of them.

$CASSANDRA_HOME=/u00/app/cassandra/product/apache-cassandra-3.0.9
$CASSANDRA_CONF=/u00/app/cassandra/admin/csd$i/etc/

After, setting the variables for each node you can start Apache Cassandra by command-line:

./$CASSANDRA_HOME/bin/cassandra –f

Execute the same command in a separated shell, for each node. Do not forget to set the environment variables $CASSANDRA_HOME and $CASSANDRA_CONF.

Cluster status

Verify the cluster status with nodetool utility. Check if all nodes are up and running.

[cassandra@test bin]$ ./nodetool status
 Datacenter: datacenter1
 =======================
 Status=Up/Down|/ State=Normal/Leaving/Joining/Moving
 --  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
 UN  192.168.56.104  263.42 KB  256          64.5%             cddee7f4-c0d5-4cba-9ddc-b773a08a9245  rack1
 UN  192.168.56.105  212.19 KB  256          67.5%             19cd17d2-1aeb-48e5-9299-7a0282c2e92e  rack1
 UN  192.168.56.106  147.54 KB  256          68.0%             f65d5b92-ed3b-4f68-b93d-72e6162eafca  rack1

 

Enjoy 😉

Post Views: 512
Thumbnail [60x60]
by
DevOps