2 posts tagged with "cassandra"

In a previous post, I was talking about how Setting up Cassandra Multi-Site on Google Kubernetes Engine with Casskop. Since then, two new versions 0.5.1 and 0.5.2 had been released. In another post, Cyril Scetbon focused on the New Probes feature which was added with the [PR #184Ø(https://github.com/Orange-OpenSource/casskop/pull/184), in this post I will focus on the dynamic sidecars and storage configurations added to the operator, which give more flexibility to users to configure their Cassandra cluster deployments.

Purposes#

During our production migration from bare metal Cassandra Cluster to Kubernetes, the main challenge was to perform the smoothest transition for our OPS teams, allowing them to reuse their homemade tools, to facilitate the cluster operationalization. However, the operator in this previous form did not leave much room for tuning statefulset and therefore the Cassandra Cluster deployed. You could use the bootstrap image to customize your cassandra node configuration, but not for the tools revolving around. That is why we added to the CassandraCluster the possibility to define containers into the pod in addition to the cassandra ones, these are the sidecars, and to configure extract storage for the pods (ie VolumeClaimTemplates to the Statefulset configuration).

Dynamics sidecars configurations#

To keep the container’s best practices and address our OPS needs, we added the ability to define a dynamic list of containers into a CassandraCluster.Spec resource definition: cassandracluster_types.go#L803.

These sidecars are classic kubernetes container resources, leaving you the full power on what you want to do. With this example, we add two simple sidecars allowing to distinguish cassandra and GC logs in two different stdout.

Sidecars : environment variables#

All sidecars added with this configuration will have, at the container init, some of the environment variables from cassandra container merged with those defined into the sidecar container

• CASSANDRA_CLUSTER_NAME
• CASSANDRA_SEEDS
• CASSANDRA_DC
• CASSANDRA_RACK

Storage configuration#

In the previous version, the only option about storage was the data volume configuration allowing you to define :

• dataCapacity: Defines the size of the persistent volume claim, for example, "1000Gi".
• dataStorageClass: Defines the type of storage to use (or use default one). We recommend to use local-storage for better performances but it can be any storage with high ssd throughput.

The dynamic sidecar doesn’t really suit, unless you put everything in one folder.

That is why we add the CassandraCluster.Spec.StorageConfig field, to the CassandraCluster resource definition :

storageConfigs : Defines the list of storage config object, which will instantiate Persitence Volume Claim and associate volume to pod of cassandra node.

• mountPath: Defines the path into cassandra container where the volume will be mounted.
• name: Used to define the PVC and VolumeMount names.
• pvcSpec: pvcSpec describes the PVC used for the mountPath described above, it requires a kubernetes PVC spec.

In this example, we add the two volumes required by our sidecars previously configured, to be able via the sidecars to access to the logs that we want to expose on the stdout.

Volume Claim Template and statefulset#

Keep in mind that Casskop operator works on Statefulset, but have some constraints such as :

So if you want to add or remove some storages configurations, today you have to perform manually it, by removing the Statefulset, which will be recreated by the operator.

CassKop is open source so don’t hesitate to try it out, contribute by first trying to fix a discovered issue and let’s enhance it together!

In a next post, I will speak about the IP management into Casskop, and the cross IPs issue, so stay connected !

Pre-requisites#

User should need :

• terraform version v0.12.7+
• kubectl version v1.13.3+
• kubectx & kubens
• Helm version v2.15.1+
• gcloud sdk version 272.0.0+
• A service account with enough rights (for this example : editor)
• Having a DNS zone in google cloud dns.

Setup GCP environment#

To setup the GCP environment we will use terraform provisionning, to instantiate the following infrastructure :

• 2 GKE clusters :
  • First on europe-west1-b which will be the master
  • Second on europe-west1-c which will be the slave
• Firewall rules to allow clusters to communicate
• External DNS on each cluster to expose cassandra nodes
• Casskop operator on each cluster to focus on multi-casskop usage

Environment setup#

Start to set variables needed for the instantiation :

Setup base infrastructure#

Master provisionning#

With the master provisionning, we will deploy firewall and Cloud dns configuration :

Slave provisionning#

Check installation#

Check master configuration#

Now we will check that everything is well deployed in the GKE master cluster :

Check slave configuration#

Now we will check that everything is well deployed in the GKE slave cluster :

Check DNS zone configuration#

Make a note of the nameservers that were assigned to your new zone :

Check Firewall configuration#

@TODO : rework firewall source

Check Storage Class#

Multi casskop deployment#

Bootstrap API access to Slave from Master#

Multi-Casskop will be deployed in master cluster, change your kubectl context to point this cluster.

In order to allow Multi-CassKop controller to have access to slave from master, we are going to use kubemcsa from admiralty to be able to export secret from slave to master.

Install kubemcsa :

Generate secret for master :

Check that the secret is correctly created

Install Multi-CassKop#

@TODO : To correct once the watch object will be fixed

Add MultiCasskop crd on the slave cluster :

Deployment with Helm :

Create the MultiCasskop CRD#

Now we are ready to deploy a MultiCassKop CRD instance. We will use the example in multi-casskop/config/samples/gke/multi-casskop-gke.yaml :

Check multi cluster installation#

We can see that each cluster has the required pods :

If we go in one of the created pods, we can see that nodetool see pods of both clusters :

Clean up everything#

If you have set the deleteCassandraCluster to true, then when deleting the MultiCassKop object, it will cascade the deletion of the CassandraCluster object in the targeted k8s clusters. Then each local CassKop will delete their Cassandra clusters (else skip this step)

Cleaning slave cluster#

Cleaning master cluster#

Before running the following command, you need to clean dns records set.