Pods Operations

Here is the list of Operations managed by CassKop at the Pod operations level, which apply at pod level and can be triggered by specifics pods labels. Status of pod operations are also followed up at rack level.

Some Pods Operations can be triggered automatically by CassKop if :

CassandraCluster.spec.autoPilot is true, that will trigger cleanup, rebuild and upgadesstable operation in response to cluster events automatically.
the decommission operation is special and will be triggered automatically each time we need to ScaleDown a Pod.
the removenode operation is also special and may be set manually when needed.

It is also possible to trigger operations "manually", setting some labels on the Pods.

OperationCleanup#

A Cleanup may be automatically triggered by CassKop when it ends Scaling the cluster. CassKop will set some specific labels on the targeted pods. We can also set these labels manually, or using the privided plugin (kubectl casskop cleanup start) If we want to see labels for each of the pods of the cluster :

$ kubectl label pod $(kubectl get pods -l app=cassandracluster -o jsonpath='{range .items[*]}{.metadata.name}{" "}') --list
Listing labels for Pod./cassandra-demo-dc1-rack1-0:
 cluster=k8s.pic
 controller-revision-hash=cassandra-demo-dc1-rack1-56c9bbb958
 dc-rack=dc1-rack1
 statefulset.kubernetes.io/pod-name=cassandra-demo-dc1-rack1-0
 app=cassandracluster
 cassandracluster=cassandra-demo
 cassandraclusters.db.orange.com.dc=dc1
 cassandraclusters.db.orange.com.rack=rack1
...

Now, to trigger a cleanup on pod cassandra-demo-dc1-rack2-0

kubectl label pod cassandra-demo-dc1-rack2-0 operation-name=cleanup --overwrite
kubectl label pod cassandra-demo-dc1-rack2-0 operation-status=ToDo --overwrite

Automatically, CassKop will detect the change, start the action, and update the status :

    dc1-rack2:
      cassandraLastAction:
        Name: UpdateSeedList
        endTime: 2018-09-27T15:26:51Z
        startTime: 2018-09-27T15:23:54Z
        status: Done
      phase: Running
      podLastOperation:
        Name: cleanup
        endTime: 2018-09-27T16:00:52Z
        operatorName: operator-cassandr-f6d2968d4504448180ace041d3818d10-799dbb4zqss8
        podsOK:
        - cassandra-demo-dc1-rack2-0
        - cassandra-demo-dc1-rack2-0
        startTime: 2018-09-27T16:00:32Z
        status: Done

The section podLastOperation appears and we can see that it has correctly executed the cleanup operation on the 2 nodes

OperationRebuild#

This operation operates on multiple nodes in the cluster. Use this operation when CassKop add a new datacenter to an existing cluster.

kubectl casskop rebuild {--pod <pod_name> | --prefix <prefix_pod_name>} <from-dc_name>

In the background this command is equivalent to set labels on each pods like :

kubectl label pod cassandra-demo-dc2-rack1-0 operation-name=rebuild --overwrite
kubectl label pod cassandra-demo-dc2-rack1-0 operation-status=ToDo --overwrite
kubectl label pod cassandra-demo-dc2-rack1-0 operation-argument=dc1 --overwrite

OperationDecommission#

see UpdateScaleDown

RollingRestart#

This operation can be triggered with the plugin using simple commands as :

$ kubectl casskop restart --crd cassandra-e2e --rack dc1.rack1 dc2.rack1

Namespace cassandra-e2e
Trigger restart of dc1.rack1
Trigger restart of dc2.rack1

$ kubectl casskop restart --crd cassandra-e2e --dc dc1

Namespace cassandra-e2e
Trigger restart of dc1.rack1
Trigger restart of dc1.rack2

$ kubectl casskop restart --crd cassandra-e2e --full

Namespace cassandra-e2e
Trigger restart of dc1.rack1
Trigger restart of dc1.rack2
Trigger restart of dc2.rack1

After one of this command, CassKop will do a rolling restart of each rack one at a time avoiding any disruption.