Advanced Configuration
Docker login for private registry#
If you need to use a docker registry with authentication, then you will need to create a specific kubernetes secret with this information. Then you will configure the CRD with the secret name, so that it provides the data to each Statefulset, which in turn propagate it to each created Pod.
Create the secret :
Then we will add a imagePullSecrets parameter in the CRD definition with value the name of the previously created secret. You can give several secrets :
Management of allowed Cassandra nodes disruption#
CassKop makes use of the kubernetes PodDisruptionBudget objetc to specify how many cassandra nodes disruption is allowed on the cluster. By default, we only tolerate 1 disrupted pod at a time and will prevent to makes actions if there is aloready an ongling disruption on the cluster.
In some edge cases it can be useful to make force the operator to continue it's actions even if there is already a
disruption ongoing. We can tune this by updating the spec.maxPodUnavailable parameter of the cassandracluster CRD.
important
it is recommended to not touch this parameter unless you know what you are doing.
Cross Ip Management#
Global mecanism#
Cassandra works on IPs and not on hostname, so in a case where two cassandra cross their Ips, no Cassandra will be able to run properly and will loop on the following error :
Following the issue #170, at least using Kubernetes and Project Calico, we may fall into this issue, for example using a fixed ip pool size.
To manage this case we introduced the restartCountBeforePodDeletion CassandraCluster spec field which takes an int32 as value.
note
If you set it with a value lower or equals to 0, or if you omit it, no action will be performed
In setting this field, the cassandra operator will check for each CassandraCluster if a pod is in a restart situation (based on restart count of the cassandra container inside the pod).
In the case where the restartCount of the pod is greater than the value of restartCountBeforePodDeletion field and if we are in a Ip cross situation, we will delete the pod, which will be recreated by the Statefulset.
In the case of Project Calico usage, this force the pod to get another available IP, which fixes our bug.
Ip Cross situation detection#
To detect that we are in a Ip cross situation, we add a new status field CassandraNodeStatus which will maintain a cache about the map of Ip node and his hostId,
for all ready pods.
note
to have more information about this status field, you can check CassandraCluster Status
So when we check pods, we perform a Jolokia call to get a map of the cluster nodes IPs with their corresponding HostId.
If a pod is failing with the constraints described above, we compare the hostId associated to the Pod's IP, and the hostId
associated to the Pod name stored into the CassandraNodeStatus :
- if they match, or there are no match for the pod Ip into the map returned by Jolokia, we are not in a Ip cross situation,
- if they mismatch, we are in a Ip cross situation.