Features

To highligt some of the features we needed and were not possible with the operators available, please keep reading

Fine Grained Node Config Support

We needed to be able to react to events in a fine-grained way for each Node - and not in the limited way StatefulSet does (which, for example, removes the most recently created Nodes). Some of the available solutions try to overcome these deficits by placing scripts inside the container to generate configs at runtime (a good example is our Cassandra Operator), whereas the Orange NiFi operator's configurations are deterministically placed in specific Configmaps.

Graceful NiFi Cluster Scaling

Apache NiFi is a good candidate to create an operator, because everything is made to orchestrate it through REST Api calls. With this comes automation of actions such as scaling, following all required steps : https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#decommission-nodes.

Graceful Rolling Upgrade

Operator supports graceful rolling upgrade. It means the operator will check if the cluster is healthy.

Dynamic Configuration Support

NiFi operates with two type of configs:

• Read-only
• PerNode

Read only config requires node restart to update all the others may be updated dynamically. Operator CRD distinguishes these fields, and proceed with the right action. It can be a rolling upgrade, or a dynamic reconfiguration.

Dataflow lifecycle management via CRD

In a cloud native approach, we are looking for important management features, which we have applied to NiFi Dataflow :

• Automated deployment : Based on the NiFi registry, you can describe your NiFiDataflow resource that will be deployed and run on the targeted NiFi cluster.
• Portability : On kubernetes everything is a yaml file, so with NiFiKop we give you the ability to describe your clusters but also the registry clients, parameter contexts and dataflows of your NiFi application, so that you can redeploy the same thing in a different namespace or cluster.
• State management : With NiFiKop resources, you can describe what you want, and the operator deals with the NiFi Rest API to make sure the resource stays in sync (even if someone manually makes changes directly on NiFi cluster).
• Configurations : Based on the Parameter Contexts, NiFiKop allows you to associate to your Dataflow (= your applications) with a different configuration depending on the environment !

Users and access policies management

Without the management of users and access policies associated, it was not possible to have a fully automated NiFi cluster setup due to :

• Node scaling : when a new node joins the cluster it needs to have some roles like proxy user request, view data etc., by managing users and access policies we can easily create a user for this node with the right accesses.
• Operator admin rigth : For the operator to manage efficiently the cluster it needs a lot of rights as deploying process groups, empty the queues etc., these rights are not available by default when you set a user as InitialAdmin. Once again by giving the ability to define users and access policies we go through this.
• User's access : as seen just below we need to define the operator as InitialAdmin, in this situation there is no more users that can access to the web UI to manually give access to other users. That's why we extend the InitialAdmin concept into the operator, giving the ability to define a list of users as admins.

In addition to these requirements to have a fully automated and managed cluster, we introduced some useful features :

• User management : using NifiUser resource, you are able to create (or bind an existing) user in NiFi cluster and apply some access policies that will be managed and continuously synced by the operator.
• Group management : using NifiUserGroup resource, you can create groups in NiFi cluster and apply access policies and a list of NifiUser that will be managed and continuously synced by the operator.
• Default group : As the definition of NifiUser and NifiUserGroup resources could be heavy for some simple use cases, we also decided to define two default groups that you can feed with a list of users that will be created and managed by the operator (no kubernetes resources to create) :
  • Admins : a group giving access to everything on the NiFi Cluster,
  • Readers : a group giving access as viewer on the NiFi Cluster.

By introducing this feature we are giving you the ability to fully automate your deployment, from the NiFi Cluster to your managed NiFi Dataflow.