Upgrade performance with high performant disk type

In addition to the default standard performance disk type, there is an option to turn on the high performance storageClass on certain Kubernetes storage providers.

The option is only valid on the first installation or by following the steps highlighted bellow in the section Upgrade performance with high performant disk type

If you wish to turn on high performant disk type on the first installation or a re-creation of the RabbitMQ cluster, please merge the following block into the custom-values-infra.yaml (See CustomValueFile) outside the global block.

Do not forget to set the storageClass to the correct value for your Kubernetes target deployment (the example below is for Azure platform but this is also the default value for that platform).

global:
  nexeedDeploymentTargets:
    azure:
      storageClasses:
        dedicatedHighPerformance: managed-premium
rabbitmq:
  statefulSets:
    rabbitmq:
      volume:
        performance: high

Instructions for changing the storage class without disruptions

BCI developed a specific script, called ChangeRabbitMQStorageClass.sh, capable of assisting with the change in the storage class.

Requests the script and readme file from your BCI contact.

Preparation activities

Check that the new storage class is available for the environment to be updated
Ensure RabbitMQ is in a healthy state from the K8s point of view, e.g. all pods are up and running and there are no issues.
Check the status of the cluster with rabbitmqctl cluster status
Check the status of RabbitMQ via the Management UI and ensure that there are no corrupted queues, e.g. queues that are "running" with "?" in their stats

Add overwrite for the new storage class (as instructed at the beginning of this document) to the custom-values-infra.yaml file. This is required for the new stateful set to be created with the new storage class.

Execution of script

Copy the script ChangeRabbitMQStorageClass.sh in a location where you can execute it from the command line.
Get the followings parameters before executing the script
- <kubeconfig>: KubeConfig File Location
- <namespace>: Namespace of where RabbitMQ is running
- <statefulset>: Name of existing RabbitMQ StatefulSet
Please ensure you have a stable terminal and network connection. Otherwise, the script will not work properly and some manual steps might be required.
Run the script from where you have access to the K8s API for the cluster, e.g. Laptop or from a jump host with access to the K8s API. If running it from a jump host, use tools like tmux or screen to keep the session alive in case of network issues.
Execute the following command from where script can be accessed:

ChangeRabbitMQStorageClass.sh <kubeconfig> <namespace> <statefulset>

Confirm that the script can delete the old stateful set. Just enter Y in the command prompt
Let the script run and do not close the terminal where it is running. It is waiting for the new stateful set to be created to continue with the next steps

Deployment of new RabbitMQ changes via Helm

Run helm upgrade command to deploy the new changes to the RabbitMQ StatefulSet. This will create a new stateful set with the new storage class.

Finalization steps

Let script finish. If successful, the script will say it finalized the migration successfully.
Check that all new pods are using the new storage class
Check that all pods are healthy and running
Check in the management UI that all queues are healthy, e.g. no corruption, and all have "+2" mirrors, except for the exclusive queues (excl)
If not synchronized, e.g. some of them show +1, please restart the RabbitMQ node which is missing most of the mirrors

If the script fails

If for any reasons the script fails, you’ll have to bring the cluster back in a healthy state. After fixing the cluster, you can simply re-run the procedure - the script will only alter the pods which doesn’t use persistent volumes with the right storage class.