Deploying RabbitMQ Cluster On Kubernetes — Part 1

7 min readAug 18, 2021

RabbitMQ is one of the most widely used open-source message brokers. It is lightweight and easy to deploy on-premises and in the cloud. RabbitMQ was originally based on the Advanced Message Queuing Protocol (AMQP) and it’s architecture main components are Producer ,Exchange ,Queue and Consumer application.

For more RabbitMQ Architecture read my previous article RabbitMQ In Event-Driven Communication Between Microservices.

Prerequisites & Setup

Kubernetes : Basic knowledge on Kubernetes architecture and its components (Link) ,also you need to install Kubernetes cluster.

If your using OpenShift Container Platform you should have basic knowledge on OpenShift.

RedHat OpenShift Cluster: OpenShift Container Platform — platform as a service built around docker containers orchestrated and managed by Kubernetes on a foundation of Red Hat Enterprise Linux. In this article we are using OpenShift cluster provision on AWS. You can read my pervious article how to setup OpenShift cluster on AWS ( Provision Red Hat OpenShift Cluster On AWS) or you can use Red Hat OpenShift Developer Sandbox account (Link), It will provide one month free Openshift cluster for you.

Part 1 : Deploy a RabbitMQ cluster on Kubernetes platform and basic RabbitMQ configuration.

Part 2 : Implement simple publisher/subscriber application using MassTransit deploy on Kubernetes.

Deploy RabbitMQ Cluster

In here I will put every command that can run on both Kubernetes (> kubectl command) and openshift (> oc command ) .If your using OpenShift run the openshift command.

Step 01 Create a namespace or project

Let’s create a namespace or project either Kubernetes or Openshift. All resources in Kubernetes are started in a namespace. Unless one is specified, to have better control over the deployment process use the following command to create a distinct namespace.

kubectl create ns rabbits

oc create project rabbits

Step 02 Create Deployment

1.) Create “rabbit-configmap.yaml” using vi command.(Link)

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
data:
  enabled_plugins: |
    [rabbitmq_federation,rabbitmq_management,rabbitmq_peer_discovery_k8s].
  rabbitmq.conf: |
    loopback_users.guest = false
    listeners.tcp.default = 5672
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    
    cluster_formation.node_cleanup.only_log_warning = true

Execute the following command to create configMap.

kubectl create -n rabbits -f rabbit-configmap.yaml

oc create -n rabbits -f rabbit-configmap.yaml

2.) Create Role-based access control (RBAC)

Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within your organization.

Create “rabbit-rbac.yaml” using vi command (Link)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: rabbitmq
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
rules:
- apiGroups:
    - ""
  resources:
    - endpoints
  verbs:
    - get
    - list
    - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rabbitmq
subjects:
- kind: ServiceAccount
  name: rabbitmq
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  
  name: rabbitmq

Execute the following command to create RBAC.

kubectl create -n rabbits -f rabbit-rbac.yaml

oc create -n rabbits -f rabbit-rbac.yaml

3.) Create Secret for the Erlang Cooke.

RabbitMQ nodes and CLI tools use a cookie to determine whether they are allowed to communicate with each other. For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie.

Create “rabbit-secret.yaml” using vi command (Link)

apiVersion: v1
	kind: Secret
	metadata:
	  name: rabbit-secret
	type: Opaque
	data:
	  
      RABBITMQ_ERLANG_COOKIE: V0lXVkhDRFRDSVVBV0FOTE1RQVc=

Execute the following command to create Erlang cookie secret.

kubectl create -n rabbits -f rabbit-secret.yaml


oc create -n rabbits -f rabbit-secret.yaml

4.) Create StatefulSet for RabbitMQ

RabbitMQ requires using a Stateful Set to deploy a RabbitMQ cluster to Kubernetes. The Stateful Set ensures that the RabbitMQ nodes are deployed in order, one at a time. There are other, equally important reasons for using a Stateful Set instead of a Deployment: sticky identity, simple network identifiers, stable persistent storage and the ability to perform ordered rolling upgrades.

By using a StatefulSet as is intended for the case where Pods have persistent data that is associated with their identity. If the RabbitMQ cluster down ,pods restart or pods failure Kubernetes will create new pods ,but having persistent storage, rabbitmq queue messages will not delete.

Create “rabbit-statefulset.yaml” using vi command (Link)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
spec:
  serviceName: rabbitmq
  replicas: 3
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      initContainers:
      - name: config
        image: busybox
        command: ['/bin/sh', '-c', 'cp /tmp/config/rabbitmq.conf /config/rabbitmq.conf && ls -l /config/ && cp /tmp/config/enabled_plugins /etc/rabbitmq/enabled_plugins']
        volumeMounts:
        - name: config
          mountPath: /tmp/config/
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      containers:
      - name: rabbitmq
        image: rabbitmq:3.8-management
        ports:
        - containerPort: 15672
          name: discovery
        - containerPort: 5672
          name: amqp
        env:
        - name: RABBIT_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: RABBIT_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: RABBITMQ_NODENAME
          value: rabbit@$(RABBIT_POD_NAME).rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        - name: RABBITMQ_USE_LONGNAME
          value: "true"
        - name: RABBITMQ_CONFIG_FILE
          value: "/config/rabbitmq"
        - name: RABBITMQ_ERLANG_COOKIE
          valueFrom:
            secretKeyRef:
              name: rabbit-secret
              key: RABBITMQ_ERLANG_COOKIE
        - name: K8S_HOSTNAME_SUFFIX
          value: .rabbitmq.$(RABBIT_POD_NAMESPACE).svc.cluster.local
        volumeMounts:
        - name: data
          mountPath: /var/lib/rabbitmq
          readOnly: false
        - name: config-file
          mountPath: /config/
        - name: plugins-file
          mountPath: /etc/rabbitmq/
      volumes:
      - name: config-file
        emptyDir: {}
      - name: plugins-file
        emptyDir: {}
      - name: config
        configMap:
          name: rabbitmq-config
          defaultMode: 0755
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "gp2"
      resources:
        requests:
          storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
spec:
  clusterIP: None
  ports:
  - port: 15672
    targetPort: 15672
    name: discovery
  - port: 5672
    targetPort: 5672
    name: amqp
  selector:
    
    app: rabbitmq

Replace the storageClassName in the “rabbit-statefulset.yaml”.

kubectl get storageclass

oc get storageclass

Execute the following command to create StatefulSet and Service.

kubectl create -n rabbits -f rabbit-statefulset.yaml

oc create -n rabbits -f rabbit-statefulset.yaml

5.) Create Route for RabbitMQ service.

Kubernetes :

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: rabbitmq-route
spec:
  to:
    kind: Service
    name: rabbitmq
    weight: 100
  port:
    targetPort: discovery
  
  wildcardPolicy:None

Create rabbit-route.yaml file and execute the following command to create route.

kubectl create -n rabbits -f rabbit-route.yaml

OpenShift

Step 03 Login to the RabbitMQ Cluster.

Username : guest
Password :guest

Step 04 Create new admin user in RabbitMQ.

To create a new user, we need to login into RabbitMQ web management plugin using default credentials (guest) like previous steps. After logging into the application, create a new user , navigate to Admin tab.

After navigate to Admin tab, we can see the default user (“guest”) details who is having “administrator” privileges and below that we have Add a user panel to add new user.

After we click on “Add a user” panel, it will show the panel with different parameters to create a new user like as shown below.

If you observe above picture, we have a section called Tags. Here, Tags are the rights or privileges which we assigned while creating a user and we can set single or multiple privileges to a user based on our requirements. In case, if we want to multiple privileges, then set privileges as a comma (,) separated like administrator, management, etc. based on our requirements.

Following are the different type of privileges or rights which supported in management plugin while creating a new user in RabbitMQ.

After adding a user, you can see all the users in All users panel for that just expand it like as shown below.

Step 05 RabbitMQ Set Privileges to Access Virtual Hosts.

To set a permission to access virtual hosts, just click on the user which we have created “adminuser” and go to Permissions panel which is just below to overview panel and click on Set permission button to set permissions like as shown below.

Now we will use newly created user (“adminuser”) details to login into RabbitMQ web management. Following is the snapshot of RabbitMQ web management plugin after logging with new user “adminuser”.

Conclusion

In this article, You have successfully installed RabbitMQ on Kubernetes or Openshift. Also you can use RabbitMQ Cluster Operator to provision RabbitMQ cluster either Kubernetes or Openshift (Link). You have also gained access to the RabbitMQ management create new admin user and assign user privilege. In my next article I will implement simple .Net core publisher/subscriber application using MassTransit deploy on Kubernetes.

AWS Amazon MQ now supports RabbitMQ as a message broker, Amazon MQ is a fully managed service that provisions and manages open source message brokers like RabbitMQ and Apache ActiveMQ for you.(Link)

Enjoy!!! stay safe