Share

Why do you need to integrate Spinnaker with Redis?

Redis is a great choice for implementing a highly available in-memory cache for storing pipeline information with sub-millisecond response times.  All the information stored in Redis can be utilized for auditing, e.g. which pipeline has taken how long to complete. 

At OpsMx, we help customers to collect pipeline execution details and showcase them to the product managers and compliance managers (or auditors) using audit dashboards. Managers can then use this data to investigate a case or to make a decision if to improve a pipeline or make some configuration changes in the software delivery pipelines. 

Sample Redis server works for any application:

Sample Redis server

If Redis is local or configured in on-prem, what is the problem?

Scaling the pipeline is difficult and manual intensive:

When Kubernetes persistent volume claim (PVC)  gets overloaded with data, the pvcPVC has to be increased manually. But, in order to do so, the Redis must be   scaled down   before increasing the PVC, and then again it must be scaled up. This, however, requires some downtime from Redis. Thus, all the other pipeline executions will be halted and healthcheckthe health checks of Kubernetes pods will fail, thus degrading the overall experience of Spinnaker.

Below is the rough edit of a PVC to get your Redis goes down.

				
					kubectl edit pvc -n  
				
			
rough edit of a PVC to get your Redis

If the cluster where Redis is installed goes down, we will lose the data

If the Redis does not follow a highly available architecture, there is a  risk of losing all our data whenever the cluster goes down, even accidentally.

Solution: How to integrate Spinnaker with Redis service in AWS

You can achieve the integration of Redis with Spinnaker in two steps:

    1. Create Elastic Caching Redis cluster in AWS
    2. Integrate Spinnaker with newly created Redis Service in AWS

Prerequisites of Integrating Redis with Spinnaker 

    • Ensure we are creating a new Elastic Caching Redis Cluster on AWS cloud with the same vpc and subnet of EKS cluster.

Steps to Create Elastic Caching Redis Cluster in AWS 

Step-1: Go to console.aws.com and search for ElastiCache

search for ElastiCache

Step-2: Click on Create

ElastiCache Dashboard

Step-3: Use default options as it is and fill the details in Redis-settings

Step-4: Make use while are selecting the vpc and security group which are linked to our EKS cluster

Redis settings

While creating you can see the Redis-cluster in the creation process starting in the AWS console. After successful creation you will see an image like below:

Redis-cluster creation process
Redis-cluster in the creation process starting in the AWS console

Step-5: After the instance is created, copy the Primary Endpoint for the cluster.

Redis-cluster instance created
Redis-cluster parameter groups

Step-6: Go to Parameter Groups of AWS console and create a new parameter group. You can select the Redis version you have created in the previous step and provide a new name and description. The you also need to tag your parameter group with  key and values as mentioned below: 

				
					notify-keyspace-events  gxE


				
			
create parameter group
search parameter group aws

Steps to Integrate Spinnaker with Redis Service in AWS

Step-1: Check that redis-endpoint (AWS) reachable or not by following commands:

We will execute the below command to check if our Spinnaker is ready to connect to the external Redis endpoint.

				
					redis-cli -h spinredis.rjjqb0.ng.0001.apse1.cache.amazonaws.com -p 6379 
ping
PONG
				
			

Step-2: Now configure Exec into the halyard pod and perform configurational changes you can execute the command:

				
					Kubectl exec -it <halyard pod name> bash -n <namespace>
				
			

If you want to update all Spinnaker services at once, place this snippet into ~/.hal/default/service-settings/redis.yml, and replace $REDIS_PRIMARY_ENDPOINT with your endpoint.

				
					overrideBaseUrl: redis://$REDIS_PRIMARY_ENDPOINT
skipLifeCycleManagement: true
enabled: false
				
			

Step-3: After updating the base URLs, place the below mentioned into ~/.hal/default/profiles/gate-local.yml.

				
					redis:
     configuration:
       secure: true
				
			

Step-4: To get all the changes to take effect into your Spinnaker, run the below command.

				
					hal deploy apply
				
			

Wait for some time to get all the pods up/running

				
					kubectl scale deploy spin-redis -n spinnaker --replicas=0
				
			

Conclusion

To summarize, the integration of Redis with Spinnaker helps you to horizontally scale your external Redis on-demand, therefore eliminating the need for causing any downtime of Redis affecting Spinnaker services. Most importantly, with this integration, you can ensure the safety of the data without the loss of metadata which ultimately is useful for audit purposes.

If you find this article helpful or have any questions, please feel free to reach out to me at [email protected]. I would love to hear from you.

References:

 
Tags : Spinnaker

0 Comments

Submit a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.