11 software delivery problems solved by intelligence software delivery platform  Download
Select Page
Share

Background 

The self-signed certs (certificates) for mutual TLS, used as a means of security and authentication for communication between Spinnaker microservices, are usually short-lived (maybe for three months) and expire after the specified period of time. If the mTLS certs are approaching expiry, these certs need to be updated to extend their expiry time. The cert-manager is a native Kubernetes certificate management controller that helps with issuing certificates from a variety of sources, such as Let’s EncryptHashiCorp VaultVenafi, a simple signing key pair, or self-signed. It ensures certificates are valid and up to date, and attempts to renew certificates at a configured time before expiry on its own. To know more, about the certs, cert issuers – CA (certificate authorities), how to create them and use them as means of authentication, please refer to the following articles: 

However, when there is no external certificate management solution such as cert-manager to rely upon, you will need to update the certs yourself. While updating you can extend their validity to two years or more. But you must first update them in advance without incurring downtime. However, if your certs have already expired, then they are already in an invalid state, and avoiding downtime might not be possible. When the certificate fails the Spinnaker microservices, say Redis, Clouddriver, Gate services fail to talk to each other, and Spinnaker itself is not accessible more. So even the pre-scheduled jobs, like Cron jobs fail to run.

In the high paced world of Continuous Delivery, if Spinnaker microservices fail to communicate with each other due to expired certificates resulting in authentication failure, will result in a service outage for CI/CD pipelines and bring catastrophic losses to the business. Therefore it is best to proceed with replacing the expiring certificates with valid ones in advance. 

Here in this blog, we will show you how to update the expired certs used for mTLS authentication in Spinnaker. 

Here we will create two certificates:

  1. Mtlsca-new cert – this is the certificate of CA signed by the self-signed issuer
  2. Mtlspkcs12-new cert – this is the certificate for mTLS, signed by the CA issuer.

Steps to extend the validity of expired CA certificate of self-signed issuer:

  • Recreate the mTLS certificate issued by CA with a new secret 

Here we create “mtlsca-new” as the new CA (Certificate Authority) authorized certificate signed by a self-signed issuer with a new secret “cacert-new” using the YAML code, as shown below:

				
					kubectl create -f 
apiVersion: cert-manager.io/v1beta1
kind: Certificate
metadata:
  name: mtlsca-new
  namespace: spinnaker
spec:
  commonName: mtlsca
  dnsNames:
  - spin-clouddriver.spinnaker.svc
  - spin-echo.spinnaker.svc
  - spin-front50.spinnaker.svc
  - spin-gate.spinnaker.svc
  - spin-igor.spinnaker.svc
  - spin-orca.spinnaker.svc
  - spin-rosco.spinnaker.svc
  - spin-fiat.spinnaker.svc
  - '*.spinnaker.svc'
  - localhost
  duration: 21600h0m0s
  isCA: true
  issuerRef:
    kind: ClusterIssuer
    name: selfsigned-issuer
  secretName: cacert-new
				
			
  • Verify the status of the newly created certificate and the secret:

				
					kubectl -n spinnaker get certs
kubectl -n spinnaker get secrets
				
			
  • Create a new issuer “caissuer-new” and the related secret name “cacert-new” as used above while recreating the new mTLS certificate. Make sure the issuer names and secret are different from the original issuer and it’s secret using the below YAML code as shown below:

				
					kubectl create -f 
apiVersion: cert-manager.io/v1beta1
kind: Issuer
metadata:
  name: caissuer-new
  namespace: spinnaker
spec:
  ca:
    secretName: cacert-new
				
			
  • Verify the status of the new issuer by running the following command by checking:

				
					kubectl -n spinnaker get issuers
				
			
  • Create a new “mtlspkcs12” certificate which is a certificate for mTLS, signed by the new CA issuer using the below YAML code as shown below: 

Note:

  1. Make sure the name of the cert and the secret name are different from the old cert and secret.  
  2. Make sure you have the same name for the issuer as created above in the issuerRef block.
				
					kubectl create -f 
apiVersion: cert-manager.io/v1beta1
kind: Certificate
metadata:
  name: mtlscerts-pkcs12-new
  namespace: spinnaker
spec:
  commonName: spinnaker.svc
  dnsNames:
  - spin-clouddriver.spinnaker.svc
  - spin-echo.spinnaker.svc
  - spin-front50.spinnaker.svc
  - spin-gate.spinnaker.svc
  - spin-igor.spinnaker.svc
  - spin-orca.spinnnaker.svc
  - spin-rosco.spinnaker.svc
  - spin-fiat.spinnaker.svc
  - '*.spinnaker.svc'
  - localhost
  duration: 21600h0m0s
  issuerRef:
    kind: Issuer
    name: caissuer-new
  keystores:
    pkcs12:
      create: true
      passwordSecretRef:
        key: passphrase
        name: passphrasesecret
  renewBefore: 3600h0m0s
  secretName: mtlscerts-pkcs12-new
  usages:
  - digital signature
  - key encipherment
  - server auth
  - client auth
				
			
  • Update the cacerts using the existing secret, and delete old certs using their aliases:

				
					kubectl -n spinnaker get secret cacerts -o jsonpath='{.data.cacerts}'| base64 -d > cacerts
keytool -delete -alias ca -keystore cacerts
keytool -delete -alias caprod -keystore cacerts
keytool -list -keystore cacerts -v -storepass changeit -noprompt > myfile
				
			
  • Get the ca.crt from the new secret:

				
					kubectl -n spinnaker get secret mtlscerts-pkcs12-new -o jsonpath='{.data.ca\.crt}'| base64 -d > ca.crt
				
			
  • Verify expiration date again:

				
					openssl x509 -in ca.crt -text -noout
				
			
  • Import ca.crt to cacerts:

				
					keytool -importcert -keystore cacerts -alias caprod -file ca.crt -storepass changeit -noprompt
				
			
  • Verify if the new alias exists:

				
					keytool -list -keystore cacerts -v -storepass changeit -noprompt > myfile
vi myfile
				
			

Check the alias caprod and the expiry date.

  • Create a new secret for cacerts:

				
					kubectl -n spinnaker create secret generic cacerts-new --from-file cacerts
				
			
  • Copy tls.crt and tls.key to halyard:

				
					kubectl -n spinnaker cp tls.crt spinnaker-spinnaker-halyard-0:/home/spinnaker/.hal/default/tls-new.crt
kubectl -n spinnaker cp tls.key spinnaker-spinnaker-halyard-0:/home/spinnaker/.hal/default/tls-new.key
				
			
  • Change in redis sts (statefulset ):

				
					Replace mtlscerts-pkcs12 with mtlscerts-pkcs12-new
				
			
  • In Hal config, replace old TLS key and cert names with new TLS key and cert names 
				
					tls.crt and tls.key to tls-new.crt and tls-new.key
				
			
  • In the cd, service-setting replace cacerts with cacerts-new and secret mtlscerts-pkcs12 with mtlscerts-pkcs12-new in all service setting files.

				
					find . -name '*.yml'| xargs grep cacerts
find . -name '*.yml'| xargs sed  's/cacerts/cacerts-new/g'
find . -name '*.yml'| xargs sed  -i 's/cacerts/cacerts-new/g'
find . -name '*.yml'| xargs grep cacerts-new
find . -name '*.yml'| xargs grep mtlscerts-pkcs12
find . -name '*.yml'| xargs sed  's/mtlscerts-pkcs12/mtlscerts-pkcs12-new/g'
find . -name '*.yml'| xargs sed -i  's/mtlscerts-pkcs12/mtlscerts-pkcs12-new/g'
find . -name '*.yml'| xargs grep mtlscerts-pkcs12-new
hal deploy apply
				
			
  • Verify if the spinnaker is working fine after the certs update: 

  1. Check if Spinnaker is accessible via the browser
  2. Check logs of all the spinnaker services
  • If the above verification is okay and Spinnaker is working fine

  1. Run one sample pipeline in the spinnaker
  2. Delete tls.crt and tls.key from default (old cert and key) in halyard pod
  3. Delete old secret, certs, and issuers
				
					Secret : mtlscerts-pkcs12 , cacert and cacerts
Certs: mtlscerts-pkcs12 and mtlsca
Issuer: issuer
				
			

Finally, check once again to see that everything is working fine once all the changes have been implemented.

Conclusion:

After applying the above configuration changes to your Spinnaker deployment, the Mutual TLS (mTLS) Authentication certs for Spinnaker Services are updated thereby making it secure to communicate securely over the network with other services.


If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us at OpsMx Support.

Manoj Ramurthy

Manoj Ramurthy has 5+ years of experience in the software field in Unix, Aws cloud, and DevOps areas. Worked at Infosys, HCL, and Techjini solutions. Expertise in Unix, Aws, docker, Kubernetes, Terraform, Prometheus, Grafana, and other tools.

0 Comments

Submit a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You May Like

Why OpsMx for Spinnaker?

March 7, 2019
Share

Protected: Documentation

October 27, 2016
Share