Select Page

Sagayaraj D

last updated on April 21, 2023
Spinnaker users can store the Kubernetes accounts in the halconfig file or externalize in a Git or AWS S3 store or secure the kubeconfig files in Hashicorp Vault. In this blog, we will focus on storing the kubeconfig files in Vault and we will describe the steps to externalize kubeconfig files of Kubernetes accounts under the following environment:
  • Spinnaker 1.17.6 (However the same can work on new versions of Spinnaker)
  • Hashicorp Vault as external kubeconfig files store
  • Halyard service’s deployment profile is the default (i.e $HOME/.hal/default directory)
Download Advanced deployment strategies
How do you store the Spinnaker kubeconfig file for Kubernetes in Vault?

Spinnaker loads Kubernetes provider accounts using the clouddriver service. In most cases, all of the Spinnaker services’ accounts are stored in halconfig file. If we configure Spinnaker’s inclusive ‘Spring Cloud Config server’ with Git external store, it can read the accounts from a Git repository. In any case, the kubeconfig file path is inferred as an absolute path pertaining to Clouddriver Pod.
Either way, we want to store the kubeconfig files in a Vault store. The kubeconfig files of the accounts are stored under a dedicated path in a backend engine. These kubeconfig files are pulled to Clouddriver Pod using a periodic/Cron script. This pull is achieved using a Sidecar container within Clouddriver Pod. 

The kubeconfig file path on the Cloulddriver Pod should be the same as in the configuration of the Clouddriver accounts.You can change the Vault backend, secret path, and other values as per your need.

Procedure Outline to store the Spinnaker kubeconfig file for Kubernetes in Vault
  1. Vault administrator creates a Token for Spinnaker
  2. Store the Kubernetes Accounts’ kubeconfig files in Vault
  3. Update Clouddriver Pod with Sidecar container to pull Kubeconfig files
  4. Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly

Detailed Procedure

1. Vault administrator creates a Token for Spinnaker

The Vault administrator creates a token that has privileges to read, write, update secrets in a given backend – spinnaker.
Steps: Open Vault URL > Click ‘Enable a Secrets Engine’ > Select ‘Generic’ engine (kv type) > Type Path as ‘spinnaker’. We refer this ‘spinnaker’ path as secret-engine/backend.
If Vault is an enterprise edition, you will also create a namespace – let’s call it ‘vaultns-spinnaker’.

2. Store the Kubernetes Accounts’ kubeconfig files in Vault

Kubernetes accounts’ kubeconfig files are stored in Vault as an independent entity. Hence, storing and retrieving the files to the Clouddriver Pod should be implemented by the user.
We use the following specification to store the kubeconfig files. – Vault backend: spinnaker – Path: k8configs/<account> – Key/Field: kubeconfig – Value: Content of Kubeconfig file
Every kubeconfig file is store in Vault with the command
					vault kv put spinnaker/k8configs/my-account kubeconfig=@my-account.yml

3. Update Clouddriver Pod with Sidecar container to pull Kubeconfig files 

Once Kubeconfig files are stored in encrypted Vault, we need to make sure those files are downloaded to Clouddriver Pod so that Clouddriver can infer the files locally. To achieve this, we will leverage the Sidecar container feature of the Pod object. By sharing a directory volume between the main and sidecar containers, we will enable the kubeconfig files available to the main container. Those kubeconfig files are downloaded by running a Cron script on the sidecar container.
The Cron script is created as a ConfigMap and then it is mounted as volume (file) to the sidecar container. Once the script is available on the container, it is executed periodically to download the new kubeconfig files from the Vault path.
The script is focused on doing the following …
  • Checks for any new account vault kv list spinnaker/k8configs
  • Downloads the new account’s kubeconfig file vault kv get -field=kubeconfig spinnaker/k8configs/my-account > my-account.yml
The following script goes as a ConfigMap
#This script is executed in Clouddriver or its Sidecard container to download the kubeconfig files
#Make sure vault cli is loaded on the Container and is accessible by the scripts
#Ensure the VAULT_ADDR and VAULT_TOKEN is set in the SHELL environment
#Set these variables to your choice
export VK8S_PATH='spinnaker/k8configs' #Prefix V for Vault
export LK8S_PATH='/tmp/k8configs' #Prefix L for Local
export R_FILE=/tmp/remote.txt
export L_FILE=/tmp/local.txt
export NEWKEYS=/tmp/newkeys.txt
#Make sure vault CLI is available
command -v vault > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Vault CLI is not available, exiting..."
#Create local config path if not exists
[ ! -d $LK8S_PATH ] && (mkdir -p $LK8S_PATH; echo Created k8sconfig directory) #|| echo Content mkdir -p $LK8S_PATH
vault kv list $VK8S_PATH | egrep -iv 'Keys|----' > $R_FILE
ls -1 $LK8S_PATH | sed s/\.yml$// > $L_FILE
#New in Vault
comm -23 $R_FILE $L_FILE > $NEWKEYS
if [ ! -s $NEWKEYS ]; then
  echo "There are no new kubeconfigs to download"
#Download new kubeconfig file from Vault
for i in `cat $NEWKEYS`; do
  echo -en "Fetching kubeconfig file of Account : $i"
  vault kv get -field=kubeconfig $VK8S_PATH/$i > $LK8S_PATH/$i.yml
  if [ $? -eq 0 ]; then
    echo " [success]"
    echo "  File: $LK8S_PATH/$i.yml"
    echo " [failed with return code $?]"
echo "Completed Kubeconfig downlod syncing"

The Vault address and its token are stored as secret

					apiVersion: v1
kind: Secret
type: Opaque
  name: vaultsec
  namespace: spinnaker
  #Replace the vaule here http://vault-host as vaultaddress, and token as encrypted using base64
  vaultaddr: abcovLzE3Mi40Mi40Mi4xMTE6ODIwMA==
  vaulttoken: abca054QXhQbW01T3hDUzVNaU1RbWl1Qmc=

Configuring the Sidecar container with the script:

					kind: Deployment
      - name: clouddriver
        - mountPath: /tmp/k8configs
          name: vvault
      - name: vault-c
        image: alpine:latest
        command: ["/bin/sh", " -c"]
        - |
          while true; do sh /tmp/; sleep 30m; done
        - name: VAULT_ADDR
              key: vaultaddr
              name: vaultsec
        - name: VAULT_TOKEN
              key: vaulttoken
              name: vaultsec
        - mountPath: /tmp/k8configs
          name: vvault
        - mountPath: /tmp/
          name: vcm-vault
      - emptyDir: {}
        name: vvault
      - name: vcm-vault
          defaultMode: 420
          name: vault-k8s
 If you carefully observe the above sidecar container, we are configuring two things:
  1. A shared directory volume of type ‘EmptyDir’ is created and mounted to both containers under the path ‘/tmp/k8configs’. The ‘EmptyDir’ type creates an empty directory on the Host machine before the containers are started, and then shared between the colocated containers as long as the volume is referenced
  2. The ConfigMap is mounted to the sidecar container as a script in the path /tmp/, which gets executed every 30 minutes to download any new kubeconfig files.

4. Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly

Once the changes are applied to Clouddriver,
  • Go to Clouddriver’s sidecar container and check if the kubeconfig files are downloaded.
  • Also, go to Clouddriver’s main container and verify if the kubeconfig file is available in the same path as in the Account’s kubeconfig file reference
To Verify the accounts,
  • We can login to Spinnaker and check the URL http:///credentials to see available accounts. 
Then, you can go to any Kubernetes pipeline and add a ‘Deploy (Manifest)’ stage to perform a dummy deployment to the target Kubetnets account/cluster. If the deployment goes through successfully, it is clear that the Kubeconfig files are downloaded to Clouddriver Pod successfully and are getting connected to the target cluster from the Pod.

About OpsMx

Founded with the vision of “delivering software without human intervention,” OpsMx enables customers to transform and automate their software delivery processes. OpsMx builds on open-source Spinnaker and Argo with services and software that helps DevOps teams SHIP BETTER SOFTWARE FASTER.


Submit a Comment

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.