Spinnaker – Externalising Kubeconfig files of Kubernetes Accounts


One can store the Kubernetes accounts in halconfig or externalize in Git or AWS S3 store, but protect the kubeconfig files in Hashicorp Vault.

In this blog we will focus on storing the kubeconfig files in Vault.


This article describes the steps under the following environment,
– Spinnaker 1.17.6 (However the same can work on new versions of Spinnaker)
– Hashicorp Vault as external kubeconfig files store
– Halyard service’s deployment profile is default (i.e $HOME/.hal/default directory)

How does it work?

Spinnaker loads Kubernetes provider accounts using the clouddriver service. In most cases all of the Spinnaker services’ accounts are stored in halconfig file. If we configure Spinnaker’s inclusive ‘Spring Cloud Config server’ with Git external store, it can read the accounts from a Git repository. In any case, the kubeconfig file path is inferred as absolute path pertaining to Clouddriver Pod.

Either way, we want to shield the kubeconfig files in a Vault store. The kubeconfig files of the accounts are stored under dedicated path in a backend engine. These kubeconfig files are pulled to Clouddriver Pod using a periodic/Cron script. This pull is achieved using a Sidecar container within Clouddriver Pod. The kubeconfig file path on the Cloulddriver Pod should be the same as in Clouddriver accounts configuration.

You can change the Vault backend, secret path and other values as per your need.

Procedure Outline

  1. Vault administrator creates a Token for Spinnaker
  2. Store the Kubernetes Accounts’ kubeconfig files in Vault
  3. Update Clouddriver Pod with Sidecar container to pull Kubeconfig files
  4. Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly

Detailed Procedure

1. Vault administrator creates a Token for Spinnaker

The Vault administator creates a token that has previleges to read, write, update secrets in a given backend – spinnaker.

Steps: Open Vault URL > Click ‘Enable a Secrets Engine’ > Select ‘Generic’ engine (kv type) > Type Path as ‘spinnaker’. We refer this ‘spinnaker’ path as secret-engine/backend.

If Vault is enterprise edition, you will also create a namespace – lets call it as ‘vaultns-spinnaker’.

2. Store the Kubernetes Accounts’ kubeconfig files in Vault

Kubernetes accounts’ kubeconfig files are stored in Vault as independent entity. Hence, storing and retieving the files to the Clouddriver Pod should be implemented by the user.

We use the following specification to store the kubeconfig files.
– Vault backend: spinnaker
– Path: k8configs/<account>
– Key/Field: kubeconfig
– Value: Content of Kubeconfig file

Every kubeconfig file is store in Vault with the command vault kv put spinnaker/k8configs/my-account kubeconfig=@my-account.yml

3. Update Clouddriver Pod with Sidecar container to pull Kubeconfig files

Once Kubeconfig files are stored in encrypted Vault, we need to make sure those files are downloaded to Clouddriver Pod, so that Clouddriver can infer the files locally. To achieve this, we will leverage the Sidecar container feature of POD object. By sharing a directory volume between the main and sidecar containers, we will enable the kubeconfig files available to main container. Those kubeconfig files are downloaded by running a Cron script on the sidecar container.

The cron script is created as a ConfigMap and then it is mounted as volume (file) to the sidecar container. Once the script is available on the container, it is executed periodically to download the new kubeconfig files from the Vault path.

The script is focused on doing the following …​

  • Checks for any new account
    vault kv list spinnaker/k8configs
  • Downloads the new account’s kubeconfig file
    vault kv get -field=kubeconfig spinnaker/k8configs/my-account > my-account.yml

Following script goes as a ConfigMap

#This script is executed in Clouddriver or its Sidecard container to download the kubeconfig files

#Make sure vault cli is loaded on the Container and is accessible by the scripts
#Ensure the VAULT_ADDR and VAULT_TOKEN is set in the SHELL environment

#Set these variables to your choice
export VK8S_PATH='spinnaker/k8configs' #Prefix V for Vault
export LK8S_PATH='/tmp/k8configs' #Prefix L for Local
export R_FILE=/tmp/remote.txt
export L_FILE=/tmp/local.txt
export NEWKEYS=/tmp/newkeys.txt

#Make sure vault CLI is available
command -v vault > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Vault CLI is not available, exiting..."

#Create local config path if not exists
[ ! -d $LK8S_PATH ] && (mkdir -p $LK8S_PATH; echo Created k8sconfig directory) #|| echo Content mkdir -p $LK8S_PATH

vault kv list $VK8S_PATH | egrep -iv 'Keys|----' > $R_FILE

ls -1 $LK8S_PATH | sed s/\.yml$// > $L_FILE

#New in Vault
comm -23 $R_FILE $L_FILE > $NEWKEYS
if [ ! -s $NEWKEYS ]; then
  echo "There are no new kubeconfigs to download"

#Download new kubeconfig file from Vault
for i in `cat $NEWKEYS`; do
  echo -en "Fetching kubeconfig file of Account : $i"
  vault kv get -field=kubeconfig $VK8S_PATH/$i > $LK8S_PATH/$i.yml
  if [ $? -eq 0 ]; then
    echo " [success]"
    echo "  File: $LK8S_PATH/$i.yml"
    echo " [failed with return code $?]"

echo "Completed Kubeconfig downlod syncing"

The Vault address and its token are stored as secret

apiVersion: v1
kind: Secret
type: Opaque
  name: vaultsec
  namespace: spinnaker
  #Replace the vaule here http://vault-host as vaultaddress, and token as encrypted using base64
  vaultaddr: abcovLzE3Mi40Mi40Mi4xMTE6ODIwMA==
  vaulttoken: abca054QXhQbW01T3hDUzVNaU1RbWl1Qmc=

Configuring the Sidecar container with the script

kind: Deployment
      - name: clouddriver
        - mountPath: /tmp/k8configs
          name: vvault
      - name: vault-c
        image: alpine:latest
        command: ["/bin/sh", " -c"]
        - |
          while true; do sh /tmp/k8configs-sync.sh; sleep 30m; done
        - name: VAULT_ADDR
              key: vaultaddr
              name: vaultsec
        - name: VAULT_TOKEN
              key: vaulttoken
              name: vaultsec
        - mountPath: /tmp/k8configs
          name: vvault
        - mountPath: /tmp/k8configs-sync.sh
          name: vcm-vault
          subPath: k8configs-sync.sh
      - emptyDir: {}
        name: vvault
      - name: vcm-vault
          defaultMode: 420
          name: vault-k8s

If you carefully observe the above sidecar container, we are configuring two things
1. A shared directory volume of type ‘EmptyDir’ is created and mounted to both containers under the path ‘/tmp/k8configs’. The ‘EmptyDir’ type creates a empty directory on the Host machine before the containers are started, and then shared between the colocated containers as long as the volume is referenced
2. The ConfigMap is mounted to the sidecar container as a script in the path /tmp/k8configs-sync.sh, which gets executed every 30 minutes to download any new kubeconfig files.

4. Verify the Kubernetes Accounts’ kubeconfig files downloaded correctly

Once the changes are applied to Clouddriver,
– Go to Clouddriver’s sidecar container and check if the kubeconfig files are downloaded.
– Also, go to Clouddriver’s main container and verify if the kubeconfig file is available in the same path as in the Account’s kubeconfig file refefence

To Verify the accounts, – We can login to Spinnaker and check the URL http://<Gate-Endpoint>/credentials to see available accounts. – Then, you can go to any Kubernetes pipeline and add a ‘Deploy (Manifest)’ stage to perform a dummy deployment to the target Kubetnets account/cluster.

If the deployment goes through successfully, it is clear that the Kubeconfig files are downloaded to Clouddriver Pod successfully and are getting connected to the target cluster from the Pod.

Leave a Comment

Your email address will not be published.

You may like