Dynamic External Account Configuration in Spinnaker

Introduction

Can we keep the cloud provider account details in a remote repository? If so, what if an account change happens in that remote configuration while Spinnaker service is running? Will it take effect without service restart? What are the steps involved? What other use cases are possible with remote configuration?

This blog provides answers to the above questions.

In Spinnaker world, the credentials configured through halyard using which Spinnaker deploys the applications into various Cloud Platforms are known as Accounts. Typically complete account configuration resides in hal config. As the Spinnaker is evolving, new features are being incorporated and one of them is External Account Configuration which allows the configurations to be externalized via Git, Hashicorp Vault, databases, CredHub etc.  

‘hal deploy apply’ is a Spinnaker mantra for any account change to take effect. Broadly account can be of any cloud provider, artifactory, CI system, storage system etc. When such account is added or updated or deleted, ‘hal deploy apply’ makes Spinnaker understand the change. But this also means a restart of one or more affected Spinnaker services. 

Spinnaker is built on Spring framework and uses various Spring modules. One of such modules is Spring Cloud Configuration which allows dynamic external configuration using Config Server and Client implementation. A client application can be refreshed for reloading any account changes that Config server is designed to capture. Spinnaker service need not be re-started for the account changes in external configuration. This refresh can be automatic or manual. As of Spinnaker 1.17.5, only Clouddriver service supports automatic refreshing of external accounts and only for Kubernetes and Cloud Foundry cloud providers. For manual triggering, /refresh endpoint on the service be http POSTed. 

Configuration

A file named spinnakerconfig.yml is a bootstrap configuration file that provides location of remote configuration so that Spinnaker services like clouddriver, igor fetch configuration from the remote location at start up. The following examples show the configuration with Git as remote repository. Spinnaker uses profiles like <service>-local.yml for additional configuration and profile is loaded from remote configuration, if present. Remote locations can be one or a mix of the following backends.

  • Git
  • Hashicorp Vault
  • JDBC
  • CredHub
  • Local File System

Next sections provide step by step instructions for the following dynamic configurations:

  • Account Configuration
  • Secrets Configuration
  • Secrets Encryption
  • Configuration Files

Using this dynamic account configuration feature, one or more cloud provider accounts(Kubernetes, Azure, AWS, Cloud Foundry etc) and/or CI accounts and/or notification services can be configured in the external sources mentioned above. 

Keeping all the secrets in plain text in hal config may not always align with organizational policies so using dynamic configuration feature, it is possible to ship the secrets to external repositories. The secrets in the external repositories can also be encrypted and Spinnaker services decrypt them at runtime.

Configuration files like Kubernetes config file, Google Cloud configuration file can also be kept in Git or other backends and it can be configured in Spinnaker to retrieve them at runtime.       

Account Configuration

This section explains how to configure the following accounts with Git as an external source. Same configuration can be extended to any other cloud provider accounts.

  • Kubernetes Account
  • AWS Account
  • Azure Account
  • Jenkins Master Account

1. Create a Git repository for storing the accounts

Let the repository be private with name spinnaker-config and the github account username be opsmx, so the git repository url is https://github.com/opsmx/spinnaker-config.git.  

2. Place spinnakerconfig.yml file under ~/.hal/default/profiles/

The file spinnakerconfig.yml (or <service-name>config.yml, i.e., clouddriverconfig.yml, igorconfig.yml etc) contains the external repository details using which Spinnaker services fetch the account configurations. Following spinnakerconfig.yml must be placed under ~/.hal/default/profiles/ in halyard pod or the machine in which halyard service is running. If a spinnaker service (say clouddriver pod) is configured to run without halyard, then file is pushed under /opt/spinnaker/config/ in the respective spinnaker service pod at start up.  

spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/spinnaker-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo

3. Create a file ‘clouddriver-local.yml’ in Git repository.

The file clouddriver-local.yml contains all the cloud provider accounts that are to be externally configured. Following configurations are contained in this file:

Kubernetes Account
    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: /home/opsmx/.hal/default/staging/dependencies/1031763043-config
        oAuthScopes: []
        onlySpinnakerManaged: false

Note that the kubeconfigFile is not configured in Git. External config files configuration is explained in the following sections. 

AWS Account
    aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '123456789012'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
        awsAccessKey: AKIAXXXXXXXXXXXZHKU5 
        awsSecretKey: YqfRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ
      accessKeyId: ALNMXXXXXXXXXXXZHKT8
      secretAccessKey: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl4dK
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole
Azure Account
   azure:
      enabled: true
      accounts:
      - name: my-azure-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        clientId: 921f7add-XXXX-XXXX-XXXX-c1a695s08755
        appKey: 23373f24-XXXX-XXXX-XXXX-6b0b5113f5a0
        tenantId: d1ea7473-XXXX-XXXX-XXXX-ad8041071ee6
        subscriptionId: 959c83bf-XXXX-XXXX-XXXX-2067be19692e
        defaultResourceGroup: RG-OpsMx
        defaultKeyVault: SpinHal-OpsMx-Vault
        packerResourceGroup: RG-OpsMx
        regions:
        - westus
        - eastus
        useSshPublicKey: 'true'
      primaryAccount: my-azure-account
      bakeryDefaults:
        templateFile: azure-linux.json
        baseImages: []

For simplicity, only one account per each cloud provider are configured but they can be more than one. Note that any accounts of the above three cloud provides configured in hal config are overridden by the above configuration.

4. Create a file ‘igor-local.yml’ in Git repository.

We configure Jenkins Master account in igor-local.yml and place it directly under the Git repository created in step 1. Following configuration is contained in this file:

 ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: http://jenkins.opsmx.com
        username: opsmx
        password: mypassword

5. Run ‘hal deploy apply’

At this point, there are two files – clouddriver-local.yml and igor-local.yml – in the Git repository and Git repository details are configured in ~/.hal/default/profiles/spinnakerconfig.yml. Upon running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into all spinnaker service pods under /opt/spinnaker/config/ and the pods fetch the respective <service>-local.yml files into their /tmp/config-repo/ directory and load the accounts. In our scenario, clouddriver and igor pods loads the accounts from clouddriver-local.yml and igor-local.yml files respectively.  

Secrets Configuration

Following procedure details the secrets configuration using Git. Rest of the configuration lies with halyard. Two scenarios are explained here : 1. AWS access key id and secret key and 2. Jenkins Master address, username and password.   

1. Configure hal config with Spring property placeholders.

AWS Account Configuration in hal config

Configure AWS Account in hal config with spring property placeholders for AWS Access Key Id and Secret Key  which get replaced with the actual values at run time by the clouddriver service. After configuring, the AWS part of the hal config looks like the following:

   aws:
      enabled: true
      accounts:
      - name: my-aws-account
        requiredGroupMembership: []
        providerVersion: V1
        permissions: {}
        accountId: '732813442182'
        regions:
        - name: us-west-2
        assumeRole: role/spinnakerManaged
        lifecycleHooks: []
      primaryAccount: my-aws-account
      bakeryDefaults:
        baseImages: []
      accessKeyId: ${aws.access-key}
      secretAccessKey: ${aws.secret-key}
      defaultKeyPairTemplate: '{{name}}-keypair'
      defaultRegions:
      - name: us-west-2
      defaults:
        iamRole: BaseIAMRole 
Jenkins Master Account Configuration in hal config

        Similarly configure hal config for Jenkins Master Account with spring property placeholders for Jenkins Master address, username and password as shown below:

  ci:
    jenkins:
      enabled: true
      masters:
      - name: opsmx-jenkins-master
        permissions: {}
        address: ${jenkins.address}
        username: ${jenkins.username}
        password: ${jenkins.password}

 

2. Create a Git repository for storing the secrets.

Let the repository be private with name secrets-config and the github account username be opsmx, so the git repository url is https://github.com/opsmx/secrets-config.git.

3. Create clouddriver-local.yml file in Git repository

The file clouddriver-local.yml contains the actual values for the AWS Access Key and Secret Key:

aws:
    access-key: ALIAXXXXXXXXXXXZHKU5
    secret-key: XqrRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXNl9dJ

4. Create igor-local.yml file in Git repository

The file igor-local.yml contains the actual values for the Jenkins Master Account’s address, username and password:

jenkins:
    address: http://jenkins.opsmx.com
    username: opsmx
    password: mypassword

5. Enable external configuration through spinnakerconfig.yml

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which halyard service is running with the following configuration.

spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/secrets-config.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo

6. Run ‘hal deploy apply’

Now all the configuration is ready and by running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into clouddriver and igor pods which fetch the secrets from Git and replace the spring property placeholders at runtime.

Secrets Encryption

Please refer to this blog to know how to externalize encrypted secrets:

Managing Secrets in Spinnaker – Encryption using Symmetric Key

Configuration Files

Following steps explain how to configure a kubeconfig file which is checked-in a git repository.

1. Create a Git repository for storing the config files

Let the repository be private with name config-files and the github account username be opsmx, so the git repository url is https://github.com/opsmx/config-files.git.

2. Create a file kubeconfig.yml in the Git repository

Create kubeconfig.yml which provides cluster access for organizing Kubernetes accounts, in the Git repository created in step 1.

3. Refer kubeconfig.yml in clouddriver-local.yml

Create clouddriver-local.yml file in the Git repository file which contains the Kubernetes account details including the path to kubeconfig file. Since clouddriver service is backed by Spring Cloud’s config server, by configuring appropriately, it fetches the kubeconfig file from the Git repository.  

clouddriver-local.yml
    kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-v2
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: configserver:kubeconfig.yml
        oAuthScopes: []
        onlySpinnakerManaged: false

4. Place spinnakerconfig.yml file under ~/.hal/default/profiles/.

Place the file spinnakerconfig.yml under ~/.hal/default/profiles/ in the halyard pod or the machine in which halyard service is running with the following configuration.

spinnakerconfig.yml
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/opsmx/config-files.git
          username: opsmx
          password: xxxxxxxx
          basedir: /tmp/config-repo

5. Run ‘hal deploy apply’

By running ‘hal deploy apply’, halyard pushes the spinnakerconfig.yml into clouddriver pod which loads the Kubernetes account and kubeconfig.yml from Git.

 

Leave a Comment

Your email address will not be published.

You may like