Spinnaker – Configuring Dynamic Accounts in Clouddriver for Kubernetes

One of the use-cases that advanced users of Spinnaker look for is to create  Kubernetes cluster during the pipeline deployment or create it just before the pipeline deployment as another pipeline and use the new Kubernetes cluster as the target deployment environment.

To achieve this, Spinnaker has introduced a dynamic account loading feature from Spinnaker version 1.15.x. This feature is an outcome of Spring Cloud Config integration into Cloud-driver, to add support for fetching account configuration from external sources like Git, Vault or an S3 bucket. This feature also refreshes credentials for Kubernetes dynamically, while Cloud-driver is still running.

Scope

This document describes the steps for configuring the dynamic account in Spinnaker Clouddriver under the following environment,

  • Spinnaker 1.17.4 (However the same can work on new versions of Spinnaker)
  • External configuration store as a private GitHub repository (https://github.com/sagayd/spin-dynamicaccounts.git)
  • Halyard’s deployment profile is the default (i.e ~/.hal/default directory)

How does it work?

Spinnaker loads Account details using the cloud driver service configuration. By default, the Clouddriver configuration of account details are retrieved from halconfig file. We can configure Spinnaker ‘Spring Cloud Config server’ to look for Clouddriver configuration from the external Git source.

Once the Spinnaker is started by Halyard service, Sping Cloud server will load the Clouddriver configuration file (cloud driver-local.yml) from Git repo. Hence, one can update the cloud driver-local.yml file on the Git repo, and the accounts will be refreshed automatically. This means you don’t have to run hal deploy apply command every time you add a new account and so Spinnaker service is not disturbed.

Detailed Procedure

1. Instruct Spinnaker to look for external configuration from Git repo.

  • Create a file ~/.hal/default/profiles/spinnakerconfig.yml with the content below
spring:
  profiles:
    include: git
  cloud:
    config:
      server:
        git:
          uri: https://github.com/OpsMx/spin-dynamicaccounts.git
          username: opsmxdemo
          password: xxxxxxxx
          basedir: /tmp/config-repo
refresh-rate: 10
 
                 Note: Change the Git repo, its credential and basedir information as per your requirement.
 
  • Do hal deploy apply and the clouddriver service gets reloaded. This is done only for the first time when you create the spinnakerconfig.yml file. Note: Add your K8s account from halconfig also to clouddriver-local.yml, because clouddriver config is the source of truth for accounts once it is loaded.

2. Use your convenient method (Terraform/CloudFormation/any) to create K8s cluster in EKS/GKE/AKS/Baremetal environment and store the new kubeconfig file in Git repo – spin-dynamicaccounts.git

3. Commit the new Cluster’s Kubeconfig file my-k8s-cluster.config under the Repo path: spin-dynamicaccounts.git/k8sconfigs/

4. Update clouddriver-local.yml with new account information. Account *name* should be unique and *kubeconfigFile* path should match relative path of Git repo. In the code snippet below, the lines after ‘accounts:‘ line and from ‘name:‘ line are the block to be cloned everytime you add a new account.

kubernetes:
      enabled: true
      accounts:
      - name: my-k8s-account
        requiredGroupMembership: []
        providerVersion: V2
        permissions: {}
        dockerRegistries: []
        configureImagePullSecrets: true
        cacheThreads: 1
        namespaces: []
        omitNamespaces: []
        kinds: []
        omitKinds: []
        customResources: []
        cachingPolicies: []
        kubeconfigFile: configserver:k8sconfigs/my-k8s-cluster.config
        oAuthScopes: []
        onlySpinnakerManaged: false

5. The new account gets populated after one minute approximately.

Verify if your new accounts are loaded

1. Open your browser, login to Spinnaker (if required) and access Credentials page (e.g. https://spindd.opsmx.com:30084/credentials/”). Check if the account information is available.

2. Use a sample pipeline and add a Deploy (Manifest) stage, then in the Stage configuration, check the Account field is able to list your new account.

3. If the above steps don’t load the new configuration, check the clouddriver pod’s log of what is going wrong.

 

1 Comment
  1. Aniket Kshirsagar 10 months ago
    Reply

    Hi Sandesh,

    Thanks for this documentation. It is really helpful.

    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:

    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable

    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.

    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.

    Hal version:

    Thanks & Regards,
    Aniket Kshirsagar

  2. Aniket Kshirsagar 10 months ago
    Reply

    Hi Sandesh,
    Thanks for this documentation. It is really helpful.
    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:
    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable
    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.
    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.
    Hal version:
    Thanks & Regards,
    Aniket Kshirsagar

  3. Aniket Kshirsagar 10 months ago
    Reply

    Hi Sandesh,
    Thanks for this documentation. It is really helpful.
    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:
    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable
    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.
    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.
    Hal version:
    Thanks & Regards,
    Aniket Kshirsagar

Leave a Comment

Your email address will not be published.

You may like