Spinnaker – Configuring Dynamic Accounts in Clouddriver for Kubernetes

One of the use-cases that advanced users of Spinnaker look for is to create a Kubernetes cluster during the pipeline deployment or create it just before the pipeline deployment as another pipeline and use the new Kubernetes cluster as the target deployment environment.

To achieve this, Spinnaker has introduced a dynamic account loading feature from Spinnaker version 1.15.x. This feature is an outcome of Spring Cloud Config integration into Cloud-driver, to add support for fetching account configuration from external sources like Git, Vault, or an S3 bucket. This feature also refreshes credentials for Kubernetes dynamically, while Clouddriver is still running.

Configure dynamic account in Spinnaker for Kubernetes

This document describes the steps for configuring the dynamic account in Spinnaker Clouddriver under the following environment,

  • Spinnaker 1.17.4 (However, the same can work on new versions of Spinnaker)
  • External configuration store as a private GitHub repository (https://github.com/sagayd/spin-dynamicaccounts.git)
  • Halyard’s deployment profile is the default (i.e ~/.hal/default directory)

How does Spinnaker load external Git-based Clouddriver configuration to be used for Kubernetes?

Spinnaker loads account details using the cloud driver service configuration. By default, the Clouddriver configuration of account details is retrieved from the halconfig file. We can configure Spinnaker ‘Spring Cloud Config server’ to look for Clouddriver configuration from the external Git source.

Once the Spinnaker is started by the Halyard service, the Spring Cloud server will load the Clouddriver configuration file (cloud driver-local.yml) from the Git repo. Hence, one can update the cloud driver-local.yml file on the Git repo, and the accounts will be refreshed automatically. This means you don’t have to run the “hal deploy apply” command every time you add a new account and so Spinnaker service is not disturbed.

Detailed Procedure to setup Dynamic Account in Spinnaker for Kubernetes

  1. Instruct Spinnaker to look for external configuration from Git repo. 
     
    • Do hal deploy apply and the clouddriver service gets reloaded. This is done only for the first time when you create the spinnakerconfig.yml file. Note: Add your K8s account from halconfig also to clouddriver-local.yml, because clouddriver config is the source of truth for accounts once it is loaded.nstruct Spinnaker to look for external configuration from Git repo.
      • Create a file ~/.hal/default/profiles/spinnakerconfig.yml with the content below
      spring:
        profiles:
          include: git
        cloud:
          config:
            server:
              git:
                uri: https://github.com/OpsMx/spin-dynamicaccounts.git
                username: opsmxdemo
                password: xxxxxxxx
                basedir: /tmp/config-repo
      refresh-rate: 10

    Note: Change the Git repo, its credential, and basedir information as per your requirement.

  2. Use your convenient method (Terraform/CloudFormation/any) to create a K8s cluster in EKS/GKE/AKS/Baremetal environment and store the new kubeconfig file in Git repo – spin-dynamicaccounts.git
  3. Commit the new Cluster’s Kubeconfig file my-k8s-cluster.config under the Repo path: spin-dynamicaccounts.git/k8sconfigs/
  4. Update clouddriver-local.yml with new account information. Account *name* should be unique and the *kubeconfigFile* path should match the relative path of the Git repo. In the code snippet below, the lines after ‘accounts:‘ line and from the ‘name:‘ line is the block to be cloned every time you add a new account.
     
    kubernetes:
          enabled: true
          accounts:
          - name: my-k8s-account
            requiredGroupMembership: []
            providerVersion: V2
            permissions: {}
            dockerRegistries: []
            configureImagePullSecrets: true
            cacheThreads: 1
            namespaces: []
            omitNamespaces: []
            kinds: []
            omitKinds: []
            customResources: []
            cachingPolicies: []
            kubeconfigFile: configserver:k8sconfigs/my-k8s-cluster.config
            oAuthScopes: []
            onlySpinnakerManaged: false
  5. The new account gets populated after one minute approximately.

Verify if your new accounts are loaded

  1. Open your browser, log in to Spinnaker (if required), and access the Credentials page (e.g. https://spindd.opsmx.com:30084/credentials/“). Check if the account information is available.
  2. Use a sample Spinnaker pipeline, and add a Deploy (Manifest) stage. Then in the stage configuration, check if the Account field is able to list your new account.
  3. If the newly added account is not visible in your Spinnaker pipeline, check the clouddriver pod’s log of what is going wrong.

If you want to know more about the Spinnaker or request a demonstration, please book a meeting with us.

OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us

1 Comment
  1. Aniket Kshirsagar 1 year ago
    Reply

    Hi Sandesh,

    Thanks for this documentation. It is really helpful.

    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:

    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable

    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.

    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.

    Hal version:

    Thanks & Regards,
    Aniket Kshirsagar

  2. Aniket Kshirsagar 1 year ago
    Reply

    Hi Sandesh,
    Thanks for this documentation. It is really helpful.
    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:
    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable
    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.
    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.
    Hal version:
    Thanks & Regards,
    Aniket Kshirsagar

  3. Aniket Kshirsagar 1 year ago
    Reply

    Hi Sandesh,
    Thanks for this documentation. It is really helpful.
    But I am getting an issue while configuring dynamic accounts into cloudriver of the spinnaker. Please find error message below:
    2020-01-24 10:44:45.406 WARN 1 — [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization – cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ‘configurationRefreshListener’ defined in URL [jar:file:/opt/clouddriver/lib/clouddriver-web.jar!/com/netflix/spinnaker/clouddriver/listeners/ConfigurationRefreshListener.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘kubernetesV1ProviderSynchronizable’: Invocation of init method failed; nested exception is com.netflix.spinnaker.kork.configserver.ConfigFileLoadingException: File “/home/aniket/.kube/config” not found or is not readable
    Because of this error, clouddriver pod is not getting up and in the “Running” state. I have followed all the steps you have mentioned in the above documentation but facing this issue.
    Can I request you to please help me resolve this issue? Let me know if you need any details from my side.
    Hal version:
    Thanks & Regards,
    Aniket Kshirsagar

Leave a Comment

Your email address will not be published.

You may like