Create and update the service principal key for Azure Kubernetes Service (AKS)

Create and update the service principal key in AKS

While deploying an application to the Azure kubernetes clusters with Spinnaker, users need Azure accounts with proper authorization to access the Azure kubernetes resources. An Azure service principal (a special user) is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access key is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.

By default, AKS clusters are created with a service principal that has a one-year expiration time. In the Azure portal for any kubernetes cluster which is older than one year, you can have issues with client secret keys (which are expiring). Before the expiration date, the Azure server admin of the operations team in your organization can create a new service principal and update the kubernetes cluster to use these new credentials to extend the service principal for an additional period of time. Failure to renew and update the Azure service principal can halt your deployments to the AKS clusters.

PVC (persistent volume claim) is storage allocated in the Azure kubernetes cluster used to store Spinnaker applications/pipeline details/metadata. You can access the PVC storage to retrieve and restore these data any time you need. 

Issue: So while creating PVC, running the following commands can give you errors:

kubectl get pvc 

NAME  STATUS   VOLUME   CAPACITY   ACCESSMODES STORAGECLASS  AGE
Halyard-home-myoes-spinnaker-halyard-0 pending       default                 5d17h
oes-db-postgresql-oes-db-0 pending       default                 5d17h
redis-data-myoes-redis-master-0 pending       default                 5d17h

      kubectl describe pvc  <<name of the pod>> 

Error:

Warning ProvisioningFailed 13s (x2 over 2m27s) persistentvolume-controller (combined from similar events): Failed to provision volume with StorageClass "default": azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e/resourceGroups/mc_aks-spinnaker_spinnaker_westus2/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-e7ba0b44-be64-47d6-bdc8-3e5a6d521cf3?api-version=2018-09-30: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: f742d43c-2699-41f6-a32b-7bc8fc4e0600\r\nCorrelation ID: 53c26262-fbb9-4d67-a00b-15234cb1736c\r\nTimestamp: 2020-11-20 10:44:27Z","error_codes":[7000222],"timestamp":"2020-11-20 10:44:27Z","trace_id":"f742d43c-2699-41f6-a32b-7bc8fc4e0600","correlation_id":"53c26262-fbb9-4d67-a00b-15234cb1736c","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

Solution:

Steps to create a new service principal and update the AKS cluster :

  1. Login to Azure portal using Azure CLI

    az login

    To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AEKKZKDNJ to authenticate. Please make sure that you have admin privileges to the Azure portal. Use the admin account for device verification.

  2. Create a new service principal


    To create a service principal and then update the AKS cluster to use these new credentials, use the
    az ad sp create-for-rbac command, –skip-assignment parameter prevents any additional default assignments being assigned:

    az ad sp create-for-rbac --skip-assignment

    The output is similar to the following example. Make a note of your own appId and password. These values are used in the next step.

    {

      "appId": "7d837646-b1f3-443d-874c-fd83c7c739c5",

      "name": "7d837646-b1f3-443d-874c-fd83c7c739c",

      "password": "a5ce83c9-9186-426d-9183-614597c7f2f7",

      "tenant": "a4342dc8-cd0e-4742-a467-3129c469d0e5"

    }

    Now define variables for the service principal ID and client secret using the output from your own az ad sp create-for-rbac command, as shown in the following example. The SP_ID is your appId, and the SP_SECRET is your password:

    SP_ID=7d837646-b1f3-443d-874c-fd83c7c739c5

    SP_SECRET=a5ce83c9-9186-426d-9183-614597c7f2f7

  3. Update AKS cluster with new service principal credentials

    Now update the AKS cluster with your new credentials using the az aks update-credentials command. The variables for the –service-principal and   –client-secret are used:

    az aks update-credentials \

    --resource-group <<myResourceGroup>> \

    --name <<myAKSCluster>> \

    --reset-service-principal \

    --service-principal $SP_ID \

    --client-secret $SP_SECRET

    For small and medium-sized clusters, it takes a few moments for the service principal credentials to be updated in the AKS but for large clusters, updating the AKS cluster with a new service principal may take a long time to complete.

    Once the update is completed you are good to go and work on the cluster. Now running the following command gives you error-free result,

    kubectl get pvc 

    NAME  STATUS  VOLUME  CAPACITY   ACCESSMODES STORAGECLASS  AGE
    Halyard-home-myoes-spinnaker-halyard-0 Bound    pvc-3ed929f7-f9cc-41c1-a5fc-af3e341d217c 10Gi          RWO                  default                 5d17h
    oes-db-postgresql-oes-db-0 Bound    pvc-7d2938d0-8a0d-4b81-b8fe-b5eb216949f9 8Gi          RWO                  default                 5d17h
    redis-data-myoes-redis-master-0 Bound    pvc-871ce371-1fb9-4897-80ec-80d95fd98087  8Gi  RWO                  default                 5d17h

    OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us.

Leave a Comment

Your email address will not be published.

You may like