While deploying an application to the Azure Kubernetes Service (AKS) clusters with Spinnaker, users need Azure accounts with proper authorization ( service principal key ) to access the Azure Kubernetes Service (AKS) resources. An Azure service principal (a special user) is an identity created for use with applications, hosted services, and automated tools to access Azure resources. This access key is restricted by the roles assigned to the service principal, giving you control over which resources can be accessed and at which level.
Has the AKS Service Principal Key Expired ? Read on to reset duration.
By default, Azure Kubernetes Service (AKS) clusters are created with a service principal that has a one-year expiration time. In the Azure portal for any Kubernetes cluster which is older than one year, you can have issues with client secret keys (which are expiring).
Error Code: aadsts7000222
Before the expiration date, the Azure server admin of the operations team in your organization can create a new service principal and update the Kubernetes cluster to use these new credentials to extend the service principal for an additional period of time. Failure to renew and update the Azure service principal can halt your deployments to the AKS clusters.
PVC (persistent volume claim) is storage allocated in the Azure Kubernetes cluster used to store Spinnaker applications/pipeline details/metadata. You can access the PVC storage to retrieve and restore these data any time you need.
Issue: So while creating PVC, running the following commands can give you errors:
kubectl get pvc
| NAME | STATUS | VOLUME | CAPACITY | ACCESSMODES | STORAGECLASS | AGE |
| Halyard-home-myoes-spinnaker-halyard-0 | pending | default | 5d17h | |||
| oes-db-postgresql-oes-db-0 | pending | default | 5d17h | |||
| redis-data-myoes-redis-master-0 | pending | default | 5d17h |
kubectl describe pvc <>
Error:
Warning ProvisioningFailed 13s (x2 over 2m27s) persistentvolume-controller (combined from similar events): Failed to provision volume with StorageClass "default": azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/subscriptions/959c83bf-0f16-4a56-a542-2067be19692e/resourceGroups/mc_aks-spinnaker_spinnaker_westus2/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-e7ba0b44-be64-47d6-bdc8-3e5a6d521cf3?api-version=2018-09-30: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials\r\nTrace ID: f742d43c-2699-41f6-a32b-7bc8fc4e0600\r\nCorrelation ID: 53c26262-fbb9-4d67-a00b-15234cb1736c\r\nTimestamp: 2020-11-20 10:44:27Z","error_codes":[7000222],"timestamp":"2020-11-20 10:44:27Z","trace_id":"f742d43c-2699-41f6-a32b-7bc8fc4e0600","correlation_id":"53c26262-fbb9-4d67-a00b-15234cb1736c","error_uri":"https://login.microsoftonline.com/error?code=7000222"}
Solution:
Steps to create a new service principal and update the AKS cluster :
1. Login to Azure portal using Azure CLI
az login
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AEKKZKDNJ to authenticate. Please make sure that you have admin privileges to the Azure portal. Use the admin account for device verification.
2. Create a new service principal
To create a service principal and then update the AKS cluster to use these new credentials, use the az ad sp create-for-rbac command, –skip-assignment parameter prevents any additional default assignments being assigned:
az ad sp create-for-rbac --skip-assignment
The output is similar to the following example. Make a note of your own appId and password. These values are used in the next step.
{
"appId": "7d837646-b1f3-443d-874c-fd83c7c739c5",
"name": "7d837646-b1f3-443d-874c-fd83c7c739c",
"password": "a5ce83c9-9186-426d-9183-614597c7f2f7",
"tenant": "a4342dc8-cd0e-4742-a467-3129c469d0e5"
}
Now define variables for the service principal ID and client secret using the output from your own az ad sp create-for-rbac command, as shown in the following example. The SP_ID is your appId, and the SP_SECRET is your password:
SP_ID=7d837646-b1f3-443d-874c-fd83c7c739c5
SP_SECRET=a5ce83c9-9186-426d-9183-614597c7f2f7
3. Update AKS cluster with new service principal credentials
Now update the AKS cluster with your new credentials using the az aks update-credentials command. The variables for the –service-principal and –client-secret are used:
az aks update-credentials \
--resource-group <> \
--name <> \
--reset-service-principal \
--service-principal $SP_ID \
--client-secret $SP_SECRET
For small and medium-sized clusters, it takes a few moments for the service principal credentials to be updated in the AKS but for large clusters, updating the AKS cluster with a new service principal may take a long time to complete.
Once the update is completed you are good to go and work on the cluster. Now running the following command gives you error-free result,
kubectl get pvc
| NAME | STATUS | VOLUME | CAPACITY | ACCESSMODES | STORAGECLASS | AGE |
| Halyard-home-myoes-spinnaker-halyard-0 | Bound | pvc-3ed929f7-f9cc-41c1-a5fc-af3e341d217c | 10Gi | RWO | default | 5d17h |
| oes-db-postgresql-oes-db-0 | Bound | pvc-7d2938d0-8a0d-4b81-b8fe-b5eb216949f9 | 8Gi | RWO | default | 5d17h |
| redis-data-myoes-redis-master-0 | Bound | pvc-871ce371-1fb9-4897-80ec-80d95fd98087 | 8Gi | RWO | default | 5d17h |
Thank Bhai, you really saved me