Introduction
Spinnaker as a renowned Continuous Delivery tool supports app deployment into all major cloud environments including Azure cloud. In this blog, we will guide you on how to configure a Spinnaker CI/CD pipeline to deploy an app to the Azure Cloud using Spinnaker.
Prerequisites
- Valid Azure Subscription/Account
- Spinnaker 1.16.2 or newer
- Spinnaker enabled with Jenkins integration to perform your CI build
- Our app is a Java SpringBoot application, to be deployed into the Azure cloud.
- Inputs required for configuring Load Balancer and Firewall:
- The app is listening on port 8080
- Health Check URL on the Hosted machine is http://localhost:8080/health
- App is accessible on the machine at http://localhost:8080/greeting
Preparing Azure for Spinnaker Integration
- Azure CLI for creating Azure resources for Spinnaker
- Azure Portal (portal.azure.com) for Virtual network setup and Action monitoring
Installing Azure CLI
Azure CLI is required to set up pre-requirements on the Azure cloud. Install Azure CLI on your Laptop or any VM which has connectivity to the Azure cloud. Simply run this command
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
For detailed manual instruction, you can follow the guide from the link:
https://docs.microsoft.com/en-gb/cli/azure/install-azure-cli-apt?view=azure-cli-latest
Create Azure resources for Spinnaker Integration
On your Azure account, you will need a few items created…
- A Service Principle (Halyard to communicate with Azure),
- A Resource Group (All of the Azure resources like temp VMs, machine image, etc to be grouped under this name)
- Azure Vault (Default Username and password/key of Azure VMs created Spinnaker are stored here) – SSH Username (Default username of VM) – SSH Password (Default password of VM
Throughout the procedure here, I choose to configure and deploy Azure resources in the eastern US region. But you can change it as per your choice.
Configure Azure CLI to create Azure resources
1. Login to Azure by Azure CLI
az login
Note: We have launched a browser for you to login. For old experience with device code, use “az login –use-device-code” to open in an existing browser session. Once you have logged-in, find all the subscriptions to which you have access:
[
{
"cloudName": "AzureCloud",
"id": "959c83bf-PKljgfr0f16-4a56-a54267be765432e", #Subscription ID
"isDefault": true,
"name": "Microsoft Azure Sponsorship",
"state": "Enabled",
"tenantId": "d1ea76543-64d0-48u4ed-a093-pk8041071ee6",
"user": {
"name": "sagayaraj.d@opsmx.io",
"type": "user"
}
}
]
2. Get your Subscription details.
Note: You can also get your subscription from Azure portal: Home -> Subscriptions
Run the following command to get your subscription details:
az account list
Refer the output looks something like the below:
[
{
"cloudName": "AzureCloud",
"id": "959c83bf-PKljgfr0f16-4a56-a54267be765432e", #Subscription ID
"isDefault": true,
"name": "Microsoft Azure Sponsorship", #Subscription Name
"state": "Enabled",
"tenantId": "d1eay7473-64d0-44poed-a093-ad8041071ee6",
"user": {
"name": "sagayaraj.d@opsmx.io",
"type": "user"
}
}
]
3. Make your to configure Azure CLI to work with your subscription.
#Change the SUBSCRIPTION_ID to match yours
SUBSCRIPTION_ID=959c83bf-PKljgfr0f16-4a56-a54267be765432e
az account set --subscription $SUBSCRIPTION_ID
Create Azure resources1. Create a Service Principal. This is the account by which Spinnaker’s Halyard will communicate with Azure cloud.
# az ad sp create-for-rbac --name "<service-principle-name>" #No longer working due to Azure Cloud updates
az ad sp create-for-rbac --name <service-principle-name> --role contributor --scopes /subscriptions/<Subscription-ID>
Sample:
saga@sagamc:~$ az ad sp create-for-rbac --name "spinnaker-sp"
Changing "spinnaker-sp" to a valid URI of "http://spinnaker-sp", which is the required format used for service principal names
Creating a role assignment under the scope of "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e
Retrying role assignment creation: 1/36
Retrying role assignment creation: 2/36
Retrying role assignment creation: 3/36
{
"appId": "fc061b7e-d5k8c5-474d-a888-f34fd69bd85b",
"displayName": "spinnaker-sp",
"name": "http://spinnaker-sp",
"password": "1fd416e8-15e2-478995-8a6e-aea4eabca3be",
"tenant": "d1ea7473-64d0-44ed-a093-ad8041071ee6"
}
2. Make a note of AppId and TenantId. Optionally set them as variables. Change the values to match yours
3. Create a resource group in eastus region.
APP_ID=fc0ki61b7e-d5c5-474d-a888-f34fd69bd85b
TENANT_ID=d1ea7473-64hygd0-44ed-a093-ad8041071ee6
az account list-locations --query [].name
az group create --name <resource-group-name> --location <location>
Sample:
saga@sagamc:~$ RESOURCE_GROUP=SpinHal-RG
saga@sagamc:~$ az group create --name $RESOURCE_GROUP --location eastus
{
"id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG",
"location": "eastus",
"managedBy": null,
"name": "SpinHal-RG",
"properties": {
"provisioningState": "Succeeded"
},
"tags": null,
"type": "Microsoft.Resources/resourceGroups"
}
4. Create an Azure Vault. The vault will store the default username and password for VMs.
Note: Vault names are used in a DNS record (like spinnakervault.vault.azure.net), so the name must be unique globally.
5. Set Vault permissions
6. Insert the default SSH username of VMs into your Azure vault
VAULT_NAME=<vault-name>
az keyvault create --enabled-for-template-deployment true --resource-group $RESOURCE_GROUP --name $VAULT_NAME
Sample:
saga@sagamc:~$ VAULT_NAME=SpinHal-Vault
saga@sagamc:~$ az keyvault create --enabled-for-template-deployment true --resource-group $RESOURCE_GROUP --name $VAULT_NAME
#Output redacted
{
"id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
"location": "eastus",
"name": "SpinHal-Vault",
"properties": {
"accessPolicies": [
{
"applicationId": null,
"objectId": "9aeada99-4d00y6-4f53-8567-956c8645e905",
"permissions": { "certificates": [..], "keys": [..], "secrets": [..], "storage": [..] },
"tenantId": "d1ea7473-647ytd0-44ed-a093-ad8041071ee6"
}
],
"provisioningState": "Succeeded",
"tenantId": "d1ea7473-6874d0-44ed-a093-ad8041071ee6",
},
"resourceGroup": "SpinHal-RG",
}
az keyvault set-policy --secret-permissions get --name $VAULT_NAME --spn $APP_ID
#Sample output (Redacted):
{
"id": "/subscriptions/959c83bf-PKljgfr0f16-4a56-a54267be765432e/resourceGroups/SpinHal-RG/providers/Microsoft.KeyVault/vaults/SpinHal-Vault",
"location": "eastus",
"name": "SpinHal-Vault",
"properties": {
"accessPolicies": [
{
"applicationId": null,
"objectId": "9aeada99-4d980-4f53-8567-956c8645e905",
"permissions": { "certificates": [..], "keys": [..], "secrets": [..], "storage": [..] },
"tenantId": "d1ea7473-hgtfrd0-44ed-a093-ad8041071ee6"
},
{
"applicationId": null,
"objectId": "05676264-65njhgtc8-48c7-925d-00daba4bc5d6",
"permissions": { "certificates": null, "keys": null, "secrets": [ "get" ], "storage": null },
"tenantId": "d1ea7473-64nhbgvfd0-44ed-a093-ad8041071ee6"
}
],
"createMode": null,
"enablePurgeProtection": null,
"enableSoftDelete": null,
"enabledForDeployment": false,
"enabledForDiskEncryption": null,
"enabledForTemplateDeployment": true,
"networkAcls": null,
"provisioningState": "Succeeded",
"sku": { "name": "standard" },
"tenantId": "d1eahygtr473-64d0-49874ed-a093-ad8041071ee6",
"vaultUri": "https://spinhal-vault.vault.azure.net/"
},
"resourceGroup": "SpinHal-RG",
"tags": {},
"type": "Microsoft.KeyVault/vaults"
}
7. Set the public key in Vault
a. Generate SSH pubkey to insert into Vault
saga@sagamc:~$ ssh-keygen -f ddsaga -N '' #Username is ddsaga (Change it as per your choice)
#cat ddsaga.pub and copy the content without the last part @ [e.g. opsmxuser@spinnaker-saga]
saga@sagamc:~$ VMUSER_KEY="ssh-rsa AAAAB3NzaC1iuytgtgggyc2EAAAADAQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"
b. Insert the default SSH pubkey of VMs into your Azure vault
saga@sagamc:~$ VMUSER_KEY="ssh-rsa AAAAB3NzaC1iuytgtgggyc2EAAAADAQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"
{
"attributes": {
"created": "2019-11-14T03:21:02+00:00",
"enabled": true,
"expires": null,
"notBefore": null,
"recoveryLevel": "Purgeable",
"updated": "2019-11-14T03:21:02+00:00"
},
"contentType": null,
"id": "https://spinhal-vault.vault.azure.net/secrets/VMSshPublicKey/38ab80bc184d4beab03e35dff30c9c8f",
"kid": null,
"managed": null,
"tags": {
"file-encoding": "utf-8"
},
"value": "ssh-rsa AAAAB3NzaC1yc2gfrEAAAADAKKKKQABAAABAQDMGBDPxjYydvel/ZyYoaLM3zDB9tGY+YbkcpbnI/esqR2qGgs/rG9C1MWP3ews3d2F58Ss2F7C03tj+5phbhB+10tmnPn2ezAq3FXcjEenQluVb2QInxnr+bdfDeZLyYcApFemgs0mgU5LcP2ixGQ6Tli9xyziITRuaIr904pIOcbdo8+OLDXFS7W0+KhSu3RjCkD4EWNpbCB48j+Oz8YD6t6X5aixEjFg7v4Swo0+NxAeFteCyzU8hWZAdVW0eWdjo5zeQosMW7tVe+UFBDVgQfjrZd0ue0xOyYRf1HLplIEcyJpjy2DhWbG99FJW9gdbJqxeGLop7lKvbatrpBMp"
}
VMUSER=<vm-login-username>
az keyvault secret set --name VMUsername --value $VMUSER --vault-name $VAULT_NAME
#Note, By default SSH key will be used. If you need password authentication, run the below command and also make sure to update your Azure account in halconfig key useSshPublicKey set to false
az keyvault secret set --name VMPassword --value <password> --vault-name $VAULT_NAME
#Sample
saga@sagamc:~$ VMUSER=ddsaga
saga@sagamc:~$ az keyvault secret set --name VMUsername --value $VMUSER --vault-name $VAULT_NAME
{
"attributes": {
"created": "2019-11-14T03:12:18+00:00",
"enabled": true,
"expires": null,
"notBefore": null,
"recoveryLevel": "Purgeable",
"updated": "2019-11-14T03:12:18+00:00"
},
"contentType": null,
"id": "https://spinhal-vault.vault.azure.net/secrets/VMUsername/a7d987a593552874f76afef83cfd11d8232",
"kid": null,
"managed": null,
"tags": {
"file-encoding": "utf-8"
},
"value": "ddsaga"
}
That brings up the end of activities on Azure CLI.
Integrating Azure provider in Spinnaker
Use Halyard command-line program to enable the integration of Azure in Spinnaker.
Note: The Procedure below should be done on Halyard engine (VM/Pod/Container)
1. During the above procedure “Create Azure resources for Spinnaker Integration“, you would have created resources like Service Principle, Resource Group and Azure Vault. Make a note of those values and set them as Env variables here:
spinnaker@hal-xyz:~$ cat > azureapp.vars <<-EOF
SUBSCRIPTION_ID=959c83bf-PKljgfr0f16-4a56-a54267be765432e
APP_ID=fc061b7e-d5c5-4decde74d-a888-fdsws34fd69bd85b
TENANT_ID=d1ea74xscde73-64d0-44ed-a04de393-ad8041071ee6
RESOURCE_GROUP="SpinHal-RG"
VAULT_NAME=SpinHal-Vault
EOF
spinnaker@hal-xyz:~$ source azureapp.vars
Optionally, you can verify the variables are set correctly
echo $APP_ID
echo $TENANT_ID
echo $SUBSCRIPTION_ID
echo $VAULT_NAME
echo $RESOURCE_GROUP
2. Execute the ‘hal’ command to configure hal with Azure. This step will prompt you for password. The password was outputted for you when Service principle was created via Azure CLI.
spinnaker@hal-xyz:~$ hal config provider azure account add saga-azure-account \
--client-id $APP_ID \
--tenant-id $TENANT_ID \
--subscription-id $SUBSCRIPTION_ID \
--default-key-vault $VAULT_NAME \
--default-resource-group $RESOURCE_GROUP \
--packer-resource-group $RESOURCE_GROUP \
--app-key
3. Enable Azure provider in Spinnaker
spinnaker@hal-xyz:~$ hal config provider azure enable
4. You may review the Azure provider configuration
spinnaker@hal-xyz:~$ hal config provider azure account list
#Output would contain this value - saga-azure-account. Use this value in the next command
#Enable eastus region. The region should match the region configured in the earlier Azure CLI steps
spinnaker@hal-xyz:~$ hal config provider azure account get saga-azure-account
spinnaker@hal-xyz:~$ hal config provider azure account edit saga-azure-account --regions=eastus
5. Complete the hal configuration with “hal deploy apply” command. Note: Now if you create a new application in Spinnaker, the application should have the provider ‘azure’ in the list.
Azure Virtual Network
Spinnaker will deploy applications on a virtual network in the Azure cloud. Hence, ensure to have a Virtual Network and a minimum one Subnet created in eastus region (or another region, it should match hal configuration). You may create the virtual network on portal.azure.com My Virtual network details are…
Network name: Spin_Vnet
CIDR: 172.16.0.0/22
Subnet: default
The Virtual Network and Subnet will be used by Firewall, Load Balancer, and Server group configurations.
Azure and Spinnaker Resource Mapping
Spinnaker generalizes the Cluster terms across multiple clouds. The resource in Spinnaker could mean a different things in the Azure cloud.
- Spinnaker Firewall means Network Security Group in Azure cloud.
- Spinnaker Load Balancer is referred to as Load Balancer only in Azure.
- Spinnaker Server Group is referred to as Virtual Machine ScaleSet in Azure.
Configuring Azure application in Spinnaker UI
Note: Perform these steps in Spinnaker UI.
1. Create an Application
Under Applications, Create an application ‘appxinazure’. (Change the name to your choice). Ensure to select ‘azure‘ in the provider list.
2. Configure Application’s Infrastructure
Prior to configuring a CD pipeline, we need to configure the Infrastructure of the application. A separate resource group by name – will be created in Azure cloud automatically. In our case, ‘appxinazure-eastus’ is the new resource group. The resources created in the Spinnaker UI like Firewall, Load Balancer, VM ScaleSet are stored on this resource group; however, the baked images are stored on the resource group configured in Halyard. ** To successfully configure the Firewall, Load Balancer and Server group – Virtual network and Subnet are one of the pre-requirement. You should have them created. If not, refer to the section Azure Virtual Network to set up one.
a. Create Firewall
b. Create a Load Balancer
You should know the application listening port, health check page, app URL so that the load balancer can be configured correctly.
c. Create Server Group
Do not create a server group using the Infrastructure tab, because you may be unsuccessful – while it will look for an image that will not available until we perform a Bake stage. Hence, the configuring server group is part of the pipeline setup process. You will have to create a server group in the Deploy stage.
Configuring the Spinnaker CI/CD pipeline for deployments to Azure
Under the Pipeline section, click ‘Create’ to create a New Pipeline with the type being selected as Pipeline. Input a valid name for your pipeline.
1. Configure Stage
Here, select your Jenkins CI account to take Build information. Prior to this step, you should have your Jenkins configured in Halyard under CI systems.
2. Bake Stage
We are targeting the package to Ubuntu server (16+), while the base image is baked with our app installed on it. The deb package file name should have been created with the naming convention: name_version-release_arch. Here, select your package (.deb) name to create a baked image. The package is essentially the output of Jenkins Build. This stage will look for a valid package within the Jenkins project directory. Select your region ‘eastus‘ to deploy the application into. Remember the available regions here are activated by hal command. Then, type your package name without any version information
3. Deploy Stage
The deployment server group in Spinnaker will create a VMScaleSet on Azure, the LoadBalancer will route the traffic to the VM ScaleSet.
Add a Server group and configure every section in ‘Configure Deployment Cluster‘. Choose the account, select the region as ‘eastus‘, valid stack that may denote environment like dev/qa/stg. Detail may contain any text, but I chose to have ‘svrgroup’. Then choose the load balancer you had created in one of the above steps.
Choose the Virtual network and Subnet, where the VM ScaleSets will be created during CD pipeline execution.
Choose the Instance type that best fits for your application. Here, we have gone with General purpose computer.
You can associate some tags that may be used as custom meta-data for your VMs, which can be used for filtering and other operations.
Save the configuration. And, you are done with your Pipeline configuration. You are now ready to execute your Pipeline and see how it goes.Troubleshooting Known Issues
1. Problem:
In the Bake stage, the package name was mentioned as restapp_ (since our build output package was restapp_1.0_all.deb). The Baking process throw the error “Unable to find deployable artifact starting with [RestApp ] and ending with .deb in [] and [restapp_1.0_all.deb]”
Solution:
Make sure your deb package file name complies with the naming convention: name_version- release_arch. In the Bake stage, just mention the name part alone (Do not include the underscore).
2. Problem:
The Baking process throws the error “==> azure-arm: E: Unable to locate package restapp”
Solution:
This is possible when you have not configured your Deb path in Spinnaker. Update the rosco-local file with deb repo and redeploy with hal deploy apply command.
On halyard Pod/VM,
$ cat >> /home/spinnaker/.hal/default/profiles/rosco-local.yml <<-EOF
debianRepository: http://spindd.s3-website-us-west-2.amazonaws.com trusty main azure: enabled: true
EOF
03. Problem:
The Baking process throws the error “==> azure-arm: E: Failed to fetch http://spindd.s3-website-us-west-2.amazonaws.com/dists/trusty/main/binary-amd64/Packages 404 Not Found ==> azure-arm: E: Some index files failed to download. They have been ignored, or old ones used instead.”
Solution:
If Deb repository is an S3 bucket, you should enable the web-hosting option and use the Endpoint URL in the rosco-local.yml file. The endpoint URL becomes available once you enable Web-hosting.
04. Problem:
The Baking process throws the error “==> azure-arm: -> Additional Disk: skipping, managed disk was used… Build ‘azure-arm’ errored: unexpected EOF”
Solution:
The problem occurs due to a defect in Packer version 1.4.2 as stated here https://github.com/has hicorp/packer/issues/7816. The fix is to update the Packer version to 1.4.4 which is packaged with Spinnaker 1.17. This is confirmed here https://github.com/spinnaker/rosco/pull/452. Either update the Spinnaker to 1.17 or you can update the rosco.yml file with overridden docker image version which has the Packer 1.4.4 updated. See below how to update the docker version for Rosco.
$ cat > /home/spinnaker/.hal/default/service-settings <<-EOF
artifactId: gcr.io/spinnaker-marketplace/rosco:0.15.0-20191023172815
EOF
05. Problem:
Got an error in the deployment stage… “Exception ( Monitor Deploy ) Multiple errors occurred: NotFound, NotFound. Please see details., Unexpected exception: null! Please login into Azure Portal and manually delete any resource associated with the appxinazure-dev-fe-v000 server group such as storage accounts, internal load balancer, public IP, and subnets”
Solution:
Spinnaker did not provide enough log information as to which resource is NotFound. In tracing the resources everything we created, found that the items – VMUsername, and VMSshPublicKey were missing in Azure vault SpinHal-vault
After adding the SSH key and public-key, the deployment stages went successfully.
Refer to the Azure CLI procedure again to create the Username and Public-key. In particular the commands,
saga@sagamc:~$ az keyvault secret set --name VMUsername --vault-name $VAULT_NAME --value $VMUSER
saga@sagamc:~$ az keyvault secret set --name VMSshPublicKey --vault-name $VAULT_NAME --value "$VMUSER_KEY"
06. Problem:
Solution:
References:
https://www.spinnaker.io/setup/install/providers/azure/ https://www.spinnaker.io/guides/tutorials/codelabs/azure-vmss-source-to-prod/
If you want to know more about Spinnaker or request a demonstration, please book a meeting with us.
OpsMx is a leading provider of Continuous Delivery solutions that help enterprises safely deliver software at scale and without any human intervention. We help engineering teams take the risk and manual effort out of releasing innovations at the speed of modern business. For additional information, contact us
Hey,
I am getting below error while trying to bake an image. Can you please here?
==> azure-arm: Running builder …
==> azure-arm: Getting tokens using client secret
azure-arm: Creating Azure Resource Manager (ARM) client …
==> azure-arm: WARNING: Zone resiliency may not be supported in eastus, checkout the docs at https://docs.microsoft.com/en-us/azure/availability-zones/
==> azure-arm: Creating resource group …
==> azure-arm: -> ResourceGroupName : ‘packer-Resource-Group-ei9rkh0m5d’
==> azure-arm: -> Location : ‘eastus’
==> azure-arm: -> Tags :
==> azure-arm: Validating deployment template …
==> azure-arm: -> ResourceGroupName : ‘packer-Resource-Group-ei9rkh0m5d’
==> azure-arm: -> DeploymentName : ‘pkrdpei9rkh0m5d’
==> azure-arm: Deploying deployment template …
==> azure-arm: -> ResourceGroupName : ‘packer-Resource-Group-ei9rkh0m5d’
==> azure-arm: -> DeploymentName : ‘pkrdpei9rkh0m5d’
==> azure-arm: ERROR: -> InvalidTemplateDeployment : The template deployment failed because of policy violation. Please see details for more information.
==> azure-arm: ERROR: -> RequestDisallowedByPolicy : Resource ‘pkrvmei9rkh0m5d’ was disallowed by policy. Policy identifiers: ‘[{“policyAssignment”:{“name”:”Allowed virtual machine SKUs”,”id”:”/providers/Microsoft.Management/managementGroups/Active/providers/Microsoft.Authorization/policyAssignments/832e92f535ee4d3db44f8301″},”policyDefinition”:{“name”:”Allowed virtual machine SKUs”,”id”:”/providers/Microsoft.Authorization/policyDefinitions/cccc23c7-8427-4f53-ad12-b6a63eb452b3″}}]’.
==> azure-arm:
==> azure-arm: resources.DeploymentsClient#CreateOrUpdate: Failure sending request: StatusCode=400 — Original Error: Code=”InvalidTemplateDeployment” Message=”The template deployment failed because of policy violation. Please see details for more information.” Details=[{“additionalInfo”:[{“info”:{“evaluationDetails”:{“evaluatedExpressions”:[{“expression”:”type”,”expressionKind”:”Field”,”expressionValue”:”Microsoft.Compute/virtualMachines”,”operator”:”Equals”,”path”:”type”,”result”:”True”,”targetValue”:”Microsoft.Compute/virtualMachines”},{“expression”:”Microsoft.Compute/virtualMachines/sku.name”,”expressionKind”:”Field”,”expressionValue”:”Standard_DS2_v2″,”operator”:”In”,”path”:”properties.hardwareProfile.vmSize”,”result”:”False”,”targetValue”:[“standard_b1s”,”standard_b2s”,”standard_b2ms”,”standard_a2_v2″,”standard_d2_v3″,”standard_d2s_v3″,”standard_f1s”,”standard_f1″,”standard_f2″,”standard_f2s”,”standard_d1_v2″,”standard_ds1_v2″,”standard_f2s_v2″,”standard_e2s_v3″]}]},”policyAssignmentDisplayName”:”Allowed virtual machine SKUs”,”policyAssignmentId”:”/providers/Microsoft.Management/managementGroups/Active/providers/Microsoft.Authorization/policyAssignments/832e92f535ee4d3db44f8301″,”policyAssignmentName”:”832e92f535ee4d3db44f8301″,”policyAssignmentParameters”:{“listOfAllowedSKUs”:{“value”:[“standard_b1s”,”standard_b2s”,”standard_b2ms”,”standard_a2_v2″,”standard_d2_v3″,”standard_d2s_v3″,”standard_f1s”,”standard_f1″,”standard_f2″,”standard_f2s”,”standard_d1_v2″,”standard_ds1_v2″,”standard_f2s_v2″,”standard_e2s_v3″]}},”policyAssignmentScope”:”/providers/Microsoft.Management/managementGroups/Active”,”policyDefinitionDisplayName”:”Allowed virtual machine SKUs”,”policyDefinitionEffect”:”Deny”,”policyDefinitionId”:”/providers/Microsoft.Authorization/policyDefinitions/cccc23c7-8427-4f53-ad12-b6a63eb452b3″,”policyDefinitionName”:”cccc23c7-8427-4f53-ad12-b6a63eb452b3″},”type”:”PolicyViolation”}],”code”:”RequestDisallowedByPolicy”,”message”:”Resource ‘pkrvmei9rkh0m5d’ was disallowed by policy. Policy identifiers: ‘[{\”policyAssignment\”:{\”name\”:\”Allowed virtual machine SKUs\”,\”id\”:\”/providers/Microsoft.Management/managementGroups/Active/providers/Microsoft.Authorization/policyAssignments/832e92f535ee4d3db44f8301\”},\”policyDefinition\”:{\”name\”:\”Allowed virtual machine SKUs\”,\”id\”:\”/providers/Microsoft.Authorization/policyDefinitions/cccc23c7-8427-4f53-ad12-b6a63eb452b3\”}}]’.”,”target”:”pkrvmei9rkh0m5d”}]
==> azure-arm:
==> azure-arm: Cleanup requested, deleting resource group …