Azure Automation PS Version Mismatch Woes

7 minute read

Description

When using Azure Automation, it is common for powershell to work on your computer but behave different when ran from the Azure Automation account in the portal. Windows 10 uses runtime 5.1 but the Az module version for Automation Accounts is only 8.0.0 (you can see the on the Modules blade of an Automation Account in the portal). This causes many pain points so be aware. Also reference this doc from Microsoft when you find yourself having issues coding runbooks.

Anyhow, I would like to discuss an issue I had around 2023-08 where I was trying to get runbooks to shut down/start up AKS clusters…

For the scripts like 12x5-start-aks-appg.ps1 (link coming soon), I thought I could update them because they were using code like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

   Set-Item Env:\SuppressAzurePowerShellBreakingChangeWarnings "true"

   try
   {
      "Logging in to Azure..."
      Connect-AzAccount -Identity
   }

   catch {
      Write-Error -Message $_.Exception
      throw $_.Exception
   }

   get-AzAksCluster | ForEach-Object {                       
   Get-AzResource -ResourceId $_.id } | Get-AzResource  | where { $_.Properties.Powerstate.code -eq "Running" -and $_.Properties.ProvisioningState -eq "Succeeded" -and $_.Tags.Schedule -eq "adhoc"} | 
   ForEach-Object { 
      Stop-AzAksCluster -Name $_.Name -ResourceGroupName $_.ResourceGroupName -AsJob 
      Get-AzApplicationGateway -ResourceGroupName $_.ResourceGroupName | Stop-AzApplicationGateway -AsJob}

But it turned out to me a lot more difficult than I had planned! Here is what happened:

Steps

  1. So first, I tried a more modern approach like:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

    # Step 1: Loop through all Clusters, filter based on tags, and stop them

$Clusters = Get-AzAksCluster

Write-Output "Processing $($Clusters.count) clusters..."

$filteredClusters = $Clusters | Where-Object { $_.Powerstate.code -eq "Running" -and $_.ProvisioningState -eq "Succeeded" -and $_.Tags.Schedule -eq "adhoc" }

if ( $null -eq $filteredClusters )
{
   Write-Output "No Clusters meet criteria to continue..."
   Write-Output "Completed"
}
Else
{
   Foreach ($Cluster in $filteredClusters )
   {
      Write-Output "Stopping cluster $($Cluster.Name) ..."
      Stop-AzAksCluster -Name $($Cluster.Name) -ResourceGroupName $($Cluster.ResourceGroupName) -AsJob
      Write-Output "Stopping cluster $($Cluster.Name) ...Completed"
      Write-Output "Stopping all App Gateways in $($Cluster.ResourceGroupName) ..."
      Get-AzApplicationGateway -ResourceGroupName $($Cluster.ResourceGroupName) | Stop-AzApplicationGateway -AsJob
      Write-Output "Stopping all App Gateways in $($Cluster.ResourceGroupName) ...Completed"
   }
}
    • Clean and simple. Works on my machine when testing. But when it runs in the portal you get:
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

    Logging in to Azure...


Mode             : Process
ContextDirectory : 
ContextFile      : 
CacheDirectory   : 
CacheFile        : 
Settings         : {}

Logging in to Azure...Completed

Processing 1 clusters...

No Clusters meet criteria to continue...

Completed

  2. I eventually found it was because it was not able to get the power state:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

    # Step 1: Loop through all Clusters, filter based on tags, and stop them

$Clusters = Get-AzAksCluster

Write-Output "Processing $($Clusters.count) clusters..."

Write-output "Clusters:"
Write-Output $Clusters

$filteredClusters = $Clusters | Where-Object { $_.Powerstate.code -eq "Running" -and $_.ProvisioningState -eq "Succeeded" -and $_.Tags.Schedule -eq "adhoc" }

Write-Output $filteredClusters

Write-OUtput "Powerstate:"
Write-Output $clusters[0].Powerstate.code
    • Produces
    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

    Logging in to Azure...


Mode             : Process
ContextDirectory : 
ContextFile      : 
CacheDirectory   : 
CacheFile        : 
Settings         : {}

Logging in to Azure...Completed

Processing 1 clusters...

Clusters:


ProvisioningState       : Succeeded
MaxAgentPools           : 100
KubernetesVersion       : 1.25.6
DnsPrefix               : aa-nonprd-scus-spk-aks
Fqdn                    : aa-nonprd-scus-spk-aks-nwzpzdge.hcp.westus.azmk8s.io
PrivateFQDN             : aa-nonprd-scus-spk-aks-nwzpzdge.hcp.westus.azmk8s.io
AgentPoolProfiles       : {aksctrlplane, user1}
WindowsProfile          : Microsoft.Azure.Commands.Aks.Models.PSManagedClusterWindowsProfile
AddonProfiles           : {[azureKeyvaultSecretsProvider, 
                        Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAddonProfile], [azurepolicy, 
                        Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAddonProfile], 
                        [ingressApplicationGateway, 
                        Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAddonProfile], [omsagent, 
                        Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAddonProfile]}
NodeResourceGroup       : MC_aa-nonprd-scus-spk-aks-rg-7gmwo_aa-my-akso_westus
EnableRBAC              : True
EnablePodSecurityPolicy : 
NetworkProfile          : Microsoft.Azure.Commands.Aks.Models.PSContainerServiceNetworkProfile
AadProfile              : Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAadProfile
ApiServerAccessProfile  : Microsoft.Azure.Commands.Aks.Models.PSManagedClusterAPIServerAccessProfile
IdentityProfile         : {[kubeletidentity, 
                        Microsoft.Azure.Commands.Aks.Models.PSManagedClusterPropertiesIdentityProfile]}
Identity                : Microsoft.Azure.Commands.Aks.Models.PSManagedClusterIdentity
LinuxProfile            : 
ServicePrincipalProfile : Microsoft.Azure.Commands.Aks.Models.PSContainerServiceServicePrincipalProfile
ResourceGroupName       : aa-nonprd-scus-spk-aks-rg-7gmwo
Id                      : /subscriptions/2cf2f1ee-99712e83048/resourcegroups/aa-nonprd-scus-spk-aks-rg-7gm
                        wo/providers/Microsoft.ContainerService/managedClusters/aa-my-akso
Name                    : aa-my-akso
Type                    : Microsoft.ContainerService/ManagedClusters
Location                : westus
Tags                    : {[AppEnv, nonprd], [Apppoc, me@domain.com], [CC, 113], [Creation_Time, 
                        2023-03-16_07:52:27 PST]...}


Powerstate:

  3. So next, I thought, maybe you have to pipe to Get-AzResource by passing the ResourceId and then use that to get the power state:

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

    $filteredClusters = @{}

Foreach ($Cluster in $Clusters )
{
      
   $currentCluster = Get-AzResource -ResourceId $($Cluster.Id)
   $powerstate = $($currentCluster.Properties.Powerstate.Code)
   $provisioning = $($Cluster.ProvisioningState)
   $sched = $($Cluster.Tags.Schedule)

   Write-Output "Processing $($Cluster.Name) ..."
   Write-Output "Resource ID: $($Cluster.Id) "
   Write-Output "Power State: $($Cluster.Powerstate.Code) "
   Write-Output "Provisioning State: $($Cluster.ProvisioningState) "
   Write-Output "Schedule Tag: $($Cluster.Tags.Schedule)"

   If ( $powerstate -eq "Running" -and $provisioning -eq "Succeeded" -and $sched -eq "adhoc")
   {
      Write-Output "Adding cluster to filtered clusters list:  $($Cluster.Name), $($Cluster.ResourceGroupName) "
      $filteredClusters.Add( $($Cluster.Name), $($Cluster.ResourceGroupName) )
   }
}
    • The above was trimmed a little to show the line $currentCluster = Get-AzResource -ResourceId $($Cluster.Id). This produced an error:
    1

    Get-AzResource : No registered resource provider found for location 'westus' and API version '2023-08-02-preview' for type 'managedClusters'. The supported api-versions are '2017-08-31, 2018-03-31, 2018-08-01-preview, 2019-02-01, 2019-04-01, 2019-06-01, 2019-08-01, 2019-10-01, 2019-11-01, 2020-01-01, 2020-02-01, 2020-03-01, 2020-04-01, 2020-06-01, 2020-07-01, 2020-09-01, 2020-11-01, 2020-12-01, 2021-02-01, 2021-03-01, 2021-05-01, 2021-07-01, 2021-08-01, 2021-09-01, 2021-10-01, 2021-11-01-preview, 2022-01-01, 2022-01-02-preview, 2022-02-01, 2022-02-02-preview, 2022-03-01, 2022-03-02-preview, 2022-04-01, 2022-04-02-preview, 2022-05-02-preview, 2022-06-01, 2022-06-02-preview, 2022-07-01, 2022-07-02-preview, 2022-08-01, 2022-08-02-preview, 2022-08-03-preview, 2022-09-01, 2022-09-02-preview, 2022-10-02-preview, 2022-11-01, 2022-11-02-preview, 2023-01-01, 2023-01-02-preview, 2023-02-01, 2023-02-02-preview, 2023-03-01, 2023-03-02-preview, 2023-04-01, 2023-04-02-preview, 2023-05-01, 2023-05-02-preview, 2023-06-01, 2023-06-02-preview, 2023-07-01, 2023-07-02-preview'. The supported locations are 'australiacentral, australiacentral2, australiaeast, australiasoutheast, brazilsouth, brazilsoutheast, canadacentral, canadaeast, centralindia, centralus, eastasia, eastus, eastus2, francecentral, francesouth, germanynorth, germanywestcentral, japaneast, japanwest, jioindiacentral, jioindiawest, koreacentral, koreasouth, northcentralus, northeurope, norwayeast, norwaywest, polandcentral, qatarcentral, southafricanorth, southafricawest, southcentralus, southindia, southeastasia, swedencentral, switzerlandnorth, switzerlandwest, uaecentral, uaenorth, uksouth, ukwest, westcentralus, westeurope, westus, westus2, westus3'. StatusCode: 400 ReasonPhrase: Bad Request OperationID : a19e3844-c665-4246-9a2d-ba9890f9087b At line:38 char:22 + $currentCluster = Get-AzResource -ResourceId $($Cluster.Id) + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : CloseError: (:) [Get-AzResource], ResourceManagerCloudException + FullyQualifiedErrorId : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.GetAzureResourceCmdlet

  4. This is frustrating! Why can’t the Get-Azresource cmdlet work with ResourceID, it is the main purpose of the cmdlet! So then I did some digging to see why the Get-AzAKSCluster cmdlet can’t get the powerstate on its own. Well the module loaded in the Automation account is 4.1.0 per the Modules blade of the Automation Account.

    • So I then went to its source and downloaded the nupkg and extracted. Unfortuntely everything is in dll’s so I can’t view.
    • I then wanted to look online to see what version the Automation Account returns when you run the Get-AzAks resource.
    • So what you do is scroll down on the cmdlet page and go to the outputs section and you will see PSKubernetesCluster
    • Next, you change the drop down on the left to Azure - Powershell Commands => 5.9.0. This seems to coorelate to Powershell 5 which matches the runtime of 5.1.
    • Now you can see where we get ProvisioningState and Tags but no power state.
    • Mystery solved. So how do we get the Power state?

  5. So I went back to the Get-AzResource and found that I got the same thing on my machine when I passed in a ResourceId.

    1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

    Foreach ($Cluster in $Clusters )
{
   $currentCluster = Get-AzResource -ResourceId $($Cluster.Id)
   $powerstate = $($currentCluster.Powerstate.Code)
   $provisioning = $($currentCluster.ProvisioningState)
   $sched = $($currentCluster.Tags.Schedule)

   Write-Output "Processing $($Cluster.Name) ..."
   Write-Output "Resource ID: $currentCluster "
   Write-Output "Power State: $powerstate "
   Write-Output "Provisioning State: $provisioning "
   Write-Output "Schedule Tag: $sched"

   If ( $powerstate -eq "Running" -and $provisioning -eq "Succeeded" -and $sched -eq "adhoc")
   {
      Write-Output "Adding cluster to filtered clusters list:  $($Cluster.Name), $($Cluster.ResourceGroupName) "
      $filteredClusters.Add( $($Cluster.Name), $($Cluster.ResourceGroupName) )
   }
}
Processing aa-my-akso ...
Resource ID: Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.PSResource
Power State:
Provisioning State:
Schedule Tag: adhoc
    • Hmm, why am I getting the class Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkModels.PSResource instead of the actual resource id?
    • So then I decided to try Get-AzResource -Name $($Cluster.Name) -ResourceGroupName $($Cluster.ResourceGroupName) and that seemed to work. I got back an object. But when I checked its properties, it was null.

  6. So then I googled “No properties returned for get-azresource” and found this post.

    • The answer was to pass an -ExpandProperties switch to the cmdlet. Yeah right I thought, this Automation Account cmdlet is probably running an ancient version that doesn’t support that switch but ran it anyways.
    • To my amazement, it worked! So there you have it, a single little switch allows us to the get the Powerstate which we can then plug in to our script and now we can shut down/start up AKS clusters.

Comments

You May Also Enjoy