Jekyll2024-01-13T22:07:18-06:00https://automationadmin.com/feed.xmlAutomation AdminCloud engineering with Github, Azure, Terraform, Powershell and other Automation technologiesgerryw1389Update To Building Jekyll Locally2023-11-23T23:24:00-06:002023-11-23T23:24:00-06:00https://automationadmin.com/2023/11/update-build-local<!--more-->
<h3 id="description">Description</h3>
<p>It’s been a while since I have updated my website, so I went through these steps as of 2023-11 to build/test locally:</p>
<h3 id="steps">Steps</h3>
<ol>
<li>
<p>First, new computer so I had to install <code class="language-text highlighter-rouge">wsl</code>:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre>wsl <span class="nt">--install</span> <span class="nt">-d</span> Ubuntu-22.04
<span class="nb">sudo</span> <span class="nt">--</span> sh <span class="nt">-c</span> <span class="s1">'apt-get update; apt-get upgrade -y; apt-get dist-upgrade -y; apt-get autoremove -y; apt-get autoclean -y'</span>
<span class="nb">sudo </span>apt-get <span class="nb">install </span>ruby-full build-essential zlib1g-dev
<span class="nb">echo</span> <span class="s1">'# Install Ruby Gems to ~/gems'</span> <span class="o">>></span> ~/.bashrc
<span class="nb">echo</span> <span class="s1">'export GEM_HOME="$HOME/gems"'</span> <span class="o">>></span> ~/.bashrc
<span class="nb">echo</span> <span class="s1">'export PATH="$HOME/gems/bin:$PATH"'</span> <span class="o">>></span> ~/.bashrc
<span class="nb">source</span> ~/.bashrc
gem <span class="nb">install </span>jekyll bundler
bundle <span class="nb">install
</span>bundle <span class="nb">exec </span>jekyll serve
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>This produced:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
</pre></td><td class="rouge-code"><pre>Configuration file: /mnt/c/_gwill/repo-home/h1website/_config.yml
Source: /mnt/c/_gwill/repo-home/h1website
Destination: /mnt/c/_gwill/repo-home/h1website/_site
Incremental build: disabled. Enable with --incremental
Generating...
Remote Theme: Using theme mmistakes/minimal-mistakes
Jekyll Feed: Generating feed for posts
GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
done in 104.025 seconds.
Auto-regeneration: disabled. Use --watch to enable.
/home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve/servlet.rb:3:in `require': cannot load such file -- webrick (LoadError)
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve/servlet.rb:3:in `<top (required)>'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:184:in `require_relative'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:184:in `setup'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:102:in `process'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:93:in `block in start'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:93:in `each'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:93:in `start'
from /home/gerry/gems/gems/jekyll-3.9.3/lib/jekyll/commands/serve.rb:75:in `block (2 levels) in init_with_program'
from /home/gerry/gems/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `block in execute'
from /home/gerry/gems/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `each'
from /home/gerry/gems/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `execute'
from /home/gerry/gems/gems/mercenary-0.3.6/lib/mercenary/program.rb:42:in `go'
from /home/gerry/gems/gems/mercenary-0.3.6/lib/mercenary.rb:19:in `program'
from /home/gerry/gems/gems/jekyll-3.9.3/exe/jekyll:15:in `<top (required)>'
from /home/gerry/gems/bin/jekyll:25:in `load'
from /home/gerry/gems/bin/jekyll:25:in `<main>'
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>So then I googled and found this <a href="https://stackoverflow.com/questions/69890412/bundler-failed-to-load-command-jekyll">Stack Overflow</a> post with the answer:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre>gerry@gw-host:/mnt/c/_gwill/repo-home/h1website<span class="nv">$ </span>bundle add webrick
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Fetching webrick 1.8.1
Installing webrick 1.8.1
gerry@gw-host:/mnt/c/_gwill/repo-home/h1website<span class="nv">$ </span>bundle <span class="nb">exec </span>jekyll serve
Configuration file: /mnt/c/_gwill/repo-home/h1website/_config.yml
Source: /mnt/c/_gwill/repo-home/h1website
Destination: /mnt/c/_gwill/repo-home/h1website/_site
Incremental build: disabled. Enable with <span class="nt">--incremental</span>
Generating...
Remote Theme: Using theme mmistakes/minimal-mistakes
Jekyll Feed: Generating feed <span class="k">for </span>posts
GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
<span class="k">done in </span>98.374 seconds.
Auto-regeneration may not work on some Windows versions.
Please see: https://github.com/Microsoft/BashOnWindows/issues/216
If it does not work, please upgrade Bash on Windows or run Jekyll with <span class="nt">--no-watch</span><span class="nb">.</span>
Auto-regeneration: enabled <span class="k">for</span> <span class="s1">'/mnt/c/_gwill/repo-home/h1website'</span>
Server address: http://127.0.0.1:4000
Server running... press ctrl-c to stop.
^Cgerry@gw-host:/mnt/c/_gwill/repo-home/h1website<span class="nv">$ </span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>That’s it! Surprisingly not to much to test locally.</p>
</li>
</ol>gerryw1389Create PayAsYouGo Azure Subs Using Terrform2023-11-22T23:24:00-06:002023-11-22T23:24:00-06:00https://automationadmin.com/2023/11/create-tf-subscriptions-payg<!--more-->
<h3 id="description">Description</h3>
<p>One of the first things I wanted to show after <a href="https://automationadmin.com/2023/08/setting-up-github-org">creating an organization</a> in Github is how to configure Azure completely via IaC, with little to no clicking in the GUI. Well the first step to do that is to create subscriptions. I was certain that you could not do that with “Pay as You Go” types, but sure enough, it’s <a href="https://registry.terraform.io/providers/hashicorp/azurerm/3.80.0/docs/resources/subscription#example-usage---creating-a-new-alias-and-subscription-for-a-microsoft-customer-account">one of the options</a>!</p>
<p class="notice--success">Note: You can see the code for this post on <a href="https://github.com/AutomationAdmin-Com/sic.mgmt">my Github repo</a>.</p>
<h3 id="steps">Steps</h3>
<ol>
<li>
<p>The first thing I did was give my Service Principal rights to create subscriptions at the correct “Invoice Section” level:</p>
<ul>
<li>
<p><img src="https://automationadmin.com/assets/images/uploads/2023/11/subscription-creator.png" alt="subscription-creator" class="img-responsive" /></p>
</li>
<li>
<p>It can be confusing the hierarchy, but thankfully the <a href="https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/view-all-accounts#microsoft-customer-agreement">docs</a> cover this well. It’s just Billing Account => Profile => Invoice Section.</p>
</li>
</ul>
</li>
<li>
<p>So before using Terraform to create new subscriptions, you have to get the <code class="language-text highlighter-rouge">id</code> property of your Billing Account, Billing Profile, and Invoice Section. This tripped me up so bad because the Terraform docs say <code class="language-text highlighter-rouge">billing_name</code> as in the <code class="language-text highlighter-rouge">name</code> property so I don’t have to expose the <code class="language-text highlighter-rouge">id</code> for the world to see, but <strong>NOPE</strong>, that won’t work! It has to be the <code class="language-text highlighter-rouge">id</code>. So confusing! In hindsight looking at the example does show ID values instead of name values.</p>
<ul>
<li>Anyhow, once you get them, just <a href="https://github.com/AutomationAdmin-Com/sic.mgmt/blob/facf9eeb7dc99966174b83cc230868eaf1c86b0b/source/common/stage1/sub.tf#L2">plug them in to your data source</a> and then run your apply.</li>
</ul>
</li>
<li>
<p>What happens is, the first two subscriptions created almost instantly, but the third one timed out. I can’t find the logs or I would show you :/</p>
</li>
<li>
<p>I already knew the fix for this from dealing with it in the past, basically you just have to <strong>wait</strong>. This is because Microsoft doesn’t want customers spinning up hundreds of subscriptions per hour because a subscription is a high level billing resource that holds many things, pretty much the highest level you go on a daily basis so they throttle it. Anyways, I was going to wait 24 hours, but I just re-ran the same pipeline about 12 hours later and it worked.</p>
<ul>
<li><img src="https://automationadmin.com/assets/images/uploads/2023/11/subscription-created.png" alt="subscription-created" class="img-responsive" /></li>
</ul>
</li>
<li>
<p>So there you go, you can create PAYG subscriptions using Terraform pretty easily these days. In the past, <a href="https://automationadmin.com/2022/10/tf-new-subscription">I always did these by hand</a> and then used <a href="https://github.com/gerryw1389/terraform-examples/blob/main/2022-10-20-tf-new-subscription/main.tf">terraform to place them in correct management groups</a> but now I can control the subscription objects themselves.</p>
</li>
</ol>gerryw1389Vscode Az Module Register Issue2023-11-18T23:24:00-06:002023-11-18T23:24:00-06:00https://automationadmin.com/2023/11/vscode-az-module-register<!--more-->
<h3 id="description">Description:</h3>
<p>When using Windows Powershell 5.1 and only in VSCode, running <code class="language-text highlighter-rouge">import-module az</code> results in many errors like:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre><span class="w"> </span><span class="n">Register-AzModule</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="nx">The</span><span class="w"> </span><span class="nx">type</span><span class="w"> </span><span class="nx">initializer</span><span class="w"> </span><span class="nx">for</span><span class="w"> </span><span class="s1">'Microsoft.Azure.Commands.Common.AzModule'</span><span class="w"> </span><span class="nx">threw</span><span class="w"> </span><span class="nx">an</span><span class="w"> </span><span class="nx">exception.</span><span class="w">
</span><span class="n">At</span><span class="w"> </span><span class="nx">C:\Users\myuser\Documents\WindowsPowerShell\Modules\Az.Advisor\2.0.0\Az.Advisor.psm1:49</span><span class="w"> </span><span class="nx">char:13</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>
<p>Doesn’t seem to happen with native powershell console.</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>I followed <a href="https://github.com/PowerShell/vscode-powershell/issues/4594">this issue</a> and <a href="https://github.com/Azure/azure-powershell/issues/21647">this issue</a> to:</p>
<ul>
<li>Delete all AZ folders in my user Powershell Modules directory.</li>
<li>
<p>Download and install again => <code class="language-text highlighter-rouge">Install-Module -Name Az -RequiredVersion 9.6.0</code></p>
</li>
<li>Then download the nupkg for Az.Accounts for <a href="https://www.powershellgallery.com/packages/Az.Accounts/2.12.1"><code class="language-text highlighter-rouge">2.12.1</code></a></li>
<li>Extract to the Az.Accounts subfolder under the version number.</li>
</ul>
</li>
<li>
<p>Now when I run <code class="language-text highlighter-rouge">Import-module az</code>, it works like in the past. My guess is this will continue to be an issue in the future and should probably switch to Powershell 7.</p>
</li>
</ol>gerryw1389Github Actions: Using AKV To Get Secrets2023-11-05T00:24:00-05:002023-11-05T00:24:00-05:00https://automationadmin.com/2023/11/using-akv-to-get-secrets<!--more-->
<h3 id="description">Description:</h3>
<p>A key goal I wish to accomplish with Github Actions is to set all the secrets I can in an Azure Key Vault and then only get them at run time to populate all the other secrets needed. I have been doing this for years with <a href="https://automationadmin.com/2021/01/function-apps-get-secrets/">Function Apps</a> and other Azure services and wish to continue this strategy. Basically, the strategy works like this: <code class="language-text highlighter-rouge">"Only store Azure Credentials as Env secrets and use those at runtime to populate more secrets"</code>. This way you can create a Service Principal with limited rights like <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#key-vault-secrets-user">“Key Vault Secrets User”</a> at the AKV level and then monitor it in Azure Active Directory to ensure it is only being used to access AKVs. Here is how I do just this:</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>The only secrets needed for <a href="https://automationadmin.com/2023/08/setting-oidc-auth">OIDC Auth</a> are <code class="language-text highlighter-rouge">${/{ secrets.CLIENT_ID }}</code>, <code class="language-text highlighter-rouge">${/{ secrets.TENANT_ID }}</code>, and <code class="language-text highlighter-rouge">${/{ secrets.SUB_ID }}</code> so I put each of these in the Repo as secrets.</p>
<ul>
<li>
<p>NOTE: <a href="https://jekyllrb.com/docs/liquid/filters/">Jekyll Liquid Filters</a> clash with <a href="https://docs.github.com/en/actions/learn-github-actions/variables#using-contexts-to-access-variable-values">Github Variables</a> so replace all instances of <code class="language-text highlighter-rouge">${/{</code> by removing the forward slash :)</p>
</li>
<li>
<p>Then I need to add <code class="language-text highlighter-rouge">${/{ secrets.REPO_BOT_PEM }}</code> as discussed <a href="https://automationadmin.com/2023/07/create-repo-bot-for-tf-modules">here</a> for access to my module repos.</p>
</li>
<li>
<p>I would have that one come from a Key Vault, but I had issues reading secrets from AKV that are private keys as explained <a href="https://automationadmin.com/2023/09/unable-to-load-priv-key-from-akv">here</a>.</p>
</li>
</ul>
</li>
<li>
<p>Once those secrets are added, we then just need to do 2 things: Populate our AKV with all possible secrets and then add a task in our pipeline that will update the worfklow and “switch” based on the <strong>needed</strong> variables.</p>
</li>
<li>
<p><a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L69">Here</a> is the action that will switch based on the specific workflow :</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Parse</span><span class="nv"> </span><span class="s">Workflow</span><span class="nv"> </span><span class="s">TF</span><span class="nv"> </span><span class="s">Folder</span><span class="nv"> </span><span class="s">From</span><span class="nv"> </span><span class="s">Matrix"</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">parse</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">cd $GITHUB_WORKSPACE</span>
<span class="s">chmod +x ./.github/scripts/parse.sh</span>
<span class="s">./.github/scripts/parse.sh</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">CURRENT_DIRECTORY</span><span class="pi">:</span> <span class="s">${/{ matrix.directories }}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>And <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/main/.github/scripts/parse.sh">here</a> is the script.</li>
</ul>
</li>
<li>
<p>You will notice a few important things about this script.</p>
<ul>
<li>First, it makes use of <a href="https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs#overview">Github Outputs</a> context which is critical to how workflows work in Github Actions.</li>
<li>I used to think it only worked with Inline bash so I would have long workflows but I found that shell scripts will inherit the <code class="language-text highlighter-rouge">$GITHUB_OUTPUT</code> <a href="https://docs.github.com/en/actions/learn-github-actions/variables#default-environment-variables">env var</a>.</li>
<li>Second, the script mostly works by looking at the current matrix item and then setting outputs based on the value of it. Just a like a powershell switch statement but using bash.</li>
</ul>
</li>
<li>Lastly, after the parsing script, you just reference any output key value pair by the <code class="language-text highlighter-rouge">key's name</code> in subsequent steps. For example:
<ul>
<li><code class="language-text highlighter-rouge">TF_VAR_subscription_id: ${/{ steps.azure-keyvault-secrets-spoke.outputs.spoke-subscription-id }}</code></li>
<li><code class="language-text highlighter-rouge">TF_VAR_hub_subscription_id: ${/{ steps.azure-keyvault-secrets-hub.outputs.hub-subscription-id }}</code></li>
<li>In these examples the steps name is <code class="language-text highlighter-rouge">azure-keyvault-secrets</code> as seen <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L112">here</a> and the secrets we are setting are <code class="language-text highlighter-rouge">spoke-subscription-id</code> and <code class="language-text highlighter-rouge">hub-subscription-id</code> as seen in various spots of the parse script above ( look for any lines with <code class="language-text highlighter-rouge">>>$GITHUB_OUTPUT</code> ).</li>
</ul>
</li>
<li>
<p>In the previous step we get the subscription ID and hub subscription ID needed dynamically so that we can <a href="https://github.com/AutomationAdmin-Com/sic.mgmt/blob/4ad7fee18f3032ad14d011affb69e3fcb44c4498/config/prd/hub/scus/stage1/none/backend.tf#L61">build providers</a> in our calling workflows. See my <a href="https://automationadmin.com/lab/">lab</a> section for how this works.</p>
</li>
<li>
<p>Also, I have recently added a random <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L101">up to 45 second delay</a> because I might have multiple parrallel workflows running and the AKV needs to only whitelist at run time so it would often error out with ‘(Forbidden) Client address is not authorized’ like so:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre>Run az account set --subscription "prd-hub"
ERROR: (Forbidden) Client address is not authorized and caller is not a trusted service.
Client address: 20.81.159.17
Caller: appid=***;oid=fe62cc9a-b71b-46ae-93b3-d154327f57a4;iss=https://sts.windows.net/***/
Vault: aa-prd-scus-hub-akv-v1;location=southcentralus
Code: Forbidden
Message: Client address is not authorized and caller is not a trusted service.
Client address: 20.81.159.17
Caller: appid=***;oid=fe62cc9a-b71b-46ae-93b3-d154327f57a4;iss=https://sts.windows.net/***/
Vault: aa-prd-scus-hub-akv-v1;location=southcentralus
Inner error: {
"code": "ForbiddenByFirewall"
}
Error: Process completed with exit code 1.
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>The fix is to simply click the <code class="language-text highlighter-rouge">rerun failed jobs</code> button. This happens due to concurrency that the 45 second delay is supposed to fix.</li>
</ul>
</li>
</ol>gerryw1389Main Terraform Workflow Repo Structure2023-10-22T00:24:00-05:002023-10-22T00:24:00-05:00https://automationadmin.com/2023/10/main-terraform-workflow-repo-structure<!--more-->
<h3 id="description">Description</h3>
<p>Continuing from my <a href="https://automationadmin.com/2023/05/main-terraform-workflow">previous post</a>, I wanted to go into more details about my repo structure because it is highly versatile and worth explaining in my opinion.</p>
<p class="notice--success">Note: You can see the code for this post on <a href="https://github.com/AutomationAdmin-Com/sic.template/tree/main">my Github repo</a>.</p>
<h3 id="steps">Steps</h3>
<ol>
<li>
<p>This is the folder as it exists in Github and on the Github Runner right at the <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L27">checkout stage</a> before any modifications.</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
</pre></td><td class="rouge-code"><pre>C:.
│ README.md
│
├───.github
│ └───workflows => This is where all pipelines are created. Github calls these "workflows"
│ main_protector.yaml => These are workflow files covered in other posts.
│ main.yml
│
├───config => This folder exists to make changes for a specific environment that is based on folder path. We call these "configs" because each dir has its own state file based on paths.
│ └───nonprd => Environment based, nonprd or prd. Could also add 'uat', 'maint', or some other stage by copying/pasting and changing the variables.tf and/or terraform.tfvar values underneath.
│ └───spoke => Create one folder for as many Azure subscriptions that you want to switch on.
│ └───scus => Create one folder for as many regions as you want to deploy to.
│ └───stage1 => We need to create one folder for each stage. For example, it's common to create an AKV in stage1 and then read in a cert from that AKV in stage2.
│ ├───blue => Last, create a sub folder for blue, green, or none. This is really only for deploying services that will do "blue green cutovers".
│ │ backend.tf => State file location as well as building providers from the secrets from a Key Vault being passed to terraform by workflow.
│ │ terraform.tfvars => Any environment specific vars.
│ │ variables.tf => Global variable definitions for this stage.
│ │
│ └───green
│ backend.tf
│ terraform.tfvars
│ variables.tf
│ └───prd => Same tree as above but for prod instead of non-prod.
│ └───spoke
│ └───eus
│ └───stage1
│ ├───blue
│ │ backend.tf
│ │ terraform.tfvars
│ │ variables.tf
│ │
│ └───none
│ backend.tf
│ terraform.tfvars
│ variables.tf
│
└───source => These are static files that should be like modules where almost all values are vars being passed in from above. Avoid hard coding anything here as it will apply to all environments!
├───common
│ └───stage0 => These should define resources that goes in stage0 for all subscription, regions, and environments.
│ rg.tf
│
└───modules => These are local module calls you can make at run time. During execution, our worflow will copy these files recursively to the `./live/*` folder.
└───rand
random_string.tf
variables.tf
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Now that we have these files in the repo and have explained their purpose, let’s examine what happens at run time. There is a <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L78">critical step</a> where we copy the files in a specific way:</p>
<ul>
<li>First, we create a folder called <code class="language-text highlighter-rouge">./live</code> that exists on the Github Runner only during execution, it does not exist anywhere in our stored repo.</li>
<li>
<p>Next, since we are using a <a href="https://automationadmin.com/2023/05/main-terraform-workflow">matrix workflow</a>, this specific Github run will be running in parrallel with <strong>all</strong> the changes you made for this Pull Requests.</p>
</li>
<li>
<p>This means we could be running one, two, or twenty parrallel exucutions but each one with a specific <code class="language-text highlighter-rouge">${/{ matrix.directories }}</code> value that coorelates to something like <code class="language-text highlighter-rouge">config/nonprd/hub/east/stage2/none</code> or <code class="language-text highlighter-rouge">config/prd/hub/east/stage2/none</code> or <code class="language-text highlighter-rouge">config/prd/hub/scus/stage2/none</code> for example.</p>
</li>
<li>
<p>NOTE: <a href="https://jekyllrb.com/docs/liquid/filters/">Jekyll Liquid Filters</a> clash with <a href="https://docs.github.com/en/actions/learn-github-actions/variables#using-contexts-to-access-variable-values">Github Variables</a> so replace all instances of <code class="language-text highlighter-rouge">${/{</code> by removing the forward slash :)</p>
</li>
<li>
<p>Next, we have a <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/main/.github/scripts/parse.sh">parse script</a> that is simply a bash script that looks at those paths and creates outputs dynamically as <a href="https://automationadmin.com/2023/11/using-akv-to-get-secrets">explained in my post here</a> to create outputs. We are just getting the stage number in this case.</p>
</li>
<li>Next, we recursively copy all files under <code class="language-text highlighter-rouge">./source/modules</code> so that all local module calls will be <code class="language-text highlighter-rouge">./modules/$moduleName</code> as seen in <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/source/common/stage1/rg.tf#L23C26-L23C40"><code class="language-text highlighter-rouge">rg.tf</code></a></li>
</ul>
</li>
<li>
<p>Here is the task in the pipeline that will modify file placement so that terraform can run in a single directory:</p>
</li>
<li>
<p>This is the folder as it exists during a workflow execution <strong>AFTER</strong> the Copy Files task:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
</pre></td><td class="rouge-code"><pre>C:.
│ README.md
│
├───.github
│ └───workflows
│ main_protector.yaml
│ main.yml
├───config
│ └───nonprd
│ └───spoke
│ └───eus
│ └───stage1
│ ├───none
│ │ backend.tf
│ │ terraform.tfvars
│ │ variables.tf
├───live => This is a brand new directory created at run time that contains all files needed for Terraform to run.
│ │ backend.tf => This came from config/nonprd/spoke/eus/stage1/none
│ │ rg.tf => This came from the source/common/stage1
│ │ terraform.tfvars => This came from config/nonprd/spoke/eus/stage1/none
│ │ variables.tf => This came from config/nonprd/spoke/eus/stage1/none
│ │
│ └───modules => This got copied from source/modules/rand. Files like `./rg.tf` above reference these locally like `./modules/moduleName
│ └───rand
│ random_string.tf
│ variables.tf
│
└───source
└───common
└───stage1
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>You then see in <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L163">subsequent steps</a> that we continue to <code class="language-text highlighter-rouge">cd $GITHUB_WORKSPACE/live</code> before running any terraform commands.</p>
</li>
<li>
<p>That’s it! So really this workflow solves so many problems and allows a large enterprise to use Terraform effectively:</p>
<ul>
<li>You can deploy to multiple Azure subscriptions in a single run. Example: “I need to deploy a key vault to prd-hub in the southcentralus region and I need to do the same thing to nonprd-spoke in the eastus region”. Done, just:
<ul>
<li>Modify the <code class="language-text highlighter-rouge">variables.tf</code> by adding a line break in <code class="language-text highlighter-rouge">config/prd/hub/scus/stage1/none</code> folder and then ensure that you are creating a Key Vault in <code class="language-text highlighter-rouge">./source/common/stage1/akv.tf</code> for example.</li>
<li>Modify the <code class="language-text highlighter-rouge">variables.tf</code> by adding a line break in <code class="language-text highlighter-rouge">config/nonprd/spoke/eus/stage1/none</code> folder and then ensure that you are creating a Key Vault in <code class="language-text highlighter-rouge">./source/common/stage1/akv.tf</code> for example.</li>
</ul>
</li>
<li>
<p>Each deployment will have a separate state file as seen in <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/config/prd/hub/scus/stage1/none/backend.tf#L8"><code class="language-text highlighter-rouge">backend.tf</code></a> and <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/config/nonprd/spoke/east/stage1/none/backend.tf#L8"><code class="language-text highlighter-rouge">backend.tf here</code></a></p>
</li>
<li>You could easily modify the workflow for a specific run, like adding a new variable for example, by changing <code class="language-text highlighter-rouge">./.github/workflows/main.yml</code> to <code class="language-text highlighter-rouge">workflow dispatch</code> and then creating a new file with similar contents that runs based on a specific path <a href="https://automationadmin.com/2023/05/main-terraform-workflow">like discussed in previous post</a> of how I used to do it. For example:</li>
</ul>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">develop"</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">config/nonprd/hub/east/stage1/none/*"</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">types</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">opened</span><span class="pi">,</span> <span class="nv">edited</span><span class="pi">,</span> <span class="nv">synchronize</span><span class="pi">]</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">develop"</span>
<span class="na">paths</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">config/nonprd/hub/east/stage1/none/*/*"</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>
<p>You could easily expand this template to have new stages, new subscriptions, new environments, or anything really. The magic lies in the parsing script that sets outputs dynamically, so be sure to read <a href="https://automationadmin.com/2023/11/using-akv-to-get-secrets">my post covering this</a> to get an idea for your organization.</p>
</li>
<li>
<p>I’m sure there are other perks but overall this template is very powerful as I have ran it hundreds of times for hundreds of scenarios!</p>
</li>
</ul>
</li>
</ol>gerryw1389Using Deploy Keys For Multiple TF Module Private Repos2023-10-08T00:24:00-05:002023-10-08T00:24:00-05:00https://automationadmin.com/2023/10/using-deploy-keys-for-multiple-tf-module-repos<!--more-->
<h3 id="description">Description:</h3>
<p>In Gihub Actions, you have two main ways you can call multiple Terraform Module private repos at run time, I have only documented and tested two ways:</p>
<ul>
<li>Using Deploy Keys (this post)</li>
<li>Using a Github App as documented <a href="https://automationadmin.com/2023/07/create-repo-bot-for-tf-modules">here</a></li>
</ul>
<p>Here I will show you how I used Deploy Keys for multiple TF Module repos.</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>Create <a href="https://github.com/AutomationAdmin-Com/module.rg">your module repo</a> and reference it like: <code class="language-text highlighter-rouge">git::ssh://git@github.com/AutomationAdmin-Com/module.rg.git?ref=v0.0.1</code> from your calling repo.</p>
<ul>
<li>Pay special attention to the <code class="language-text highlighter-rouge">/</code> after <code class="language-text highlighter-rouge">github.com</code> which is different from the ssh address you will get if you navigate to the repo and copy the <code class="language-text highlighter-rouge">ssh clone</code> command.</li>
<li>See <a href="https://developer.hashicorp.com/terraform/language/modules/sources#generic-git-repository">tf source</a> for reference.</li>
</ul>
</li>
<li>
<p>Next, we need to generate a SSH key pair and use that for the Github Agent to pull from whichever repos it needs. See <a href="https://breadnet.co.uk/terraform-init-on-github-actions-with-private-modules/?mtm_campaign=github&mtm_kwd=terraform-actions-33">here</a> for the guide I followed: <code class="language-text highlighter-rouge">ssh-keygen -t ed25519</code></p>
</li>
<li>
<p>Next, take the public portion and upload it under AutomationAdmin-Com/module.rg => Settings => Security/Deploy Keys => +Add => Insert here with name <code class="language-text highlighter-rouge">githubactions-terraform</code></p>
</li>
<li>Next, take the private portion and upload it as a secret under AutomationAdmin-Com/sic.mgmt => Settings => Security/Secrets and Variables/Actions => +New Repository Secret => <code class="language-text highlighter-rouge">SSH_KEY_GITHUB_ACTIONS</code>
<ul>
<li>Copy the full multi line private key into the secret.</li>
</ul>
</li>
<li>
<p>Lastly, you just need the public runner to be able to access those repos so you add this before the terraform init step:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="rouge-code"><pre><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Setup</span><span class="nv"> </span><span class="s">SSH</span><span class="nv"> </span><span class="s">Keys</span><span class="nv"> </span><span class="s">and</span><span class="nv"> </span><span class="s">known_hosts'</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">SSH_AUTH_SOCK</span><span class="pi">:</span> <span class="s">/tmp/ssh_agent.sock</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">ssh-agent -a $SSH_AUTH_SOCK > /dev/null</span>
<span class="s">ssh-add - <<< "${/{ secrets.SSH_KEY_GITHUB_ACTIONS }}"</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Terraform</span><span class="nv"> </span><span class="s">Init"</span>
<span class="na">id</span><span class="pi">:</span> <span class="s">init</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">cd $GITHUB_WORKSPACE/live</span>
<span class="s">terraform init -backend-config="access_key=${/{ steps.azure-keyvault-secrets.outputs.tfstateaccesskey }}"</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">SSH_AUTH_SOCK</span><span class="pi">:</span> <span class="s">/tmp/ssh_agent.sock</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>I found this method works by following <a href="https://www.webfactory.de/blog/use-ssh-key-for-private-repositories-in-github-actions">this blog post</a> step-by-step. Thanks!</li>
</ul>
</li>
<li>
<p>There is one glaring issue with this though, what if you have <strong>multiple</strong> tf modules? It doesn’t work, both Github Actions and Terraform don’t know how to handle multiple SSH Keys if you just blindly add multiple SSH keys as secrets and try to keep adding them using the <code class="language-text highlighter-rouge">ssh-add</code> like above.</p>
<ul>
<li>Github <a href="https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#cons-of-deploy-keys">docs clearly say one key per repo</a></li>
<li>Terraform just uses <code class="language-text highlighter-rouge">git clone</code>, but with all addresses like <code class="language-text highlighter-rouge">git::ssh://git@github.com</code> it won’t know how to distinguish different repos with the same host name, even if the repo name is different.</li>
</ul>
</li>
<li>
<p>Thankfully, I found the fix for this! It makes sense in hindsight but anyways, it’s to setup and configure a SSH Config file on the public runner at run time and then reference your repos by different names. Here is an example config (NOTE: This assumes you have two modules, one for RG and one for CosmosDB and have uploaded their private keys as secrets to the repo) :</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Setup</span><span class="nv"> </span><span class="s">SSH</span><span class="nv"> </span><span class="s">Keys</span><span class="nv"> </span><span class="s">and</span><span class="nv"> </span><span class="s">known_hosts"</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">cd $GITHUB_WORKSPACE</span>
<span class="s">chmod +x ./.github/scripts/update_ssh_agent.sh</span>
<span class="s">./.github/scripts/update_ssh_agent.sh</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">SSH_AUTH_SOCK</span><span class="pi">:</span> <span class="s">/tmp/ssh_agent.sock</span>
<span class="na">SSH_KEY_MODULE_RG</span><span class="pi">:</span> <span class="s">${/{ secrets.ssh_key_module_rg }}</span>
<span class="na">SSH_KEY_MODULE_COSMOSDB</span><span class="pi">:</span> <span class="s">${/{ secrets.ssh_key_module_cosmosdb }}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>
<p>NOTE: <a href="https://jekyllrb.com/docs/liquid/filters/">Jekyll Liquid Filters</a> clash with <a href="https://docs.github.com/en/actions/learn-github-actions/variables#using-contexts-to-access-variable-values">Github Variables</a> so replace all instances of <code class="language-text highlighter-rouge">${/{</code> by removing the forward slash :)</p>
</li>
<li>
<p>Here is the script:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
</pre></td><td class="rouge-code"><pre> <span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-eu</span>
ssh-agent <span class="nt">-a</span> <span class="nv">$SSH_AUTH_SOCK</span> <span class="o">></span> /dev/null
<span class="nb">mkdir</span> <span class="nt">-p</span> ~/.ssh
<span class="nb">touch</span> ~/.ssh/config
<span class="nb">touch</span> ~/.ssh/known_hosts
ssh-keyscan github.com <span class="o">>></span> known_hosts
<span class="c"># Create key files from Repo Secrets:</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$SSH_KEY_MODULE_RG</span><span class="s2">"</span> <span class="o">></span> ~/.ssh/module_rg
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$SSH_KEY_MODULE_COSMOSDB</span><span class="s2">"</span> <span class="o">></span> ~/.ssh/module_cosmosdb
<span class="c"># Set perms on each:</span>
<span class="nb">chmod </span>600 ~/.ssh/module_rg ~/.ssh/module_cosmosdb ~/.ssh/known_hosts
<span class="c"># For each module, update SSH Config:</span>
<span class="nb">echo</span> <span class="s2">"Host module-rg"</span> <span class="o">>></span> ~/.ssh/config
<span class="nb">echo</span> <span class="s2">" Hostname github.com"</span> <span class="o">>></span> ~/.ssh/config
<span class="nb">echo</span> <span class="s2">" IdentityFile=~/.ssh/module_rg"</span> <span class="o">>></span> ~/.ssh/config
<span class="nb">echo</span> <span class="s2">"Host module-cosmosdb"</span> <span class="o">>></span> ~/.ssh/config
<span class="nb">echo</span> <span class="s2">" Hostname github.com"</span> <span class="o">>></span> ~/.ssh/config
<span class="nb">echo</span> <span class="s2">" IdentityFile=~/.ssh/module_cosmosdb"</span> <span class="o">>></span> ~/.ssh/config
<span class="c"># Add private key files to ssh-agent</span>
ssh-add ~/.ssh/module_rg
ssh-add ~/.ssh/module_cosmosdb
<span class="c"># Output to console in case of issues:</span>
<span class="nb">cat</span> ~/.ssh/config
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>You can test this is working by adding a few clone commands after it in the same script or a different one:</li>
</ul>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre> <span class="nb">echo</span> <span class="s2">"cloning..."</span>
<span class="nv">GIT_SSH_COMMAND</span><span class="o">=</span><span class="s2">"ssh -vv"</span> git clone git@module-rg:AutomationAdmin-Com/module.rg.git
<span class="nb">sleep </span>3
<span class="nb">echo</span> <span class="s2">"cloning second..."</span>
git clone git@module-cosmosdb:AutomationAdmin-Com/module.cosmosdb.git
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
</ul>
</li>
<li>
<p>Lastly, you have go to each <code class="language-text highlighter-rouge">source</code> in your terraform code and update them to match the module you are calling:</p>
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
</pre></td><td class="rouge-code"><pre><span class="k">module</span> <span class="s2">"cosmosdb_rg"</span> <span class="p">{</span>
<span class="nx">source</span> <span class="p">=</span> <span class="s2">"git@module-rg:AutomationAdmin-Com/module.rg.git?ref=v0.0.1"</span>
<span class="nx">resource_group_name</span> <span class="p">=</span> <span class="s2">"aa-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">env_stage_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">region_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">sub_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">stage</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">color</span><span class="k">}</span><span class="s2">-ga-rg"</span>
<span class="nx">location</span> <span class="p">=</span> <span class="kd">var</span><span class="p">.</span><span class="nx">region</span>
<span class="nx">tags</span> <span class="p">=</span> <span class="kd">local</span><span class="p">.</span><span class="nx">tags</span>
<span class="p">}</span>
<span class="k">module</span> <span class="s2">"cosmosdb_rg_2"</span> <span class="p">{</span>
<span class="nx">source</span> <span class="p">=</span> <span class="s2">"git@module-cosmosdb:AutomationAdmin-Com/module.cosmosdb.git?ref=sprint"</span>
<span class="nx">resource_group_name</span> <span class="p">=</span> <span class="s2">"aa-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">env_stage_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">region_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">sub_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">stage</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">color</span><span class="k">}</span><span class="s2">-ga-rg-2"</span>
<span class="nx">location</span> <span class="p">=</span> <span class="kd">var</span><span class="p">.</span><span class="nx">region</span>
<span class="nx">tags</span> <span class="p">=</span> <span class="kd">local</span><span class="p">.</span><span class="nx">tags</span>
<span class="p">}</span>
<span class="k">module</span> <span class="s2">"cosmosdb_rg_3"</span> <span class="p">{</span>
<span class="nx">source</span> <span class="p">=</span> <span class="s2">"git@module-cosmosdb:AutomationAdmin-Com/module.cosmosdb.git?ref=v0.0.2"</span>
<span class="nx">resource_group_name</span> <span class="p">=</span> <span class="s2">"aa-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">env_stage_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">region_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">sub_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">stage</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">color</span><span class="k">}</span><span class="s2">-ga-rg-3"</span>
<span class="nx">location</span> <span class="p">=</span> <span class="kd">var</span><span class="p">.</span><span class="nx">region</span>
<span class="nx">tags</span> <span class="p">=</span> <span class="kd">local</span><span class="p">.</span><span class="nx">tags</span>
<span class="p">}</span>
<span class="k">module</span> <span class="s2">"cosmosdb_rg_4"</span> <span class="p">{</span>
<span class="nx">source</span> <span class="p">=</span> <span class="s2">"git@module-rg:AutomationAdmin-Com/module.rg.git?ref=main"</span>
<span class="nx">resource_group_name</span> <span class="p">=</span> <span class="s2">"aa-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">env_stage_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">region_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">sub_abbr</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">stage</span><span class="k">}</span><span class="s2">-</span><span class="k">${</span><span class="kd">var</span><span class="p">.</span><span class="nx">color</span><span class="k">}</span><span class="s2">-ga-rg-4"</span>
<span class="nx">location</span> <span class="p">=</span> <span class="kd">var</span><span class="p">.</span><span class="nx">region</span>
<span class="nx">tags</span> <span class="p">=</span> <span class="kd">local</span><span class="p">.</span><span class="nx">tags</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>See the <code class="language-text highlighter-rouge">git@module-rg</code>? See how that maches the SSH Config file identity hostname section from earlier? That’s how it works.</li>
</ul>
</li>
<li>
<p>So using this method, you can upload one, two, or twenty private keys to each calling repo’s secrets and then build a complex ssh config file at run time. You then need to update all your Terraform <code class="language-text highlighter-rouge">source</code> calls to match whatever name you give the host in that file.</p>
</li>
<li>As you can see this is clunky and might work if your org doesn’t allow Github Apps, but going the <a href="https://automationadmin.com/2023/07/create-repo-bot-for-tf-modules">Github App route</a> is much simpler in my opinion. It allows you to reference all module calls the same.</li>
</ol>gerryw1389Github Actions: Unable To Read Private Key2023-09-10T00:24:00-05:002023-09-10T00:24:00-05:00https://automationadmin.com/2023/09/unable-to-load-priv-key-from-akv<!--more-->
<h3 id="description">Description</h3>
<p>A key goal I wish to accomplish with Github Actions is to set all the secrets I can in an Azure Key Vault and then only get them at run time to populate all the other secrets needed. I have been doing this for years with <a href="https://automationadmin.com/2021/01/function-apps-get-secrets/">Function Apps</a> and other Azure services and wish to continue this strategy. Basically, the strategy works like this: <code class="language-text highlighter-rouge">"Only store Azure Credentials as Env secrets and use those at runtime to populate more secrets"</code>. This way you can create a Service Principal with limited rights like <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#key-vault-secrets-user">“Key Vault Secrets User”</a> at the AKV level and then monitor it in Azure Active Directory to ensure it is only being used to access AKVs.</p>
<p>Using that logic, I tried that in Github Actions since there is a way to store a Private Key in AKV under the secrets blade, but unfortunately, it gives various errors. So I ended up just leaving private keys as repo secrets and everything is working as expected. See below with what I tried:</p>
<h3 id="steps">Steps</h3>
<ol>
<li>
<p>First, I uploaded secrets to AKV like usual for private key files:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
</pre></td><td class="rouge-code"><pre><span class="kr">function</span><span class="w"> </span><span class="nf">Add-Secret</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="kr">param</span><span class="w"> </span><span class="p">(</span><span class="w">
</span><span class="p">[</span><span class="n">string</span><span class="p">][</span><span class="n">Parameter</span><span class="p">(</span><span class="n">Mandatory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$true</span><span class="p">)]</span><span class="w"> </span><span class="nv">$KeyVaultName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="kr">throw</span><span class="w"> </span><span class="s2">"The path to the file is required."</span><span class="p">),</span><span class="w">
</span><span class="p">[</span><span class="n">string</span><span class="p">][</span><span class="n">Parameter</span><span class="p">(</span><span class="n">Mandatory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$true</span><span class="p">)]</span><span class="w"> </span><span class="nv">$SecretName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="kr">throw</span><span class="w"> </span><span class="s2">"The path to the file is required."</span><span class="p">),</span><span class="w">
</span><span class="p">[</span><span class="n">string</span><span class="p">][</span><span class="n">Parameter</span><span class="p">(</span><span class="n">Mandatory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$true</span><span class="p">)]</span><span class="w"> </span><span class="nv">$FilePath</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="kr">throw</span><span class="w"> </span><span class="s2">"The path to the file is required."</span><span class="p">),</span><span class="w">
</span><span class="p">[</span><span class="n">System.Nullable</span><span class="p">[</span><span class="n">System.DateTime</span><span class="p">]][</span><span class="n">Parameter</span><span class="p">(</span><span class="n">Mandatory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$false</span><span class="p">)]</span><span class="w"> </span><span class="nv">$Expires</span><span class="p">,</span><span class="w">
</span><span class="p">[</span><span class="n">switch</span><span class="p">][</span><span class="n">Parameter</span><span class="p">(</span><span class="n">Mandatory</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$false</span><span class="p">)]</span><span class="w"> </span><span class="nv">$Base64</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$false</span><span class="w">
</span><span class="p">)</span><span class="w">
</span><span class="nv">$isFileFound</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Test-Path</span><span class="w"> </span><span class="nt">-Path</span><span class="w"> </span><span class="nv">$FilePath</span><span class="w"> </span><span class="nt">-PathType</span><span class="w"> </span><span class="nx">Leaf</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="bp">$false</span><span class="w"> </span><span class="o">-eq</span><span class="w"> </span><span class="nv">$isFileFound</span><span class="p">)</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="n">Write-Error</span><span class="w"> </span><span class="s2">"No file could containing the secret certificate at '</span><span class="nv">$FilePath</span><span class="s2">'"</span><span class="w">
</span><span class="kr">return</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">Write-Output</span><span class="w"> </span><span class="s2">"Creating KeyVault secret..."</span><span class="w">
</span><span class="nv">$secretValue</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$null</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$Base64</span><span class="p">)</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nv">$content</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Get-Content</span><span class="w"> </span><span class="nv">$filePath</span><span class="w"> </span><span class="nt">-AsByteStream</span><span class="w"> </span><span class="nt">-Raw</span><span class="w">
</span><span class="nv">$contentBase64</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="n">System.Convert</span><span class="p">]::</span><span class="n">ToBase64String</span><span class="p">(</span><span class="nv">$content</span><span class="p">)</span><span class="w">
</span><span class="nv">$secretValue</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ConvertTo-SecureString</span><span class="w"> </span><span class="nt">-String</span><span class="w"> </span><span class="nv">$contentBase64</span><span class="w"> </span><span class="nt">-Force</span><span class="w"> </span><span class="nt">-AsPlainText</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kr">else</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nv">$rawContent</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Get-Content</span><span class="w"> </span><span class="nv">$FilePath</span><span class="w"> </span><span class="nt">-Raw</span><span class="w">
</span><span class="nv">$secretValue</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">ConvertTo-SecureString</span><span class="w"> </span><span class="nv">$rawContent</span><span class="w"> </span><span class="nt">-Force</span><span class="w"> </span><span class="nt">-AsPlainText</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="nv">$secret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="bp">$null</span><span class="w">
</span><span class="kr">if</span><span class="w"> </span><span class="p">(</span><span class="nv">$Expires</span><span class="w"> </span><span class="o">-ne</span><span class="w"> </span><span class="bp">$null</span><span class="p">)</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nv">$secret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Set-AzKeyVaultSecret</span><span class="w"> </span><span class="nt">-VaultName</span><span class="w"> </span><span class="nv">$KeyVaultName</span><span class="w"> </span><span class="nt">-SecretName</span><span class="w"> </span><span class="nv">$SecretName</span><span class="w"> </span><span class="nt">-SecretValue</span><span class="w"> </span><span class="nv">$secretValue</span><span class="w"> </span><span class="nt">-Expires</span><span class="w"> </span><span class="nv">$Expires</span><span class="w"> </span><span class="nt">-ErrorAction</span><span class="w"> </span><span class="nx">Stop</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="kr">else</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nv">$secret</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Set-AzKeyVaultSecret</span><span class="w"> </span><span class="nt">-VaultName</span><span class="w"> </span><span class="nv">$KeyVaultName</span><span class="w"> </span><span class="nt">-SecretName</span><span class="w"> </span><span class="nv">$SecretName</span><span class="w"> </span><span class="nt">-SecretValue</span><span class="w"> </span><span class="nv">$secretValue</span><span class="w"> </span><span class="nt">-ErrorAction</span><span class="w"> </span><span class="nx">Stop</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="nv">$version</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$secret</span><span class="o">.</span><span class="nf">Version</span><span class="w">
</span><span class="n">Write-Output</span><span class="w"> </span><span class="s2">"Secret '</span><span class="nv">$SecretName</span><span class="s2">' (Version: '</span><span class="nv">$version</span><span class="s2">') has been created."</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="n">Add-Secret</span><span class="w"> </span><span class="nt">-KeyVaultName</span><span class="w"> </span><span class="s2">"aa-prd-scus-hub-akv-v1"</span><span class="w"> </span><span class="nt">-SecretName</span><span class="w"> </span><span class="s2">"module-cosmosdb-key"</span><span class="w"> </span><span class="nt">-FilePath</span><span class="w"> </span><span class="s2">"C:\scripts\cosmosdb.txt"</span><span class="w">
</span><span class="n">Add-Secret</span><span class="w"> </span><span class="nt">-KeyVaultName</span><span class="w"> </span><span class="s2">"aa-prd-scus-hub-akv-v1"</span><span class="w"> </span><span class="nt">-SecretName</span><span class="w"> </span><span class="s2">"oci-source-provider-key"</span><span class="w"> </span><span class="nt">-FilePath</span><span class="w"> </span><span class="s2">"C:\scripts\oci.txt"</span><span class="w">
</span><span class="n">Add-Secret</span><span class="w"> </span><span class="nt">-KeyVaultName</span><span class="w"> </span><span class="s2">"aa-prd-scus-hub-akv-v1"</span><span class="w"> </span><span class="nt">-SecretName</span><span class="w"> </span><span class="s2">"module-rg-key"</span><span class="w"> </span><span class="nt">-FilePath</span><span class="w"> </span><span class="s2">"C:\scripts\rg.txt"</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>When it got to the <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L111">get-secrets</a> task, it then gave this error:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="rouge-code"><pre>9s
Run az account set --subscription "prd-hub"
b3BlbnNzaC1rZXktdjEAAA..Dd2U5fnIAAAAKmdpdEBnaXRodWIuY29tOkRlbHRhRGVudGFsQ0EvbW9kdW
xlLnJnLmdpdAECAw==
-----END OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACDPVUYMyfZGzvmIZuH2c92s7xuAmsY46R3K3Quo4DC+2AAAALj5yik6+cop
OgAAAAtzc2gtZW..QF
-----END OPENSSH PRIVATE KEY-----
Error: Unable to process file command 'output' successfully.
Error: Invalid format 'b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>So then I thought, maybe it’s the <code class="language-text highlighter-rouge">--output tsv</code> option and I found online that there is a better switch, <code class="language-text highlighter-rouge">none</code>, <a href="https://learn.microsoft.com/en-us/cli/azure/format-output-azure-cli#none-output-format">that you can use for raw output</a>:</p>
</li>
<li>
<p>Unfortunately, this gave a different set of errors:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>Run az account set --subscription "prd-hub"
Warning: Can't add secret mask for empty string in ##[add-mask] command.
Warning: Can't add secret mask for empty string in ##[add-mask] command.
Warning: Can't add secret mask for empty string in ##[add-mask] command.
Warning: Can't add secret mask for empty string in ##[add-mask] command.
Warning: Can't add secret mask for empty string in ##[add-mask] command.
Warning: Can't add secret mask for empty string in ##[add-mask] command.
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>And eventually:</li>
</ul>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>Run cd $GITHUB_WORKSPACE
# github.com:22 SSH-2.0-babeld-f8b1fc6c
# github.com:22 SSH-2.0-babeld-f8b1fc6c
# github.com:22 SSH-2.0-babeld-f8b1fc6c
# github.com:22 SSH-2.0-babeld-f8b1fc6c
# github.com:22 SSH-2.0-babeld-f8b1fc6c
Error loading key "/home/runner/.ssh/module_rg": error in libcrypto
Error: Process completed with exit code 1.
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>One thing I have noticed is that in many circumstances, the <code class="language-text highlighter-rouge">error in libcrypto</code> comes from the loading of the private key incorrectly. For example, I also got this when I had a shell script like:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre><span class="c"># Create key files from Repo Secrets:</span>
<span class="nb">echo</span> <span class="nv">$SSH_KEY_MODULE_RG</span> <span class="o">></span> ~/.ssh/module_rg
<span class="nb">echo</span> <span class="nv">$SSH_KEY_MODULE_COSMOSDB</span> <span class="o">></span> ~/.ssh/module_cosmosdb
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>where the action was like:</li>
</ul>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Setup</span><span class="nv"> </span><span class="s">SSH</span><span class="nv"> </span><span class="s">Keys</span><span class="nv"> </span><span class="s">and</span><span class="nv"> </span><span class="s">known_hosts"</span>
<span class="na">run</span><span class="pi">:</span> <span class="pi">|</span>
<span class="s">cd $GITHUB_WORKSPACE</span>
<span class="s">chmod +x ./.github/scripts/update_ssh_agent.sh</span>
<span class="s">./.github/scripts/update_ssh_agent.sh</span>
<span class="na">env</span><span class="pi">:</span>
<span class="na">SSH_AUTH_SOCK</span><span class="pi">:</span> <span class="s">/tmp/ssh_agent.sock</span>
<span class="na">SSH_KEY_MODULE_RG</span><span class="pi">:</span> <span class="s">${/{ secrets.ssh_key_module_rg }}</span>
<span class="na">SSH_KEY_MODULE_COSMOSDB</span><span class="pi">:</span> <span class="s">${/{ secrets.ssh_key_module_cosmosdb }}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>
<p>NOTE: <a href="https://jekyllrb.com/docs/liquid/filters/">Jekyll Liquid Filters</a> clash with <a href="https://docs.github.com/en/actions/learn-github-actions/variables#using-contexts-to-access-variable-values">Github Variables</a> so replace all instances of <code class="language-text highlighter-rouge">${/{</code> by removing the forward slash :)</p>
</li>
<li>
<p>The fix was to surround them in quotes after getting that <code class="language-text highlighter-rouge">libcrypto</code> error:</p>
</li>
</ul>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
</pre></td><td class="rouge-code"><pre><span class="c"># Create key files from Repo Secrets:</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$SSH_KEY_MODULE_RG</span><span class="s2">"</span> <span class="o">></span> ~/.ssh/module_rg
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$SSH_KEY_MODULE_COSMOSDB</span><span class="s2">"</span> <span class="o">></span> ~/.ssh/module_cosmosdb
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Anyways, I haven’t spent too much energy on this so there may be an easy fix but for now I’m just going to set my PEM files as repo secrets. Just would prefer to follow my standard if possible.</p>
</li>
</ol>gerryw1389My Setup Of Github Organization2023-08-27T00:24:00-05:002023-08-27T00:24:00-05:00https://automationadmin.com/2023/08/setting-up-github-org<!--more-->
<h3 id="description">Description:</h3>
<p>So I decided in order to best showcase Github, it would make sense to create an organization and set a few things up. Here is my docs on it:</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>First, I registered the org <a href="https://github.com/AutomationAdmin-Com">“AutomatonAdmin-Com”</a> because why not?</p>
</li>
<li>
<p>Next, I created a <a href="https://automationadmin.com/2022/08/calling-remote-modules">module repo</a> called <a href="https://github.com/AutomationAdmin-Com/module.rg"><code class="language-text highlighter-rouge">module.rg</code></a> that I will test calling from another repo soon.</p>
</li>
<li>
<p>Next, I created a module composition, which I call a “sic” as in “Shared Infrastructure Composition”, called <a href="https://github.com/AutomationAdmin-Com/sic.mgmt"><code class="language-text highlighter-rouge">sic.mgmt</code></a>. The “mgmt” stands for management as in it should configure management groups and subscriptions. I cover these in later posts but for example subscriptions post can be found <a href="https://automationadmin.com/2023/11/create-tf-subscriptions-payg">here</a>.</p>
<ul>
<li>This repo will have many posts covering what it is and why I created it so I will leave that for them, but for now, let’s focus on calling our module.</li>
</ul>
</li>
<li>
<p>In order to call our module from the sic.mgmt repo using Github Actions, we have two main methods that I have tested and written posts on:</p>
<ul>
<li>We can <a href="https://automationadmin.com/2023/10/using-deploy-keys-for-multiple-tf-module-repos">use Deploy Keys</a></li>
<li>We can <a href="https://automationadmin.com/2023/07/create-repo-bot-for-tf-modules">use a Repo Bot Github App</a></li>
</ul>
</li>
<li>
<p>I went with the Repo bot and I suspect most people will so that as the number of module repos grow, you only have to install the app to new repos and not generate multiple keys.</p>
</li>
<li>
<p>In the interest of explaining all the actions taken, I wanted to also explain the process for creating a new repo, we will use <a href="">sic.template</a> as an example because it is my main template repo at this time:</p>
<ul>
<li>
<p>First, you do the normal create repo in the Github using the UI or you can do everything through REST API or using Github CLI, but I just create the repo in the UI with a README.md</p>
</li>
<li>
<p>Next, pull it to your machine like usual:</p>
</li>
</ul>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="nb">cd </span>to <span class="nb">dir
</span>git clone https://
<span class="c">#make change</span>
git add <span class="nb">.</span>
git commit <span class="nt">-m</span> <span class="s1">'update'</span>
git push origin main
<span class="c"># fill out github popup</span>
<span class="c">#create feature branch</span>
<span class="c">#copy files over</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Next, go to Azure => Entra ID => App Registrations => Your Terraform Service Principal => Federated Credentials => Add <a href="https://automationadmin.com/2023/08/setting-oidc-auth">oidc auth</a> credentials</p>
</li>
<li>
<p>Inside repo, create Prod environment and add the 4 secrets needed: <code class="language-text highlighter-rouge">sub_id</code>, <code class="language-text highlighter-rouge">tenant_id</code>, <code class="language-text highlighter-rouge">client_id</code> and <code class="language-text highlighter-rouge">repo_bot pem</code></p>
</li>
<li>
<p>Next, in Azure, I created a Key Vault called <code class="language-text highlighter-rouge">aa-prd-scus-hub-akv-v1</code> in my <code class="language-text highlighter-rouge">prd-hub</code> subscription that I will use to pull other secrets at run time:</p>
<ul>
<li>I add my user as <code class="language-text highlighter-rouge">Key Vault Administrator</code></li>
<li>
<p>I added my TF Service Principal as <code class="language-text highlighter-rouge">Key Vault Secrets User</code></p>
</li>
<li>I then added all my possible secrets to the AKV:</li>
</ul>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>prd-storage-account-access-key
nonprd-storage-account-access-key
spn-tenant-id
spn-client-id
nonprd-hub-id
prd-hub-id
nonprd-spoke-id
prd-spoke-id
</pre></td></tr></tbody></table></code></pre></div> </div>
<ul>
<li>
<p>My environment is not built out yet completely, but you should absolutely have separate AKVs, Storage Accounts, Subscriptions, and everything really between non-prod and prod environments. I just have prod for now since this is just an example tenant for my blog.</p>
</li>
<li>
<p>Anyways, the last step is to set to <code class="language-text highlighter-rouge">whitelist</code> in <a href="https://learn.microsoft.com/en-us/azure/key-vault/general/network-security#key-vault-firewall-enabled-ipv4-addresses-and-ranges---static-ips">networking for the AKV</a> to ensure that people on the internet cannot try to access my secrets!</p>
</li>
</ul>
</li>
<li>
<p>Lastly, I just add my files to my feature branch which will <a href="https://automationadmin.com/2023/05/main-terraform-workflow">parse files that changed</a> and then call the secrets from my Key Vault <a href="https://automationadmin.com/2023/11/using-akv-to-get-secrets">at run time</a> which is the main goal!</p>
</li>
</ol>gerryw1389Setting Up OIDC Auth For Azure Login and Terraform2023-08-13T00:24:00-05:002023-08-13T00:24:00-05:00https://automationadmin.com/2023/08/setting-oidc-auth<!--more-->
<h3 id="description">Description</h3>
<p>So almost every example you will see online for connecting to Azure or Terraform is to use a Service Principal. I have shown this a couple times, but <a href="https://automationadmin.com/2022/05/setup-azdo-terraform/">here</a> is one I point people to often. Anways, there appears to be multiple ways you can authenticate with Azure/Terraform and I want to go through what I did to setup OIDC Auth.</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>First, let’s tackle <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L91">azure login</a> because that is straight forward:</p>
<ul>
<li>Basically, they list the ways to <a href="https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure?tabs=azure-portal%2Cwindows">connect via OIDC auth</a> but to summarize:</li>
<li>Create a Service Principal</li>
<li>Navigate to it in Azure AD</li>
<li>Go to “Federated Credentials” blade</li>
<li>Click <code class="language-text highlighter-rouge">Add</code></li>
<li>For organization, choose your org: <code class="language-text highlighter-rouge">AutomationAdmin-Com</code> in my case</li>
<li>For repo, choose your repo: <code class="language-text highlighter-rouge">sic.template</code> in my case</li>
<li>For Entity type, you have a few options like <code class="language-text highlighter-rouge">Environment, branch, PR, Tag</code>, I always choose <code class="language-text highlighter-rouge">Environment</code></li>
<li>For Github Name you have to specify from the previous selection the selector. For example, environement has to match the Github Environment you will deploy from in your workflow, branch has to match a branch, etc. I chose <code class="language-text highlighter-rouge">production</code> which I will show you shortly.</li>
<li>For name, you just give a unique name for the credential: I chose <code class="language-text highlighter-rouge">sic-template-env-prod</code>.</li>
</ul>
</li>
<li>
<p>Next, inside my <a href="https://github.com/AutomationAdmin-Com/sic.template">sic.template</a> repo, I added these 3 required secrets for the action: <code class="language-text highlighter-rouge">${/{ secrets.CLIENT_ID }}</code>, <code class="language-text highlighter-rouge">${/{ secrets.TENANT_ID }}</code>, and <code class="language-text highlighter-rouge">${/{ secrets.SUB_ID }}</code> where SUB_ID is just one of my Azure Subscriptions <code class="language-text highlighter-rouge">id</code> property, it doesn’t matter which one.</p>
<ul>
<li>NOTE: <a href="https://jekyllrb.com/docs/liquid/filters/">Jekyll Liquid Filters</a> clash with <a href="https://docs.github.com/en/actions/learn-github-actions/variables#using-contexts-to-access-variable-values">Github Variables</a> so replace all instances of <code class="language-text highlighter-rouge">${/{</code> by removing the forward slash :)</li>
</ul>
</li>
<li>Next, in my Github Actions workflow, I had to enable 2 things:
<ul>
<li>First, I had to set <code class="language-text highlighter-rouge">id-token: write</code> <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L15">permission</a></li>
<li>Next, I had to set environment to <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/.github/workflows/main.yml#L47"><code class="language-text highlighter-rouge">production</code></a> even though my deployments won’t always target production as seen in my <a href="https://automationadmin.com/2023/05/main-terraform-workflow">main template post</a> ( or <a href="https://automationadmin.com/2023/10/terraform-workflow-repo-structure">part 2</a> ).</li>
</ul>
</li>
<li>
<p>OK, so we can now get past Azure Login as seen in the logs from a most recent run:</p>
<div class="language-escape highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre>Run azure/login@v1
Using OIDC authentication...
Federated token details:
issuer - https://token.actions.githubusercontent.com
subject claim - repo:AutomationAdmin-Com/sic.template:environment:production
/usr/bin/az cloud set -n azurecloud
Done setting cloud: "azurecloud"
Login successful.
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Next, we need to set Terraform to know about OIDC Auth. Thankfully, <a href="https://registry.terraform.io/providers/hashicorp/azurerm/3.80.0/docs/guides/service_principal_oidc">this is documented well</a> in the provider docs. To summarize:</p>
<ul>
<li>Remove any passing of <code class="language-text highlighter-rouge">client_secret</code> as a secret to terraform and remove the variable altogether from <code class="language-text highlighter-rouge">variables.tf</code> or any other place.</li>
<li>Next, in your providers, just replace that reference with <code class="language-text highlighter-rouge">use_oidc = true</code> everywhere you would have used client_secret. That’s it!</li>
<li>You can verify by going to any of my <a href="https://github.com/AutomationAdmin-Com/sic.template/blob/484737f27f67780c6a35a5c7288a230efec4d5c7/config/nonprd/hub/scus/stage1/none/backend.tf#L39C3-L39C36"><code class="language-text highlighter-rouge">backend.tf</code> files</a></li>
</ul>
</li>
<li>What I haven’t tested yet is verifying the providers that get built get passed correctly but I know terraform doesn’t error so I assume the providers build correctly. Will need to remember to update this later once I test!</li>
</ol>gerryw1389Creating Key Pairs For Terraform Use2023-07-16T00:24:00-05:002023-07-16T00:24:00-05:00https://automationadmin.com/2023/07/creating-key-pairs-for-terraform-use<!--more-->
<h3 id="description">Description</h3>
<p>Sometimes, you will need to create a key pair, deploy a VM, and then add that key to the Authorized Hosts. Here is how:</p>
<h3 id="to-resolve">To Resolve:</h3>
<ol>
<li>
<p>Create keypair for a new user:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="nb">cd </span>c:<span class="se">\s</span>cripts
<span class="nb">mkdir </span>keys
<span class="nb">cd</span> ./keys/
ssh-keygen <span class="nt">-t</span> rsa <span class="nt">-m</span> PEM <span class="nt">-b</span> 2048
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Take public portion and add it to your variables:</p>
<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="k">variable</span> <span class="s2">"ssh_authorized_keys"</span> <span class="p">{</span>
<span class="nx">type</span> <span class="p">=</span> <span class="nx">string</span>
<span class="nx">description</span> <span class="p">=</span> <span class="s2">"ssh authorized key"</span>
<span class="nx">default</span> <span class="p">=</span> <span class="s2">"ssh-rsa AAAAB3Nza..D0RiVG6Wz7oisRN81Xu4iraxN"</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Run the <code class="language-text highlighter-rouge">terraform apply</code> to provision the VM.</p>
</li>
<li>
<p>Next, you should be able to ssh to the machine and add other keys as needed.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre>gerry@home:C:<span class="se">\s</span>cripts<span class="se">\k</span>eys
<span class="o">></span> ssh <span class="nt">-i</span> key cloud-user@10.97.4.205
The authenticity of host <span class="s1">'10.97.4.205 (10.97.4.205)'</span> can<span class="s1">'t be established.
ECDSA key fingerprint is SHA256:E5vsJ6oDFIZxG+AvQaxxEqZ9Ck8b8IBKBBSASBrxfpA.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '</span>10.97.4.205<span class="s1">' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
[cloud-user@my-vm ~]$
[cloud-user@my-vm ~]$ sudo su
[root@my-vm cloud-user]#
[root@my-vm cloud-user]# whoami
root
[root@my-vm cloud-user]#
[root@my-vm cloud-user]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)
[root@my-vm cloud-user]#
</span></pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Here we create a new local user and add some public portion of a key pair to authorized_keys file:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre>useradd myuser
<span class="nb">mkdir</span> /home/myuser/.ssh
<span class="nb">echo </span>ssh-rsa some-key <span class="o">>></span> /home/myuser/.ssh/authorized_keys
visudo <span class="c"># add : %myuser ALL=(ALL) NOPASSWD: ALL</span>
<span class="nb">chown</span> <span class="nt">-R</span> myuser:myuser /home/myuser/.ssh
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>In Windows, to connect to a newly deployed VM you will need to use the SSH Keys you just created. Here is how you do it, for example, connecting to the <code class="language-text highlighter-rouge">my-vm</code> in previous steps:</p>
</li>
<li>First, get the private key and paste it in a file called <code class="language-text highlighter-rouge">priv.txt</code> at <code class="language-text highlighter-rouge">c:\scripts</code>.
<ul>
<li>Now right click on the file and “disable inheritence” and “convert all permissions to explicit”.</li>
<li>Then remove everyone except your user and give your user full rights to the file.</li>
</ul>
</li>
<li>
<p>Now ensure you have the Windows SSH Client feature installed, <code class="language-text highlighter-rouge">Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0</code></p>
</li>
<li>
<p>Next, open Powershell and cd to that directory and use the key file to connect to a VM:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre><span class="nb">cd </span>c:<span class="se">\s</span>cripts
ssh <span class="nt">-i</span> priv.txt cloud-user@10.10.10.10
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
</ol>gerryw1389