devDosvid blog
https://devdosvid.blog/
2024-03-25T17:30:06Z
devDosvid blog
https://devdosvid.blog/
Hugo gohugo.io
https://devdosvid.blog/assets/img/websitelogo.jpg
Mastering AWS API Gateway V2 HTTP and AWS Lambda With Terraform
2024-01-09T02:33:10Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2024/01/09/mastering-aws-api-gateway-v2-http-and-aws-lambda-with-terraform/
Serhii Vasylenko
https://devdosvid.blog/2024/01/09/mastering-aws-api-gateway-v2-http-and-aws-lambda-with-terraform/cover-image.jpg
<p>With a solid foundation in AWS API Gateway and Lambda for serverless architecture, my recent deep dive into these cloud computing services felt like uncovering new layers in familiar territory. This article aims to be a comprehensive guide for developers and DevOps professionals looking to master serverless solutions using AWS and Terraform.</p>
<p>The article provides an in-depth guide to <strong>combining AWS API Gateway V2 HTTP API</strong> (yes, this is the official name of that service π) <strong>and AWS Lambda</strong> services to implement a simple, robust, and cost-effective serverless back-end using Terraform.</p>
<p>The journey was enlightening and engaging, especially as I were transforming these services into Infrastructure as Code. Through this article, I aim to share those moments of insight and the practical, hands-on tips that emerged from weaving these AWS services into a seamless, serverless architecture.</p>
<h2 id="navigating-the-system-design-http-api-gateway-and-lambda-in-action">Navigating the System Design: HTTP API Gateway and Lambda in Action</h2>
<p>Beginning our journey, we examine the complexities of serverless architecture, focusing on HTTP API and Lambda. A comprehensive system diagram will guide us as we analyze each component’s function and their collaborative roles in the larger infrastructure.</p>
<figure>
<img loading="lazy"
src="api-gateway-lambda-authorization-flow.png"
alt="HTTP API Gateway and AWS Lambda flowchart"width="800"
height="440.50"
/> <figcaption>
<p>HTTP API Gateway and AWS Lambda flowchart</p>
</figcaption>
</figure>
<p>In our architecture, the HTTP API delegates access control to the Lambda function called “Authorizer”. This function stands as the gatekeeper, ensuring that only legitimate requests pass through to the underlying business logic.</p>
<p>The HTTP API can have multiple routes (e.g., “/calendar,” “/meters,” and so on) and use different Authorizers per route or a single one for all of them. Clients that send their requests to the API must include specific identification information in their request header or query string. In this project, I go with a single authorizer to keep it simple.</p>
<p>Upon receiving a request, the API service forwards a payload to the Authorizer containing metadata about the request, such as headers and query string components. The Authorizer processes this metadata (headers, in my case) to determine the request’s legitimacy.</p>
<p>The decision, Allow or Deny, is passed back to the API, and if allowed, the API service then forwards the original request to the back-end, which, in this case, is implemented by additional Lambda functions. Otherwise, the client gets a response with a 403 status code, and the original request is not passed to the back-end.</p>
<div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<h2 id="behind-the-decision-why-such-a-setup">Behind The Decision: Why Such a Setup?</h2>
<p>Choosing the right architectural setup is critical in balancing simplicity, cost-efficiency, and security. In this section, we uncover why integrating AWS HTTP API Gateway with Lambda Authorizer is a compelling choice, offering a streamlined approach without compromising security.</p>
<h3 id="cost-effectiveness-balancing-performance-and-price">Cost-Effectiveness: Balancing Performance and Price</h3>
<p>The AWS HTTP API is noteworthy for its streamlined and simple design compared to other API Gateway options. That translates directly into cost savings for businesses. Its efficiency makes it an ideal choice for cost-effective serverless computing, especially for those looking to optimize their cloud infrastructure with Terraform automation. Here is a more detailed comparison of different API Gateway options β <a href="https://docs.aws.amazon.com/whitepapers/latest/best-practices-api-gateway-private-apis-integration/cost-optimization.html">Cost optimization</a>.</p>
<p>Security with Lambda Authorizer. This option means a Lambda function used for authorization, which is lean and efficient. It generally requires a bare minimum of resources. It executes quickly, particularly when configured with the ARM-based environment and 128M RAM allocation, costing $0,0000017 per second of running time, with $0.20 per 1M requests per month.</p>
<p>π° This pricing and performance combination are well-suited for rapid, lightweight authorizations. Together with AWS Lambda as a back-end, it makes a cost-effective solution. For example, if we add a few more Lambdas to back-end and assume that our setup receives 10000 requests per month, it would cost around $0.6 per month. Here is the link to detailed calculations β <a href="https://calculator.aws/#/estimate?id=b1a8a473ab98ede32f5ca384c5e9487b967efafa">AWS Pricing Calculator</a>.</p>
<h3 id="simplicity-in-configuration-the-power-of-header-based-authorization">Simplicity in Configuration: The Power of Header-Based Authorization</h3>
<p>A header-based authentication method facilitates straightforward client-server communication, often requiring less coding and resources to implement compared to more complex schemes.</p>
<p>Although HTTP API offers stronger JWT-based authorization and mutual TLS authentication, header-based authorization remains a suitable choice for simpler applications that prioritize ease and quickness. <em>By the way, there is also an option for IAM-based authorization whose core idea is the “private API” or internal usage of the API (e.g., solely inside the VPC, no internet), but with “<a href="https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html">IAM Anywhere</a>,” this can be expanded to practically anywhere.</em> π</p>
<p>This architecture suits applications requiring rapid development and deployment without complex authorization mechanisms. It’s ideal for small to medium-sized serverless applications or specific use cases in larger systems where quick, cost-effective, and secure access to APIs is a priority.</p>
<p>π‘ Imagine a retail company wanting to manage its inventory efficiently. By leveraging AWS API Gateway and Lambda, they can develop a system where each item’s RFID tags are scanned and processed through an API endpoint. When a product is moved or sold, its status is updated in real-time in the database, facilitated by Lambda functions. This serverless architecture ensures high availability and scalability and significantly reduces operational costs, a crucial factor for the highly competitive retail industry. This example showcases how our serverless setup can be effectively utilized in retail for streamlined inventory tracking and management.</p>
<h2 id="exploring-aws-lambda-features-and-integration">Exploring AWS Lambda: Features and Integration</h2>
<p>Diving into AWS Lambda, this section explores its features and indispensable role within the serverless infrastructure. We will unravel the complexities of Lambda functions and examine the practicalities of deploying and managing these functions within the project.</p>
<h3 id="aws-lambda-runtime-and-deployment-model">AWS Lambda Runtime and Deployment Model</h3>
<p>π Choosing the <strong>AWS Lambda runtime arm64</strong>, combined with the OS-only runtime based on <strong>Amazon Linux 2023</strong>, strategically boosts cost efficiency and performance. This choice aligns with the best practices for serverless computing in AWS, offering an optimal solution for those seeking to leverage AWS services for scalable cloud solutions.</p>
<p>Particularly effective for Go-based functions, this runtime configuration is lean yet powerful. For applications in other languages, delving into <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html">language-specific runtimes based on AL 2023</a> can also leverage the latest efficiencies of AWS-managed operating systems.</p>
<div class="attention">
I also welcome you to read this benchmarking analysis to get more insights about the ARM-based environment for AWS Lambda β <a href="https://aws.amazon.com/blogs/apn/comparing-aws-lambda-arm-vs-x86-performance-cost-and-analysis-2/">Comparing AWS Lambda Arm vs. x86 Performance, Cost, and Analysis</a>.
</div>
<p>The .<strong>zip deployment</strong> model is chosen for its simplicity, avoiding additional management of the image registry (ECR) and Docker images. Also, AWS automatically patches .zip functions for the latest runtime security and bug fixes.</p>
<h3 id="efficient-terraform-coding-for-aws-lambda">Efficient Terraform Coding for AWS Lambda</h3>
<p>In our architecture, AWS Lambda functions serve dual purposes β as an authentication gatekeeper and a robust back-end for business logic. Despite varying code across functions, their configurations share much of similarities.</p>
<p>By adhering to the DRY (Don’t Repeat Yourself) principle, I have crafted a Terraform module to streamline the management of Lambda functions and their dependencies. This approach ensures maintainable and scalable infrastructure. The module’s structure is as follows:</p>
<ul>
<li><code>aws_lambda_function</code> β to describe the core configuration of the function</li>
<li><code>aws_iam_role</code> + <code>aws_iam_role_policy</code> + <code>aws_iam_policy_document</code> β to manage the access from Lambda to other resources (e.g., SSM Parameter Store)</li>
<li><code>aws_cloudwatch_log_group</code> β to keep the execution logs</li>
<li><code>aws_ssm_parameter</code> β to store sensitive information (e.g., secrets) and other configurations that we should keep separate from the source code.</li>
</ul>
<p>This Terraform module implements a project-specific use case for Lambda functions. However, if you’re seeking for a generic all-in-one module for AWS Lambda, I recommend checking out this one β <a href="https://registry.terraform.io/modules/terraform-aws-modules/lambda/aws/latest">Terraform AWS Lambda Module</a> by Anton Babenko.</p>
<div class="attention">
<p>To efficiently develop Terraform code for Lambda functions, use the following techniques:</p>
<ul>
<li>Use local values, expressions, and variables to implement consistent naming across different resources logically grouped by a module or project;</li>
<li>Use function environment variables to connect the code with SSM Parameter Store parameters or Secrets Manager secrets to protect sensitive data like tokens or credentials;</li>
<li>Use <code>for_each</code> meta-argument and <code>for</code> expression to reduce the amount of code and automate the configuration for resources of the same type (e.g., <code>ssm_parameter</code>) or code blocks within a resource.</li>
</ul>
</div>
<p>Below is a practical example illustrating these Terraform strategies in action:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>locals {
</span></span><span style="display:flex;"><span> full_function_name = <span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>var.project_name<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">-</span><span style="color:#a5d6ff">${</span>var.function_name<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_lambda_function"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> function_name = local.full_function_name
</span></span><span style="display:flex;"><span> role = aws_iam_role.this.arn
</span></span><span style="display:flex;"><span> architectures = [<span style="color:#a5d6ff">"arm64"</span>]
</span></span><span style="display:flex;"><span> filename = var.deployment_file
</span></span><span style="display:flex;"><span> package_type = <span style="color:#a5d6ff">"Zip"</span>
</span></span><span style="display:flex;"><span> runtime = <span style="color:#a5d6ff">"provided.al2023"</span>
</span></span><span style="display:flex;"><span> handler = <span style="color:#a5d6ff">"bootstrap.handler"</span>
</span></span><span style="display:flex;"><span> timeout = var.function_timeout
</span></span><span style="display:flex;"><span> environment {
</span></span><span style="display:flex;"><span> variables = { <span style="color:#ff7b72">for</span> item <span style="color:#ff7b72">in</span> var.function_ssm_parameter_names <span style="color:#ff7b72;font-weight:bold">:</span> upper(replace(item, <span style="color:#a5d6ff">"-"</span>, <span style="color:#a5d6ff">"_"</span>)) => aws_ssm_parameter.function_ssm_parameters[item].name }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_ssm_parameter"</span> <span style="color:#a5d6ff">"function_ssm_parameters"</span> {
</span></span><span style="display:flex;"><span> for_each = var.function_ssm_parameter_names
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"/projects/</span><span style="color:#a5d6ff">${</span>var.project_name<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/lambda/</span><span style="color:#a5d6ff">${</span>var.function_name<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/</span><span style="color:#a5d6ff">${</span>each.value<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span> type = <span style="color:#a5d6ff">"SecureString"</span>
</span></span><span style="display:flex;"><span> key_id = data.aws_kms_alias.ssm.arn
</span></span><span style="display:flex;"><span> value = <span style="color:#a5d6ff">"1"</span>
</span></span><span style="display:flex;"><span> lifecycle {
</span></span><span style="display:flex;"><span> ignore_changes = [
</span></span><span style="display:flex;"><span> value,
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudwatch_log_group"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"/aws/lambda/</span><span style="color:#a5d6ff">${</span>local.full_function_name<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span> log_group_class = <span style="color:#a5d6ff">"STANDARD"</span>
</span></span><span style="display:flex;"><span> retention_in_days = <span style="color:#a5d6ff">7</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p><em>The complete terraform module code is available in the <a href="https://github.com/vasylenko/inkyframe/blob/main/infra/modules/lambda/main.tf">project repository</a>.</em></p>
<p>In this Terraform code, I deliberately hardcoded specific arguments for an optimal Lambda runtime configuration, ensuring efficiency and performance.</p>
<p>Then variables and local values, set only once, implement a naming convention for all resource arguments, making it easy to understand the infrastructure and change the naming and attributes later.</p>
<div class="attention">
Lambda’s environment variables and corresponding SSM parameters coexist effectively with the help of <code>for_each</code> and <code>for</code>. I used the <code>for_each</code> meta-argument to dynamically create SSM Parameter resources and the <code>for</code> expression to configure environment variables in AWS Lambda. This also means that if the <code>function_ssm_parameter_names</code> variable value is not provided, then Terraform does not create either SSM parameter resources or the environment code block inside the Lambda resource because the default value of that variable is an empty set.
</div>
<p>By the way, I have another blog post that explains several techniques to enhance your Terraform proficiency β <a href="https://devdosvid.blog/2022/01/16/some-techniques-to-enhance-your-terraform-proficiency">check it out</a>!</p>
<h3 id="invoking-lambda-permissions-and-resource-based-policies">Invoking Lambda: Permissions and Resource-Based Policies</h3>
<p>Configured with just a few input variables, the Terraform module efficiently outputs the <code>aws_lambda_function</code> resource. This streamlined output is then adeptly used to facilitate subsequent configurations within the HTTP API.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">module</span> <span style="color:#a5d6ff">"lambda_api_gw_authorizer"</span> {
</span></span><span style="display:flex;"><span> source = <span style="color:#a5d6ff">"./modules/lambda"</span>
</span></span><span style="display:flex;"><span> deployment_file = <span style="color:#a5d6ff">"../backend/lambda-apigw-authorizer/deployment.zip"</span>
</span></span><span style="display:flex;"><span> function_name = <span style="color:#a5d6ff">"api-gateway-authorizer"</span>
</span></span><span style="display:flex;"><span> project_name = local.project_name
</span></span><span style="display:flex;"><span> function_ssm_parameters = [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"authorization-token"</span>
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">
</span></span></span><span style="display:flex;"><span><span style="color:#ff7b72">module</span> <span style="color:#a5d6ff">"lambda_calendar_backend"</span> {
</span></span><span style="display:flex;"><span> source = <span style="color:#a5d6ff">"./modules/lambda"</span>
</span></span><span style="display:flex;"><span> deployment_file = <span style="color:#a5d6ff">"../backend/lambda-calendar-backend/deployment.zip"</span>
</span></span><span style="display:flex;"><span> function_name = <span style="color:#a5d6ff">"calendar-backend"</span>
</span></span><span style="display:flex;"><span> project_name = local.project_name
</span></span><span style="display:flex;"><span> function_ssm_parameters = [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"google-api-oauth-token"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"google-api-credentials"</span>
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>As an example of module output usage, here is the configuration of <code>aws_lambda_permissions</code> resource that I use outside the AWS Lambda module to allow the HTTP API service to invoke the function used as Authorizer:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_lambda_permission"</span> <span style="color:#a5d6ff">"allow_api_gw_invoke_authorizer"</span> {
</span></span><span style="display:flex;"><span> statement_id = <span style="color:#a5d6ff">"allowInvokeFromAPIGatewayAuthorizer"</span>
</span></span><span style="display:flex;"><span> action = <span style="color:#a5d6ff">"lambda:InvokeFunction"</span>
</span></span><span style="display:flex;"><span> function_name = module.lambda_api_gw_authorizer.lambda.function_name
</span></span><span style="display:flex;"><span> principal = <span style="color:#a5d6ff">"apigateway.amazonaws.com"</span>
</span></span><span style="display:flex;"><span> source_arn = <span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>aws_apigatewayv2_api.this.execution_arn<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/authorizers/</span><span style="color:#a5d6ff">${</span>aws_apigatewayv2_authorizer.header_based_authorizer.id<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><div class="attention">
The Lambda resource-based policy combines the trust and permission policies, and provides a simple yet efficient way to grant other AWS services or principals the ability to invoke Lambda functions. It is important to note that for an API to invoke a function, Lambda requires its <strong>execution</strong> ARN, not the resource ARN.
</div>
<p>As a side note, check out this <a href="https://docs.aws.amazon.com/lambda/latest/operatorguide/intro.html">AWS Lambda Operator Guide</a>, which offers specialized advice on developing, securing, and monitoring applications based on AWS Lambda.</p>
<p>Let’s switch to the HTTP API part to see how it looks and learn how it integrates Lambda functions.</p>
<h2 id="deep-dive-into-http-api-gateway">Deep Dive into HTTP API Gateway</h2>
<p>Now, we focus on the HTTP API Gateway, delving into its essential concepts, seamless integration with AWS Lambda, and using Terraform efficiently for streamlined configuration.</p>
<p>But before we do that, and since we have partially covered the Terraform code already, I’d like to illustrate the logical connection between three main components of the project’s Terraform codebase: AWS Lambda, HTTP API, and API Routes.</p>
<figure>
<img loading="lazy"
src="terraform-aws-lambda-http-api-integration.png"
alt="AWS Lambda and HTTP API Terraform integration diagram"width="800"
height="552.50"
/> <figcaption>
<p>AWS Lambda and HTTP API Terraform integration diagram</p>
</figcaption>
</figure>
<p>I will explain the API Route module in detail a bit later, but for now, for the broader context, here is what happens inside Terraform:
AWS HTTP API code logically represents the “global” (within a project) set of resources and uses the function created by the Lambda Terraform module for the Authorizer configuration. Meanwhile, the API Route Terraform module configures specific routes for the HTTP API (hence, requires some info from it) with integration to back-ends implemented by Lambdas (hence, requires some info from them, too).</p>
<p>Back to HTTP API overview. The following <strong>components of the HTTP API</strong> constitute its backbone:</p>
<ul>
<li><strong>Route</strong> β a combination of the HTTP method (e.g., GET or POST) with the API route (e.g., /meters). For example: “POST /meters”. Routes can optionally use <strong>Authorizers</strong> β a mechanism to control access to the HTTP API.</li>
<li><strong>Integration</strong> β the technical and logical connection between the Route and one of the supported back-end resources. For example, with AWS Lambda integration, API Gateway sends the entire request as input to a back-end Lambda function and then transforms the Lambda function output to a front-end HTTP response.</li>
<li><strong>Stage and Deployment</strong> β A stage serves as a designated reference to a deployment, essentially capturing a snapshot of the API at a certain point. It’s employed to control and optimize a specific deployment version. For instance, stage configurations can be adjusted to tailor request throttling, set up logging, or establish stage variables to be used by API (if needed).</li>
</ul>
<h3 id="implementing-aws-api-gateway-v2-http-api-with-terraform">Implementing AWS API Gateway V2 HTTP API with Terraform</h3>
<p>Below, I detail the Terraform resources essential for implementing the HTTP API, ensuring a transparent and effective setup:</p>
<ul>
<li><code>aws_apigatewayv2_api</code> β the HTTP API itself;</li>
<li><code>aws_apigatewayv2_route</code> β the Route for the API that must specify the integration target (e.g., Lambda) and, optionally, the Authorizer;</li>
<li><code>aws_apigatewayv2_authorizer</code> β the Authorizer to use for Routes;</li>
<li><code>aws_apigatewayv2_integration</code> β the resource that specifies the back-end where the API sends the requests (e.g., AWS Lambda);</li>
<li><code>aws_lambda_permission</code> β the resource-based policy for AWS Lambda to allow the invocations from the API;</li>
<li><code>aws_apigatewayv2_stage</code> β the name of the Stage that references the Deployment.</li>
</ul>
<h3 id="applying-terraform-for-http-api-gateway-and-lambda-authorizer">Applying Terraform for HTTP API Gateway and Lambda Authorizer</h3>
<p>The HTTP API is the simplest in the API Gateway family (so far), so its Terraform resource has relatively few configuration options, most of which can be left at their default values.</p>
<p>As for the Authorizer, it can have two options for letting API know its decision: <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html#http-api-lambda-authorizer.payload-format-response">simple response and IAM policy</a>.</p>
<p>The <strong>simple response</strong> just returns a Boolean value to indicate whether the API should allow the request (True) or forbid it (False).</p>
<p>The IAM policy option is customizable and allows crafting custom policy statements that allow granular access to explicitly provided resources.</p>
<p>In this project, I follow the way of simplicity and use the “simple response”, so the response from Lambda Authorizer to HTTP API looks as follows:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"isAuthorized"</span>: <span style="color:#79c0ff">true</span><span style="color:#f85149">/</span><span style="color:#79c0ff">false</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Let’s review the HTTP API resource along with the API Authorizer that I used for all routes:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_api"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> name = local.project_name
</span></span><span style="display:flex;"><span> protocol_type = <span style="color:#a5d6ff">"HTTP"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">
</span></span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_authorizer"</span> <span style="color:#a5d6ff">"header_based_authorizer"</span> {
</span></span><span style="display:flex;"><span> api_id = aws_apigatewayv2_api.this.id
</span></span><span style="display:flex;"><span> authorizer_type = <span style="color:#a5d6ff">"REQUEST"</span>
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"header-based-authorizer"</span>
</span></span><span style="display:flex;"><span> authorizer_payload_format_version = <span style="color:#a5d6ff">"2.0"</span>
</span></span><span style="display:flex;"><span> authorizer_uri = module.lambda_api_gw_authorizer.lambda.invoke_arn
</span></span><span style="display:flex;"><span> enable_simple_responses = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> identity_sources = [<span style="color:#a5d6ff">"$request.header.authorization"</span>]
</span></span><span style="display:flex;"><span> authorizer_result_ttl_in_seconds = <span style="color:#a5d6ff">3600</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_lambda_permission"</span> <span style="color:#a5d6ff">"allow_api_gw_invoke_authorizer"</span> {
</span></span><span style="display:flex;"><span> statement_id = <span style="color:#a5d6ff">"allowInvokeFromAPIGatewayAuthorizer"</span>
</span></span><span style="display:flex;"><span> action = <span style="color:#a5d6ff">"lambda:InvokeFunction"</span>
</span></span><span style="display:flex;"><span> function_name = module.lambda_api_gw_authorizer.lambda.function_name
</span></span><span style="display:flex;"><span> principal = <span style="color:#a5d6ff">"apigateway.amazonaws.com"</span>
</span></span><span style="display:flex;"><span> source_arn = <span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>aws_apigatewayv2_api.this.execution_arn<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/authorizers/</span><span style="color:#a5d6ff">${</span>aws_apigatewayv2_authorizer.header_based_authorizer.id<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p><em>The complete code is available in the <a href="https://github.com/vasylenko/inkyframe/blob/main/infra/apigateway.tf">project repository</a></em>.</p>
<p>Consider the following key points when Terraforming this part.</p>
<p><code>identity_sources</code> argument of the <code>aws_apigatewayv2_authorizer</code> resource: This is where I defined what exactly the Authorizer should validate. I used the header named <code>authorization</code> so the Authorizer Lambda function would check its value to decide whether to authorize the request.<br>
π‘ <em>Check out other options available to use as the identity source β <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-lambda-authorizer.html#http-api-lambda-authorizer.identity-sources">Identity sources</a></em>.</p>
<p><code>authorizer_uri</code> argument of the <code>aws_apigatewayv2_authorizer</code> resource: It is the <strong>invocation</strong> ARN of the Lambda function used as Authorizer (not the Lambda’s resource ARN).</p>
<p><code>authorizer_result_ttl_in_seconds</code> argument of the <code>aws_apigatewayv2_authorizer</code> resource: This allows to skip the Authorizer invocation for the given time if a client provided the same identity source values (e.g., authorization header).</p>
<div class="attention">
AWS API Gateway HTTP can employ the identity sources as the cache key to preserve the authorization results for a while. Should a client provide identical parameters in identity sources within the preset TTL duration, API Gateway will retrieve the result from the cached authorizer instead of calling upon it again. This helps save a lot on AWS Lambda Authorizer invocations and works great with simple scenarios. However, it might be cumbersome if you need severral custom authorization responses per function or if you use custom IAM policies instead of the “simple response” option.
</div>
<p><code>source_arn</code> argument of <code>aws_lambda_permission</code>: Similar to the <code>authorizer_uri</code> argument, this one expects the <strong>execution</strong> ARN of the HTTP API followed by the Authorizer identifier.</p>
<p>Now, let’s see how Routes are codified with Terraform.</p>
<h3 id="applying-terraform-for-http-api-routes">Applying Terraform for HTTP API Routes</h3>
<p>π‘ Because an API typically has multiple routes, creating another Terraform module that implements the configurable HTTP API Gateway route is beneficial. Hence, the <code>aws_apigatewayv2_route</code>, <code>aws_apigatewayv2_integration</code>, and <code>aws_lambda_permission</code> resources would constitute such a module.</p>
<p>This Terraform module implements a specific use case for HTTP API Gateway. However, if you’re seeking for a generic all-in-one module for API Gateway, I recommend checking out this one β <a href="https://registry.terraform.io/modules/terraform-aws-modules/apigateway-v2/aws/latest">Terraform AWS API Gateway Module</a> by Anton Babenko.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_route"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> api_id = var.api_id
</span></span><span style="display:flex;"><span> route_key = var.route_key
</span></span><span style="display:flex;"><span> authorization_type = <span style="color:#a5d6ff">"CUSTOM"</span>
</span></span><span style="display:flex;"><span> authorizer_id = var.authorizer_id
</span></span><span style="display:flex;"><span> target = <span style="color:#a5d6ff">"integrations/</span><span style="color:#a5d6ff">${</span>aws_apigatewayv2_integration.this.id<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_integration"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> api_id = var.api_id
</span></span><span style="display:flex;"><span> integration_type = <span style="color:#a5d6ff">"AWS_PROXY"</span>
</span></span><span style="display:flex;"><span> connection_type = <span style="color:#a5d6ff">"INTERNET"</span>
</span></span><span style="display:flex;"><span> integration_uri = var.lambda_invocation_arn
</span></span><span style="display:flex;"><span> payload_format_version = <span style="color:#a5d6ff">"2.0"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_lambda_permission"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> statement_id = <span style="color:#a5d6ff">"allowInvokeFromAPIGatewayRoute"</span>
</span></span><span style="display:flex;"><span> action = <span style="color:#a5d6ff">"lambda:InvokeFunction"</span>
</span></span><span style="display:flex;"><span> function_name = var.lambda_function_name
</span></span><span style="display:flex;"><span> principal = <span style="color:#a5d6ff">"apigateway.amazonaws.com"</span>
</span></span><span style="display:flex;"><span> source_arn = <span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>var.api_gw_execution_arn<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/*/*/*/*"</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>First, I want to highlight several key aspects for understanding the resources’ arguments within that module.</p>
<p>The <code>target</code> argument of the <code>aws_apigatewayv2_route</code> resource implies that the integration ID should be prefixed with the “<code>integrations/</code>” keyword.</p>
<p>While the <code>connection_type</code> argument of the <code>aws_apigatewayv2_integration</code> resource specifies “INTERNET”, it does not mean that the Lambda function must have the publicly available URL. This value must be used unless you work with a VPC endpoint for API Gateway for internal usage.</p>
<p>For the <code>source_arn</code> argument in the <code>aws_lambda_permission</code> resource, similar to earlier, it requires the <strong>execution</strong> ARN of the API. However, this time, it is the integration of the HTTP API Route with Lambda. And the ARN format of this one is different and a bit tricky:</p>
<p><code>arn:partition:execute-api:region:account-id:api-id/stage/http-method/resource-path</code></p>
<p>The <code>arn:partition:execute-api:region:account-id:api-id</code> part constitutes the execution ARN of the HTTP API itself, so for the sake of simplicity, I decided to go with wildcards after it.<br>
<em>For your convenience, here is the <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/arn-format-reference.html">detailed specification</a> of API Gateway ARNs.</em></p>
<p>The HTTP API Route module expects several input variables:</p>
<ul>
<li><code>authorizer_id</code> β the identifier of the Authorizer to use on this route;</li>
<li><code>route_key</code> β the route key for the route, e.g., <code>GET /foo/bar</code>;</li>
<li><code>api_id</code> β the identifier of HTTP API created earlier;</li>
<li><code>lambda_invocation_arn</code> β the Invocation ARN of the Lambda function;</li>
<li><code>lambda_function_name</code> β the name of the Lambda function to integrate with the route;</li>
<li><code>api_gw_execution_arn</code> β the Execution ARN of the HTTP API that invokes a Lambda function.</li>
</ul>
<p>Let’s take a closer look on API Gateway V2 HTTP API route.</p>
<p>A route consists of an HTTP method and a resource path with an optional variable. Based on the pre-defined convention, it uses a simplified routing configuration and methods request model (comparable to other APIs).</p>
<div class="attention">
<p>While I was working with the HTTP API, I found this simplified approach to be great because it allows easy access to the request context from AWS Lambda functions, for example:</p>
<ul>
<li>A path variable in a route, e.g., <code>GET /calendars/{calendar-name}</code>, would be available for the integrated AWS Lambda by its name inside the pathParameter JSON field, e.g., <code>pathParamters.calendar-name</code>, of the event object sent by API to Lambda. In other words, you do not need to explicitly set the mapping between the path variable and its representation to the back-end.</li>
<li>A request query string is parsed into separate parameter-value pairs and available in the <code>queryStringParameters</code> field of the event object sent by API to Lambda. Again, without the explicit mapping configuration.</li>
</ul>
</div>
<p>Here, you can read more about the Route specification of HTTP API and how to transform requests and responses from the API side if you need to adjust something:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-routes.html">Working with routes for HTTP APIs</a></li>
<li><a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-parameter-mapping.html">Transforming API requests and responses</a></li>
</ul>
<p>Now back to Terraform. Below is the code snippet that illustrates the call of the API Route Terraform module:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">module</span> <span style="color:#a5d6ff">"route_calendars"</span> {
</span></span><span style="display:flex;"><span> source = <span style="color:#a5d6ff">"./modules/api-gateway-route"</span>
</span></span><span style="display:flex;"><span> api_id = aws_apigatewayv2_api.this.id
</span></span><span style="display:flex;"><span> route_key = <span style="color:#a5d6ff">"GET /calendars/{calendar-name}"</span>
</span></span><span style="display:flex;"><span> api_gw_execution_arn = aws_apigatewayv2_api.this.execution_arn
</span></span><span style="display:flex;"><span> lambda_invocation_arn = module.lambda_calendar_backend.lambda.invoke_arn
</span></span><span style="display:flex;"><span> lambda_function_name = module.lambda_calendar_backend.lambda.function_name
</span></span><span style="display:flex;"><span> authorizer_id = aws_apigatewayv2_authorizer.header_based_authorizer.id
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>This module logically relies on both the HTTP API and Lambda resources to configure their integration by implementing the Route.</p>
<h2 id="enhancing-security-and-monitoring-of-aws-api-gateway-v2-http-api">Enhancing Security and Monitoring of AWS API Gateway V2 HTTP API</h2>
<p>Several additional options are available to monitor and protect the HTTP API: logs, metrics, and throttling.</p>
<h3 id="overview-of-http-api-monitoring-and-protection-options">Overview of HTTP API monitoring and protection options</h3>
<p>Logging, metrics, and throttling are configured on the Stage level but allow configuration granularity for the Routes.</p>
<p>For logs, you can configure the CloudWatch log group, the log format (JSON, CLF, XML, CSV), and content filters. The <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-logging-variables.html">logging variables</a> allow you to customize the information that appears in logs. I will provide an example of such a configuration later in the article.</p>
<p>By default, API Gateway sends only API and stage-level metrics to CloudWatch in one-minute periods. However, you can enable <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-metrics.html">detailed metrics</a> and additionally collect the per-route metrics.</p>
<p>To safeguard your HTTP API from excessive requests, you can employ throttling settings, which allow you to set limits per individual route as well as for all routes collectively.</p>
<h3 id="configuring-monitoring-and-protection-for-http-api-with-terraform">Configuring monitoring and protection for HTTP API with Terraform</h3>
<p>Now, let’s see how Terraform helps configure the protection and monitoring for HTTP API.</p>
<p>As mentioned earlier, API Gateway applies these configurations at the Stage level, which is why the aws_apigatewayv2_stage resource encapsulates them all.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_stage"</span> <span style="color:#a5d6ff">"default"</span> {
</span></span><span style="display:flex;"><span> api_id = aws_apigatewayv2_api.this.id
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"$default"</span>
</span></span><span style="display:flex;"><span> auto_deploy = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> description = <span style="color:#a5d6ff">"Default stage (i.e., Production mode)"</span>
</span></span><span style="display:flex;"><span> default_route_settings {
</span></span><span style="display:flex;"><span> throttling_burst_limit = <span style="color:#a5d6ff">1</span>
</span></span><span style="display:flex;"><span> throttling_rate_limit = <span style="color:#a5d6ff">1</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> access_log_settings {
</span></span><span style="display:flex;"><span> destination_arn = aws_cloudwatch_log_group.api_gateway_logs_inkyframe.arn
</span></span><span style="display:flex;"><span> format = jsonencode({
</span></span><span style="display:flex;"><span> authorizerError = <span style="color:#a5d6ff">"$context.authorizer.error"</span>,
</span></span><span style="display:flex;"><span> identitySourceIP = <span style="color:#a5d6ff">"$context.identity.sourceIp"</span>,
</span></span><span style="display:flex;"><span> integrationError = <span style="color:#a5d6ff">"$context.integration.error"</span>,
</span></span><span style="display:flex;"><span> integrationErrorMessage = <span style="color:#a5d6ff">"$context.integration.errorMessage"</span>
</span></span><span style="display:flex;"><span> integrationLatency = <span style="color:#a5d6ff">"$context.integration.latency"</span>,
</span></span><span style="display:flex;"><span> integrationRequestId = <span style="color:#a5d6ff">"$context.integration.requestId"</span>,
</span></span><span style="display:flex;"><span> integrationStatus = <span style="color:#a5d6ff">"$context.integration.integrationStatus"</span>,
</span></span><span style="display:flex;"><span> integrationStatusCode = <span style="color:#a5d6ff">"$context.integration.status"</span>,
</span></span><span style="display:flex;"><span> requestErrorMessage = <span style="color:#a5d6ff">"$context.error.message"</span>,
</span></span><span style="display:flex;"><span> requestErrorMessageString = <span style="color:#a5d6ff">"$context.error.messageString"</span>,
</span></span><span style="display:flex;"><span> requestId = <span style="color:#a5d6ff">"$context.requestId"</span>,
</span></span><span style="display:flex;"><span> routeKey = <span style="color:#a5d6ff">"$context.routeKey"</span>,
</span></span><span style="display:flex;"><span> })
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Here, I applied the default <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-throttling.html">throttling settings</a>: for my project, 1 request per second was enough at that point.</p>
<p>π€ There is a nuance, though, that makes Terraforming API Gateway a little inconvenient β the IAM role that allows API to write logs must be defined on a region level. Therefore, if you maintain several Terraform projects for the same AWS account, you might need to have the following configuration stand separately to avoid conflicts or misunderstandings:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_api_gateway_account"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> cloudwatch_role_arn = aws_iam_role.api_gateway_cloudwatch_logs.arn
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_iam_role"</span> <span style="color:#a5d6ff">"api_gateway_cloudwatch_logs"</span> {
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"api-gateway-cloudwatch-logs"</span>
</span></span><span style="display:flex;"><span> assume_role_policy = jsonencode({
</span></span><span style="display:flex;"><span> Version = <span style="color:#a5d6ff">"2012-10-17"</span>
</span></span><span style="display:flex;"><span> Statement = [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> Effect = <span style="color:#a5d6ff">"Allow"</span>
</span></span><span style="display:flex;"><span> Principal = {
</span></span><span style="display:flex;"><span> Service = <span style="color:#a5d6ff">"apigateway.amazonaws.com"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> Action = <span style="color:#a5d6ff">"sts:AssumeRole"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> })
</span></span><span style="display:flex;"><span> managed_policy_arns = [<span style="color:#a5d6ff">"arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"</span>]
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudwatch_log_group"</span> <span style="color:#a5d6ff">"api_gateway_logs_inkyframe"</span> {
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"/aws/apigateway/inkyframe"</span>
</span></span><span style="display:flex;"><span> log_group_class = <span style="color:#a5d6ff">"STANDARD"</span>
</span></span><span style="display:flex;"><span> retention_in_days = <span style="color:#a5d6ff">7</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And one more thing about HTTP API deployments and stages. I use the special <code>$default</code> keyword to have a single stage (hence, the default one), and I also used automatic deployments: with any change made to API configuration, AWS will automatically generate a new Deployment and bound it with the Stage. If you prefer controlling deployments manually, there is a special resource exists that implements this β <code>aws_apigatewayv2_deployment</code></p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_apigatewayv2_deployment"</span> <span style="color:#a5d6ff">"example"</span> {
</span></span><span style="display:flex;"><span> api_id = aws_apigatewayv2_api.example.id
</span></span><span style="display:flex;"><span> description = <span style="color:#a5d6ff">"Example deployment"</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> triggers = {
</span></span><span style="display:flex;"><span> redeployment = sha1(join(<span style="color:#a5d6ff">","</span>, tolist([
</span></span><span style="display:flex;"><span> jsonencode(aws_apigatewayv2_integration.example),
</span></span><span style="display:flex;"><span> jsonencode(aws_apigatewayv2_route.example),
</span></span><span style="display:flex;"><span> ])))
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> lifecycle {
</span></span><span style="display:flex;"><span> create_before_destroy = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In that case, the <code>aws_apigatewayv2_stage</code> resource requires the <code>deployment_id</code> argument to link itself with a particular Deployment and, therefore, represent the state of the API configuration.</p>
<p>Also, API Gateway requires at least one configured API Route before the deployment is initiated/created. However, these resources do not explicitly depend on each other via attribute references. To avoid the race condition in Terraform, you need to reference the Route resource in the <code>aws_apigatewayv2_deployment</code> resource via the <code>triggers</code> argument (as shown above) or via the <code>depends_on</code> meta-argument. Otherwise, Terraform will try to apply changes to both resources simultaneously.</p>
<h2 id="afterword-simplifying-serverless-architectures">Afterword: Simplifying Serverless Architectures</h2>
<p>In wrapping up our exploration of AWS HTTP API Gateway, AWS Lambda, and Terraform, we’ve delved into how these powerful tools work in tandem to streamline and enhance serverless architectures. This article aimed to combine my experience with new knowledge and demystify the complexities of used services, showcasing their capabilities in creating efficient, cost-effective solutions for modern cloud-based applications.</p>
<p>We focused on practical implementation and the tangible benefits of combining these technologies. By leveraging Terraform, we’ve seen how infrastructure management can be simplified, allowing for clearer, more maintainable code. The combination of AWS Lambda and HTTP API Gateway has demonstrated the efficiency of serverless computing, offering scalability and performance without the burden of extensive configuration and management.</p>
<p>This exploration underlines the importance of choosing the right tools and strategies in cloud computing. It reminds developers and architects that creating robust and efficient serverless systems is within reach with a thoughtful approach and the right set of tools. As the cloud landscape continues to evolve, staying informed and adaptable is key to harnessing the full potential of these technologies. π</p>
Hello Terraform Data; Goodbye Null Resource
2023-04-15T23:25:18Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2023/04/16/hello-terraform-data-goodbye-null-resource/
Serhii Vasylenko
https://devdosvid.blog/2023/04/16/hello-terraform-data-goodbye-null-resource/cover-image.png
<p>Terraform version 1.4 was recently released and brought a range of new features, including improved run output in Terraform Cloud, the ability to use OPA policy results in the CLI, and a built-in alternative to the null resource β terraform_data.</p>
<p>In this blog post, I want to demonstrate and explain the <strong>terraform_data</strong> resource that serves two purposes:</p>
<ul>
<li>firstly, it allows arbitrary values to be stored and used afterward to implement lifecycle triggers of other resources</li>
<li>secondly, it can be used to trigger provisioners when there isn’t a more appropriate managed resource available.</li>
</ul>
<div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<p>For those of you, who are familiar with the null provider, the <code>terraform_data</code> resource might look very similar. And you’re right!<br>
Rather than being a separate provider, the terraform_data managed resource now offers the same capabilities as an integrated feature. Pretty cool! <br>
While the null provider is still available and there are no statements about its deprecation thus far (<a href="https://registry.terraform.io/providers/hashicorp/null/3.2.1/docs">as of April 2023, v3.2.1</a>), the <code>terraform_data</code> is the native replacement of the <code>null_resource</code>, and the latter might soon become deprecated.</p>
<p>The <code>terraform_data</code> resource has two optional arguments, <strong>input</strong> and <strong>triggers_replace</strong>, and its configuration looks as follows:</p>
<figure>
<img loading="lazy"
src="code-snippet-1.png"
alt="terraform data resource arguments"width="800"
height="182.16"
/> <figcaption>
<p>terraform data resource arguments</p>
</figcaption>
</figure>
<ul>
<li>The <code>input</code> (optional) stores the value that is passed to the resource</li>
<li>The <code>triggers_replace</code> (optional) is a value that triggers resource replacement when changes.</li>
</ul>
<p>There are two attributes, in addition to the arguments, which are stored in the state: <strong>id</strong> and <strong>output</strong> after the resource is created. Let’s take a look:</p>
<figure>
<img loading="lazy"
src="code-snippet-2.png"
alt="terraform data resource attributes"width="800"
height="271.36"
/> <figcaption>
<p>terraform data resource attributes</p>
</figcaption>
</figure>
<ul>
<li>The <code>output</code> attribute is computed based on the value of the <code>input</code></li>
<li>The <code>id</code> is just a unique value of the resource instance in the state (as for any other resource).</li>
</ul>
<h2 id="use-case-for-terraform_data-with-replace_triggered_by">Use case for terraform_data with replace_triggered_by</h2>
<p>Let’s take a look at the first use case for the terraform_data resource. It is the ability to trigger resource replacement based on the value of the input argument.</p>
<p>A bit of context here: the <strong>replace_triggered_by</strong> argument of the resource lifecycle meta-argument allows you to trigger resource replacement based on another referenced resource or its attribute.</p>
<div class="attention">
If you are not yet familiar with the <code>replace_triggered_by</code>, you can check <a href="https://devdosvid.blog/2022/05/04/new-lifecycle-options-and-refactoring-capabilities-in-terraform-1-1-and-1-2/#trigger-resource-replacement-with-replace_triggered_by">another blog post that explains it</a>.
</div>
<p>The replace_triggered_by is a powerful feature, but here is the thing about it: only a resource or its attribute must be specified, and <strong>you cannot use a variable or a local value for replace_triggered_by</strong>.</p>
<p>But with terraform_data, you can indirectly initiate another resource replacement by using either a variable or an expression within a local value for the <code>input</code> argument.</p>
<p>Let me give you an example here. Consider the following code:</p>
<figure>
<img loading="lazy"
src="code-snippet-3.png"
alt="trigger replacement based on input variable"width="800"
height="386.00"
/> <figcaption>
<p>trigger replacement based on input variable</p>
</figcaption>
</figure>
<p>By modifying the <code>revision</code> variable, the next Terraform plan will suggest a replacement action against aws_instance.webserver:</p>
<figure>
<img loading="lazy"
src="code-snippet-4.png"
alt="terraform_data with replace_triggered_by"width="800"
height="341.59"
/> <figcaption>
<p>terraform_data with replace_triggered_by</p>
</figcaption>
</figure>
<h2 id="use-case-for-terraform_data-with-provisioner">Use case for terraform_data with provisioner</h2>
<p>Before we start: HashiCorp suggests (and I also support that) avoiding provisioner usage unless you have no other options left. One of the reasons β additional, implicit, and unobvious dependency that appears in the codebase β the binary, which is called inside the provisioner block, must be present on the machine. <br>
But let’s be real, the provisioner feature is still kicking, and there’s always that one unique project that needs it.</p>
<p>Here is the code snippet that demonstrates the usage of the provisioner within the terraform_data resource:</p>
<figure>
<img loading="lazy"
src="code-snippet-5.png"
alt="terraform_data with provisioner"width="800"
height="409.00"
/> <figcaption>
<p>terraform_data with provisioner</p>
</figcaption>
</figure>
<p>In this example, the following happens:</p>
<ul>
<li>When resources are created the first time, the provisioner inside <code>terraform_data</code> runs</li>
<li>Sequential plan/apply will trigger another execution of the provisioner only when the private IP of the instance (aws_instance.webserver.private_ip) changes because that will trigger <code>terraform_data</code> recreation. At the same time, no changes to the internal IP mean no provisioner execution.</li>
</ul>
<hr>
<p>With its ability to store and use values for lifecycle triggers and provisioners, <strong>terraform_data</strong> is a powerful tool that can enhance your Terraform configuration.</p>
<p>Although the null provider still has its place in the Terraform ecosystem, terraform_data is its evolution, and its integration as a feature is certainly something to be excited about.</p>
<p>Why not give it a try in your next project and see how it can simplify your infrastructure as code workflows? Keep on coding! π</p>
Five Practical Ways To Get The Verified EC2 AMI
2022-07-24T13:21:05Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2022/07/24/five-practical-ways-to-get-the-verified-ec2-ami/
Serhii Vasylenko
https://devdosvid.blog/2022/07/24/five-practical-ways-to-get-the-verified-ec2-ami/cover-image.png
<p>EC2 AMI catalog consists of more than 160k public AMIs β a mix of shared AMIs created by users, published by vendors, and provided by AWS.</p>
<p>So how to ensure that an AMI comes from the verified vendor or that is an official AMI published by AWS?</p>
<p>How to find the trusted AMI among them all when you’re about to launch an EC2 Instance?</p>
<figure>
<img loading="lazy"
src="who-is-ami-owner.png"
alt="Find the Waldo verified AMI owner"width="800"
height="400.00"
/> <figcaption>
<p>Find the <del>Waldo</del> verified AMI owner</p>
</figcaption>
</figure>
<p>On AWS, it’s typical that something can be made or done in several ways β that’s awesome. Some of them work better than others, some methods are official, and some you can use just for fun (<a href="https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/">check</a> <a href="https://www.lastweekinaws.com/blog/17-more-ways-to-run-containers-on-aws/">that</a>).</p>
<p>In this article, I will describe five ways of getting the official and verified AMI for your next EC2 Instance launch.</p>
<h2 id="use-ec2-launch-wizard-and-ami-catalog-to-get-the-official-ami">Use EC2 Launch Wizard and AMI Catalog to get the official AMI</h2>
<p>When launching an EC2 Instance from a Management Console, you can apply the “Verified Provider” filter for the Community AMIs tab to ensure you get an AMI from a verified provider. The “Verified provider” label means an AMI is owned by an Amazon verified account.</p>
<p>In the following example, I want to make sure that the Ubuntu 20.04 AMI comes from the verified source:</p>
<figure>
<img loading="lazy"
src="verified-ami-in-ami-catalog.png"
alt="Verified AMI in the AMI Catalog"width="800"
height="413.65"
/> <figcaption>
<p>Verified AMI in the AMI Catalog</p>
</figcaption>
</figure>
<p>In the past, you had to compare the AMI Owner ID with the publicly shared list of verified Owner IDs for every region. Not rocket science, but it takes time. So now it’s much more straightforward, thanks to the “Verified Provider” label.</p>
<p>This feature also works great when you are creating a Launch Template. The Launch Template creation wizard seamlessly guides you from itself to the AMI Catalog (where you can search and pick the AMI) and back again.</p>
<h2 id="look-for-verified-amis-on-the-ami-page">Look for verified AMIs on the AMI page</h2>
<p>Another interface in the Management Console acts as the AMI browser. It does not have any fancy name except for the “AMIs page”, but you probably already know about it: it looks like a list of AMIs, and you can see it when you click on the “AMIs” menu item on the left side of the EC2 page menu.</p>
<p>The AMI page allows you to leverage the API filters to narrow down the search, and the “Owner alias” filter is the one you need to ensure that an AMI comes from a trusted owner.</p>
<p>Here is how it looks for my search of the official Amazon Linux 2 AMI:</p>
<figure>
<img loading="lazy"
src="verified-amazon-linux-2-ami-in-ami-browser-ec2-console.png"
alt="Official Amazon Linux 2 AMI"width="800"
height="400.00"
/> <figcaption>
<p>Official Amazon Linux 2 AMI</p>
</figcaption>
</figure>
<p>AMIs shared by verified sources have <code>amazon</code> (for AWS) or <code>aws-marketplace</code> (for AWS partners) as the value for the Owner alias filter.</p>
<h2 id="find-the-ec2-ami-with-terraform">Find the EC2 AMI with Terraform</h2>
<p>Finding the official AMI with Terraform is also simple β the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami">aws_ami data source</a> does the job.</p>
<p>For example, here is how you can find the same Amazon Linux 2 AMI by specifying the <code>amazon</code> as the value for the <code>owner</code> argument of the data source:</p>
<figure>
<img loading="lazy"
src="verified-official-amazon-linux-2-ami-terraform-datasource.png"
alt="Finding the official Amazon Linux 2 AMI with Terraform"width="800"
height="580.24"
/> <figcaption>
<p>Finding the official Amazon Linux 2 AMI with Terraform</p>
</figcaption>
</figure>
<p>Compare that with the filters on the AMI page β it looks similar, right? This is because of how Terraform works: it translates your code into API calls and sends them to AWS API endpoints.</p>
<p>If you’re very new to Terraform, I suggest reading this article to understand the basic concepts of Terraform and Infrastructure as Code: <a href="https://devdosvid.blog/2020/05/02/terraform-explained-in-english/">Terraform explained in English</a></p>
<h2 id="find-the-official-aws-ami-using-describe-images-cli">Find the official AWS AMI using Describe Images CLI</h2>
<p>Sometimes you might need to get the AMI from CLI to pass it along as an argument downstream of the pipeline.</p>
<p>This can be done with the <strong>ec2 describe-images</strong> command of the AWS CLI
<figure>
<img loading="lazy"
src="verified-official-amazon-linux-2-ami-in-aws-cli.png"
alt="Find the verified AMI with AWS CLI"width="800"
height="486.75"
/> <figcaption>
<p>Find the verified AMI with AWS CLI</p>
</figcaption>
</figure>
</p>
<p>The API filters I mentioned before also work here β use them to narrow your search.</p>
<h2 id="find-the-trusted-aws-ami-with-ssm">Find the trusted AWS AMI with SSM</h2>
<p>Another way that involves AWS CLI is the <strong>ssm get-parameter</strong> command:
<figure>
<img loading="lazy"
src="verified-official-amazon-eks-optimized-ami-aws-cli.png"
alt="Get the latest EKS optimized AMI from SSM"width="800"
height="213.69"
/> <figcaption>
<p>Get the latest EKS optimized AMI from SSM</p>
</figcaption>
</figure>
</p>
<p>It reveals one helpful feature of the Systems Manager β the Public parameters.</p>
<p>Systems Manager Public parameters are how AWS distributes some widely used artifacts related to their services.</p>
<p>For example, you can find official AMIs for many distributives there: Amazon Linux, Windows, macOS, Bottlerocket, Ubuntu, Debian, and FreeBSD.</p>
<p>Read more at the <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-finding-public-parameters.html">Finding public parameters</a> documentation page if you want to know more.</p>
<h2 id="are-all-verified-amis-good">Are all verified AMIs good?</h2>
<p>The “Verified provider” badge can be earned by a third party only when an AMI developer is registered as a Seller on the AWS Marketplace.</p>
<p>Becoming a Seller there is not trivial and requires some <a href="https://docs.aws.amazon.com/marketplace/latest/userguide/user-guide-for-sellers.html#seller-requirements-for-publishing-free-products">conditions</a> to be met, and the <a href="https://docs.aws.amazon.com/marketplace/latest/userguide/seller-registration-process.html">registration process</a> itself also implies submitting the tax and banking information.</p>
<p>Additionally, there are <a href="https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html">specific policies and review processes</a> apply to all AMIs submitted to the Marketplace.</p>
<p>So it is okay to trust the third-party vendors with the “Verified” badge on a certain level. However, it is also always good to have additional scans and validation of the software you use. πͺ² π</p>
Golden Image Pipelines With HCP Packer
2022-06-26T13:13:12Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2022/06/26/golden-image-pipelines-with-hcp-packer/
Serhii Vasylenko
https://devdosvid.blog/2022/06/26/golden-image-pipelines-with-hcp-packer/cover-image.png
<p>Many of us know and use Packer to build golden images for cloud providers. But did you know that Packer is not just a CLI tool?</p>
<p>There is an HCP (<em>stands for <strong>H</strong>ashiCorp <strong>C</strong>loud <strong>P</strong>latform</em>) Packer that acts as the image registry that stores the image metadata, allows you to organize images in distribution channels, and perform other management actions.</p>
<p>In this blog, I would like to showcase some features of the HCP Packer and explain how you can use it to set up an image factory for the organization (or for your own fun π) to maintain the Golden Images.</p>
<p>I will be using AWS AMI as the OS image appliance for examples in this blog, but Packer supports many other formats and clouds through its <a href="https://www.packer.io/plugins">plugins</a>.</p>
<h2 id="hcp-packer-registry">HCP Packer Registry</h2>
<p>HCP Packer is the image metadata registry that stores the information (not an image file) about OS images you create using the Packer CLI tool.</p>
<p>It solves the challenge of the Golden Image pipeline maintenance by acting as a hub that organizes and streamlines the processes of OS image creation, usage, and continuity.</p>
<figure>
<img loading="lazy"
src="hcp-packer-how-it-works.png"
alt="OS Image lifecycle with HCP Packer"width="800"
height="400"
/> <figcaption>
<p>OS Image lifecycle with HCP Packer</p>
</figcaption>
</figure>
<p>HCP Packer introduces several new concepts that compose the registry: Image Buckets, Iterations, and Channels. Further in this blog, I will explain them, but let’s start with security first.</p>
<h2 id="security-first--creating-service-principals">Security First β Creating Service Principals</h2>
<p>Before launching the builds, you need to create a Service Principal to allow your local Packer CLI to communicate with the HCP.</p>
<p>I recommend creating at least two principals: the one with the “contributor” role β used by Packer CLI to store the image metadata in HCP; and another one with the “viewer” role β used by Terraform (as it requires only read-level access for Packer HCP).</p>
<figure>
<img loading="lazy"
src="hcp-packer-service-principals.png"
alt="Service Principals for HCP"width="800"
height="237.88"
/> <figcaption>
<p>Service Principals for HCP</p>
</figcaption>
</figure>
<p>Once you have created a principal, you can generate a key for authentication. The key consists of an ID and a secret.</p>
<div class="attention">
<p>Both the Packer CLI and the Packer Terraform provider support environment variables for the principal client ID and client secret for authentication:</p>
<p><code>HCP_CLIENT_ID</code> and <code>HCP_CLIENT_SECRET</code></p>
</div>
<h2 id="image-buckets-to-store-image-metadata">Image Buckets to Store Image Metadata</h2>
<p>The central entity in HCP Packer is the Image Buckets.</p>
<p><strong>Image Bucket</strong> is a repository where the metadata from a Packer template is stored once image(s) creation is completed.</p>
<p>Image Bucket can contain a single image or several images if you define several sources for the <code>build</code> block in the Packet template.</p>
<p>For example, a bucket can span several custom AMIs based on Ubuntu AMI provided by Amazon and built and distributed within several regions.</p>
<p>You cannot create buckets manually from the web interface (at least as of June 2022), but I will show you how they are defined as code inside a Packer template file just a bit later.</p>
<figure>
<img loading="lazy"
src="image-buckets.png"
alt="HCP Packer Image Buckets"width="800"
height="278.87"
/> <figcaption>
<p>HCP Packer Image Buckets</p>
</figcaption>
</figure>
<h2 id="iterations-of-image-creation">Iterations of Image Creation</h2>
<p>Every execution of the <code>build</code> action made by Packer CLI (if used in conjunction with HCP) is recorded specially and called <strong>Iteration</strong>.</p>
<p>Each Iteration has a unique fingerprint β an SHA value of the head reference in the Git repository that contains your Packer template.</p>
<div class="attention">
Tip: you can override that with the <code>HCP_PACKER_BUILD_FINGERPRINT</code> env variable if you want to set the Iteration ID manually.
</div>
<figure>
<img loading="lazy"
src="packer-iterations.png"
alt="HCP Packer Iterations"width="800"
height="278.87"
/> <figcaption>
<p>HCP Packer Iterations</p>
</figcaption>
</figure>
<p>Every Iteration consists of at least one Build β another special record that contains image metadata produced by Packer CLI.</p>
<p>The Builds inside Iteration are represented by the number of sources specified in your Packer template’s <code>build</code> section.</p>
<figure>
<img loading="lazy"
src="packer-iteration-builds.png"
alt="HCP Packer Iteration Builds"width="800"
height="574.57"
/> <figcaption>
<p>HCP Packer Iteration Builds</p>
</figcaption>
</figure>
<h2 id="packer-template-configuration-for-hcp">Packer Template Configuration for HCP</h2>
<p>Let’s now review a code example to understand how all this combines.</p>
<p>Here is a <code>build</code> block from Packer template file.</p>
<figure>
<img loading="lazy"
src="packer-template-example.png"
alt="HCP Packer registry usage in a Packer template"width="800"
height="554.00"
/> <figcaption>
<p>HCP Packer registry usage in a Packer template</p>
</figcaption>
</figure>
<p>Look at the <code>hcp_packer_registry</code> block: it defines the Bucket where Packer will store image information and custom labels for the Bucket and the image.</p>
<p>The <code>bucket_name</code> defines my Image Bucket: Packer will either use the existing Bucket with that name or create a new one if it does not exist.</p>
<p>The <code>bucket_labels</code> map defines custom labels you specify for an Image Bucket. In my example, I set the Bucket owner and the OS name.</p>
<p>The <code>build_labels</code> map defines custom labels for the Builds within the Iteration inside a bucket.</p>
<p>And because I define two <code>sources</code> here, my Iteration will have two Builds inside it.</p>
<h2 id="using-channels">Using Channels</h2>
<p>Although all Iterations have unique identifiers, giving a familiar name to some of them would be more convenient.</p>
<p><strong>Channel</strong> is a way to assign a specific Iteration to a friendly name that you can use later:</p>
<ul>
<li>in other Packer templates, if you want to use your custom image as the base for other images</li>
<li>in Terraform code (we will review this further) to reference the image by the channel name, avoiding the hard code of the image ID.</li>
</ul>
<p>Channels are created through the web interface or using the API. And I hope HashiCorp will add HCP Packer resources to the HCP Terraform provider in the future so channel creation can be described as code.</p>
<figure>
<img loading="lazy"
src="hcp-packer-image-channel.png"
alt="HCP Packer Image Channel"width="800"
height="338.15"
/> <figcaption>
<p>HCP Packer Image Channel</p>
</figcaption>
</figure>
<p>You can manually promote an Iteration to a channel with a web interface.</p>
<p>But before promoting an Iteration to a channel, you might want to perform the following:</p>
<ul>
<li>
<p>test and validate the newly created image before its promotion to a channel: create a temporary virtual machine using Terraform and ensure it successfully boots from the image.</p>
</li>
<li>
<p>assess that VM with some vulnerability scanning service. For example, if you’re an AWS customer, then <a href="https://docs.aws.amazon.com/inspector/latest/user/what-is-inspector.html">Amazon Inspector</a> might work for you in such a case.</p>
</li>
</ul>
<p>Once an image from the Iteration is validated and passed the security assessment, it’s safe to promote that Iteration to a channel.</p>
<p>HCP Packer provides a rich <a href="https://cloud.hashicorp.com/api-docs/packer">API</a> that you can leverage to automate that process.</p>
<p>When a <code>packer build</code> successfully finishes its execution, it returns the Iteration ID (ULID) that you can use later for an API call with a request to promote the new Iteration to a channel.</p>
<figure>
<img loading="lazy"
src="packer-build-output.png"
alt="Packer build output with Iteration ULID"width="800"
height="218.00"
/> <figcaption>
<p>Packer build output with Iteration ULID</p>
</figcaption>
</figure>
<p>The “Update Channel” PATCH API method is needed to assign the Iteration to a channel.</p>
<p>First, you need to obtain the access token as described in <a href="https://support.hashicorp.com/hc/en-us/articles/6676505991699-HCP-API-Authentication-with-Curl">this guide</a>.</p>
<p>Then, the following cURL request can be used to update the channel with a new Iteration ULID (please expand the code snippet below):</p>
<div class="code-snippet">
<details>
<summary markdown="span">Click here to see the code snippet</summary>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#79c0ff">HCP_ACCESS_TOKEN</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"your token here"</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#79c0ff">HCP_ORG_ID</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"your org id here"</span>
</span></span><span style="display:flex;"><span><span style="color:#79c0ff">HCP_PROJECT_ID</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"your project id here"</span>
</span></span><span style="display:flex;"><span><span style="color:#79c0ff">HCP_BUCKET_NAME</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"amazon-linux2"</span>
</span></span><span style="display:flex;"><span><span style="color:#79c0ff">HCP_CHANNEL_NAME</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"stable"</span>
</span></span><span style="display:flex;"><span><span style="color:#79c0ff">HCP_BASE_URL</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"https://api.cloud.hashicorp.com/packer/2021-04-30"</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>curl -X PATCH <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span>--header <span style="color:#a5d6ff">"Authorization: Bearer </span><span style="color:#79c0ff">$HCP_ACCESS_TOKEN</span><span style="color:#a5d6ff">"</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span>--data <span style="color:#a5d6ff">'{
</span></span></span><span style="display:flex;"><span><span style="color:#a5d6ff">"incremental_version":"3",
</span></span></span><span style="display:flex;"><span><span style="color:#a5d6ff">"iteration_id":"01H8V7WBDWRBCMZDZ2HG3MKSDL"
</span></span></span><span style="display:flex;"><span><span style="color:#a5d6ff">}'</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span><span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">HCP_BASE_URL</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/organizations/</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">HCP_ORG_ID</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/projects/</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">HCP_PROJECT_ID</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/images/</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">HCP_BUCKET_NAME</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/channels/</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">HCP_CHANNEL_NAME</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span></code></pre></div>
</details>
</div>
<h2 id="using-hcp-packer-with-terraform">Using HCP Packer with Terraform</h2>
<p>Having a streamlined golden image creation process is good. Still, it would be even better to have an easy way to always use the latest validated image without hardcoding or some other duck taping.</p>
<p>With the help of the <a href="https://registry.terraform.io/providers/hashicorp/hcp">HCP Terraform provider</a>, you can reference the image channel in your Terraform code and have a completed end-to-end workflow.</p>
<p>Here is an example of the Terraform configuration that uses the HCP Packer registry as the source of AMI ID for an AWS instance:</p>
<figure>
<img loading="lazy"
src="terraform-hcp-packer.png"
alt="Using HCP Packer registry with Terraform"width="800"
height="529.00"
/> <figcaption>
<p>Using HCP Packer registry with Terraform</p>
</figcaption>
</figure>
<p>Two data sources do all the magic here.</p>
<p>The <code>hcp_packer_iteration</code> data source gets the most recent Iteration assigned to the specified Channel (i.e., latest). We need that because the Iteration (not the Channel) holds the image information.</p>
<p>Then the <code>hcp_packer_image</code> gets the cloud image ID (AWS AMI ID in my example) from that Iteration so you can use it later in your code.</p>
<p>The configuration of the <code>hcp</code> provider in this example is empty on purpose: this provider supports <code>HCP_CLIENT_ID</code> and <code>HCP_CLIENT_SECRET</code> env variables to use their values for the <a href="https://registry.terraform.io/providers/hashicorp/hcp/latest/docs/guides/auth">authentication</a> and avoid hard coding. Alternatively, you can use the <code>client_id</code> and <code>client_secret</code> options to configure the provider.</p>
<h2 id="image-revocation">Image revocation</h2>
<p>It is possible to revoke a specific Iteration, and therefore all Images in it, to alert the users about the Image decommission. For example, your SecOps team can revoke it due to the new CVE announced.</p>
<p>Revoked Images are treated differently by Packer CLI and Terraform CLI.</p>
<h3 id="packer-cli-and-revoked-image">Packer CLI and Revoked Image</h3>
<p>When you reference the Image in a Packer template to use it as a source for another image, its revocation makes further Packer builds to fail.</p>
<p>In other words, Packer won’t let you build a new Image on top of the revoked Image.</p>
<h3 id="terraform-cli-and-revoked-image">Terraform CLI and Revoked Image</h3>
<p>On the contrary, Terraform CLI does not prevent the usage of the revoked Image by default, although its Cloud version does it if used with the “Run tasks” feature.</p>
<p>Although you can get the Image ID, when the Iteration is revoked, the <code>hcp_packer_image</code> data source returns a non-empty <code>revoke_at</code> attribute with the value set to the revocation timestamp.</p>
<p>Therefore, you can use the <code>precondition</code> (available in <a href="https://devdosvid.blog/2022/05/04/new-lifecycle-options-and-refactoring-capabilities-in-terraform-1-1-and-1-2/#precondition-and-postcondition">Terraform CLI v1.2.0</a> and higher) to validate the Image with Terraform CLI and make sure it was not revoked</p>
<p>Here is the code example that illustrates that:</p>
<figure>
<img loading="lazy"
src="logic-for-revoked-iteration.png"
alt="Work with revoked HCP Packer image in Terraform"width="800"
height="501.24"
/> <figcaption>
<p>Work with revoked HCP Packer image in Terraform</p>
</figcaption>
</figure>
<h2 id="why-hcp-packer">Why HCP Packer?</h2>
<p>So what makes the HCP Packer a good fit and worth a try?</p>
<p>1οΈβ£ A centralized place to view and manage the OS images throughout an organization. And as for me, it is good to have a neat web panel to look at things.</p>
<p>2οΈβ£ Image Channels that help with logical organization and control.</p>
<p>3οΈβ£ Ability to revoke an image to prevent its usage.</p>
<p>4οΈβ£ API and Terraform provider as additional tools that enrich the user experience.</p>
<p>When dealing with multiple golden images or with various cloud providers, the <a href="https://cloud.hashicorp.com/products/packer">HCP Packer</a> can be a good fit for your image pipeline.</p>
<p>As a registry, it enables the end-to-end workflow for golden image usage: create, validate, use and decommission the images in a centralized way.</p>
<p>And no more hard-coded IDs, manual variable settings, or other duck tape and glue in your Terraform.</p>
<p>If you want to learn more about HCP Packer and have some practice, I suggest starting from the <a href="https://learn.hashicorp.com/collections/packer/hcp">tutorial at HashiCorp Learn portal</a>.</p>
New Lifecycle Options and Refactoring Capabilities in Terraform 1.1 and 1.2
2022-05-04T20:27:47Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2022/05/04/new-lifecycle-options-and-refactoring-capabilities-in-terraform-1-1-and-1-2/
Serhii Vasylenko
https://devdosvid.blog/2022/05/04/new-lifecycle-options-and-refactoring-capabilities-in-terraform-1-1-and-1-2/cover-image.png
<p>In this blog, I would like to tell you about new cool features that Terraform 1.1 and 1.2 bring. It feels like Terraform has doubled its speed of delivering the new features after they released the 1.0. π€©</p>
<p>It’s been only a few months since Terraform 1.1 was released with the <code>moved</code> block that empowers the code refactoring.</p>
<p>Now Terraform 1.2 is almost <a href="*https://github.com/hashicorp/terraform/releases/tag/v1.2.0-rc1*">ready</a> (as I am writing this blog in early May 2022) to bring three new efficient controls to the resource lifecycle.<br>
These are three new expressions: <code>precondition</code>, <code>postcondition</code>, and <code>replace_triggered_by</code>.</p>
<div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<h2 id="terraform-code-refactoring-with-the-moved-block">Terraform Code Refactoring With the Moved Block</h2>
<p>Starting from the 1.1 version, Terraform users can use the <code>moved</code> block to describe the changes in resource or module addresses (or resources inside a module) in the form of code. <br>
Once that is described, Terraform performs the movement of the resource within the state during the first apply.</p>
<p>In other words, what this feature gives you, is the ability to document your <code>terraform state mv</code> actions, so you and other project or module users don’t need to perform them manually.</p>
<p>As your code evolves, a resource or module can have several <code>moved</code> blocks associated with it, and Terraform will thoroughly reproduce the whole history of its movement within a state (i.e., renaming).</p>
<p>Let’s review some examples that illustrate how it works.</p>
<h3 id="move-or-rename-a-resource">Move or rename a resource</h3>
<p>In a module, I have a bucket policy that has a generic, meaningless name. It is used in a module that creates a CloudFront distribution with an S3 bucket.</p>
<figure>
<img loading="lazy"
src="figure-1.png"
alt="An example resource"width="800"
height="150"
/> <figcaption>
<p>An example resource</p>
</figcaption>
</figure>
<p>It’s pretty OK to name a resource like that if you have only a single instance of that kind in your code.</p>
<p>Later, when I need to add another policy to the module, I don’t want to name it “that”. Instead, I want my policies to have meaningful names now.<br>
For example, I could rename the old policy with the <code>terraform state mv</code> command, but other users of my module would not know about that.</p>
<p>That is where the <code>moved</code> block turns out to be helpful!</p>
<div class="attention">
The <code>moved</code> block allows you to document how you rename or move an object in Terraform so that other code users can have the same changes afterward.
</div>
<figure>
<img loading="lazy"
src="figure-2.png"
alt="Resource address update with the Moved block"width="800"
height="270"
/> <figcaption>
<p>Resource address update with the Moved block</p>
</figcaption>
</figure>
<p>Terraform follows the instructions inside the <code>module</code> block to plan and apply changes. Although the resource address update is not counted as a change in the Plan output, Terraform will perform that update during apply.</p>
<figure>
<img loading="lazy"
src="figure-3.png"
alt="Terraform plan output"width="800"
height="318"
/> <figcaption>
<p>Terraform plan output</p>
</figcaption>
</figure>
<h3 id="move-or-rename-a-module">Move or rename a module</h3>
<p>The same approach can be applied to a module β you can move or rename it as a code too.</p>
<p>Here, I use two modules to create static hosting for a website with a custom TLS certificate.</p>
<figure>
<img loading="lazy"
src="figure-4.png"
alt="Two modules with generic names"width="800"
height="437"
/> <figcaption>
<p>Two modules with generic names</p>
</figcaption>
</figure>
<p>Again, if I need to add another couple of the CDN+Certificate modules, I would like to have meaningful names in my code so clearly distinguish one from another.</p>
<p>Therefore, I would add two <code>moved</code> blocks β one per module call.</p>
<p>And by the way, since I renamed the module (from <code>cert</code> to <code>example_com_cert</code>), I need to update all references to that module’s outputs in the code too.</p>
<figure>
<img loading="lazy"
src="figure-5.png"
alt="Two modules renamed"width="800"
height="629"
/> <figcaption>
<p>Two modules renamed</p>
</figcaption>
</figure>
<p>However, there is one nuance: when you rename a module and declare that in the <code>moved</code> block, you need to run the <code>terraform init</code> before applying the change because Terraform must initialize the module with the new name first.</p>
<figure>
<img loading="lazy"
src="figure-6.png"
alt="Terraform error: module not installed"width="800"
height="246"
/> <figcaption>
<p>Terraform error: module not installed</p>
</figcaption>
</figure>
<p>There are some more advanced actions you can make with the <code>moved</code> block:</p>
<ul>
<li>Implement count and for_each meta-arguments to resources and modules</li>
<li>Break one module into multiple
Check the following detailed guide from HashiCorp that explains how to do that β <a href="https://www.terraform.io/language/modules/develop/refactoring">Refactoring</a></li>
</ul>
<p>Introducing the <code>moved</code> blocks into your codebase defacto starts the refactoring process for your module users. But the finale of that refactoring happens when you ultimately remove these blocks.</p>
<p>Therefore, here is some advice on how to manage that:</p>
<div class="attention">
<ul>
<li>
<p>Keep the <code>moved</code> blocks in your code for long. For example, when removing a <code>moved</code> block from the code, Terraform does not treat the new object name as a renaming anymore. Instead, Terraform will plan to delete the resource or module with the old name instead of renaming it.</p>
</li>
<li>
<p>Keep the complete chains of object renaming (sequence of moves). The whole history of object movement ensures that users with different module versions will get a consistent and predictable behavior of the refactoring.</p>
</li>
</ul>
</div>
<h2 id="lifecycle-expressions-conditions-and-replacement-trigger">Lifecycle expressions: conditions and replacement trigger</h2>
<p>Terraform 1.2 fundamentally improves the <code>lifecycle</code> meta-argument by adding three new configuration options with rich capabilities.</p>
<figure>
<img loading="lazy"
src="figure-7.png"
alt="New configuration options for the lifecycle meta-argument"width="800"
height="365"
/> <figcaption>
<p>New configuration options for the lifecycle meta-argument</p>
</figcaption>
</figure>
<h3 id="precondition-and-postcondition">Precondition and Postcondition</h3>
<p>When you need to make sure that specific condition is met before or after you create a resource, you can use <code>postcondition</code> and <code>precondition</code> blocks.</p>
<p>The <em>condition</em> here β is some data or information about a resource you need to confirm to apply the code.</p>
<p>Here are a few examples of such conditions:</p>
<ul>
<li>Validate some attributes of the Data Source that you cannot check using filters or other available arguments;</li>
<li>Confirm the Resource argument that can compound several variables (e.g., list);</li>
</ul>
<div class="attention">
<p><strong>Precondition</strong> works as an expectation or a guess about some external (but within a module) value that a resource depends on.</p>
<p><strong>Postcondition</strong> works as the assurance that a resource fulfills a specific condition so other resources may rely on that. If postcondition fails for a resource, this prevents changes to all other resources that depend on it.</p>
</div>
<p>Let’s review this new feature with an example of <code>postcondition</code> usage.</p>
<p>Consider the following case: our module receives AMI ID as the input variable, and that AMI should be used in the Launch Template then; we also have the requirement for the EC2 instance created from that Launch Template β its root EBS size must be equal or bigger than 600 GB.</p>
<p>We cannot validate the EBS size using the variable that accepts the AMI ID. But we can write a <strong>postcondition</strong> for the Data Source that gets the information about the AMI and reference that Data Source in the Launch Template resource afterward.</p>
<figure>
<img loading="lazy"
src="figure-8.png"
alt="Data Source Postcondition"width="800"
height="446"
/> <figcaption>
<p>Data Source Postcondition</p>
</figcaption>
</figure>
<p>The <code>condition</code> argument within the block accepts any of Terraform’s built-in functions or language operators.</p>
<p>The special <code>self</code> object is available only for the <code>postcondition</code> block because it assumes that validation can be performed after the object is created and its attributes are known.</p>
<p>Later, if a module user specifies the AMI with an EBS size lesser than 600 GB, Terraform will fail to create the Launch Template because it depends on the Data Source that did not pass the postcondition check.</p>
<figure>
<img loading="lazy"
src="figure-9.png"
alt="Resource postcondition error"width="800"
height="240"
/> <figcaption>
<p>Resource postcondition error</p>
</figcaption>
</figure>
<p>Terraform tries to evaluate the condition expressions as soonest: sometimes Terraform can check the value during the planning phase, but sometimes that can be done only after the resource is created if the value is unknown.</p>
<h3 id="validate-module-output-with-precondition">Validate module output with precondition</h3>
<p>The <code>precondition</code> block is also available for the module outputs.</p>
<p>Just like the variable validation block assures that module input meets certain expectations, the <code>precondition</code> is intended to ensure that a module produces the valid output.</p>
<p>Here is an example: a module that creates an ACM certificate must prevent the usage of a specific domain name in the certificate’s Common Name or its SANs.</p>
<figure>
<img loading="lazy"
src="figure-10.png"
alt="Module output precondition"width="800"
height="342"
/> <figcaption>
<p>Module output precondition</p>
</figcaption>
</figure>
<p>In this case, instead of validating several input variables, we can write the validation only once for the output.</p>
<div class="attention">
Validation of the module output helps with standardization and control of the data passed between Terraform modules.
</div>
<h3 id="trigger-resource-replacement-with-replace_triggered_by">Trigger resource replacement with replace_triggered_by</h3>
<p>Sometimes it’s needed to specify the dependency in the way that recreates a resource when another resource or its attribute changes.</p>
<p>This is useful when two (or more) resources do not have any explicit dependency.</p>
<p>Consider the following case: you have two EC2 instances, A and B, and need to recreate the B instance if the private IP of instance A is changed.</p>
<figure>
<img loading="lazy"
src="figure-11.png"
alt="replace_triggered_by example"width="800"
height="342"
/> <figcaption>
<p>replace_triggered_by example</p>
</figcaption>
</figure>
<p>This is extremely useful when you’re dealing with logical abstractions over the set of resources.</p>
<div class="attention">
<p>Resource replacement is triggered when:</p>
<ul>
<li>any of the resources referenced in <code>replace_triggered_by</code> are updated</li>
<li>any value is set to the resource attribute that is referenced in <code>replace_triggered_by</code></li>
</ul>
</div>
<h2 id="getting-started-with-terraform-11-and-12">Getting started with Terraform 1.1 and 1.2</h2>
<p>If you’re still using older Terraform versions, these new features might be a good motivation for you to upgrade!</p>
<p>Before upgrading, be sure to read the upgrade notes for the specific version at the <a href="https://github.com/hashicorp/terraform/releases">releases page</a>.</p>
<p>Also, an excellent tool can help with fast switching between different Terraform versions while you’re experimenting β <a href="https://tfswitch.warrensbox.com/">tfswitch</a>.</p>
Monterey Shortcuts for Easy and Fast Image Processing
2022-01-31T21:46:04Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2022/01/31/monterey-shortcuts-for-easy-and-fast-image-processing/
Serhii Vasylenko
https://devdosvid.blog/2022/01/31/monterey-shortcuts-for-easy-and-fast-image-processing/cover-image.png
<p>Here I want to share two Apple Shortcuts that I created for myself and use to process images for this blog:</p>
<p><a href="#optimization">Image Optimization</a></p>
<p>and</p>
<p><a href="#resize">Image Resize</a></p>
<p>About a year ago, I posted <a href="https://devdosvid.blog/2021/02/14/using-tinypng-image-compression-from-macos-finder-contextual-menu/" title="Using TinyPNG Image Compression From MacOS Finder Contextual Menu">the blog about the Automator quick action</a> to optimize PNG and JPEG images with TinyPNG service and save the processed images next to the original ones.</p>
<p>While that Automator-based solution still works, macOS Monterey now supports Shortcuts β a new automating tool that seems to substitute the old fellow Automator.</p>
<p>So I decided to create a couple of automation with Shortcuts: one for image optimization (reduce file size but not the size in pixels), and one for image scaling (change its size in pixels).</p>
<p>I have used them for several months to prepare images for this blog, and I really like how they work!</p>
<h2 id="optimization">Image Optimization with Monterey Shortcuts</h2>
<p>This Shortcut replicates the functionality of the Automator quick action and also uses TinyPNG service as a back-end. There are tons of other similar services, but I like TinyPNG for its simplicity: it just does one thing, and it does it well.</p>
<p>So first, you need to get yourself an <a href="https://tinypng.com/developers" title="TinyPNG Developers API">API key</a> for TinyPNG.</p>
<p>The simplest way to reuse my Shortcut is to import it from iCloud using the following URL:</p>
<p><a href="https://www.icloud.com/shortcuts/0a44de1596c745eaaad8181e61289248" title="Click here to import the Image Optimization Shortcut">β‘οΈ <strong>Click here to import the Image Optimization Shortcut</strong> β¬
οΈ</a></p>
<p>The import will work only when the link is opened in <strong>Safari</strong>.</p>
<figure>
<img loading="lazy"
src="shortcut-import-optimized.png"
alt="Shortcut Import Dialog"width="800"
height="595.00"
/> <figcaption>
<p>Shortcut Import Dialog</p>
</figcaption>
</figure>
<p>To make the Shortcut work from the context menu of the Finder, set the options on the Details panel of the Shortcut setting as displayed on the screenshot:</p>
<figure>
<img loading="lazy"
src="shortcut-settings-optimized.png"
alt="Shortcut Settings"width="800"
height="666.00"
/> <figcaption>
<p>Shortcut Settings</p>
</figcaption>
</figure>
<p>Here is what this Image Optimization Shortcut does:</p>
<blockquote>
<p><em>The Shortcut receives image files. Then, for every image file received, the Shotcut does the following:</em></p>
<ul>
<li><em>Gets the file’s name and parent directory</em></li>
<li><em>Sends the original file to TinyPNG</em></li>
<li><em>Process the response with URL to download the optimized image</em></li>
<li><em>Downloads the optimized image using the URL from the response and replaces the original image with the optimized</em></li>
</ul>
</blockquote>
<p>And here is how this Shortcuts looks if you want to create it from scratch:</p>
<figure>
<img loading="lazy"
src="image-optimization-shortcut.png"
alt="Image Optimization Shortcut"width="800"
height="1954.66"
/> <figcaption>
<p>Image Optimization Shortcut</p>
</figcaption>
</figure>
<p><em>Unfortunately, this Shortcut won’t work on iOS or watchOS because they do not support the “File Storage” actions used in the Shortcut.</em></p>
<p>π <strong>Demo</strong> π
<figure>
<img loading="lazy"
src="shortcut-demo.gif"
alt="Shortcut Demo"width="800"
height="620.00"
/> <figcaption>
<p>Shortcut Demo</p>
</figcaption>
</figure>
</p>
<h2 id="resize">Image Resize with Monterey Shortcuts</h2>
<p>Another Shortcut I actively use is the image resizer. Most of the images on my blog are 1600px width fitted into an 800px frame to look sharp on the high-res displays (e.g., Retina).</p>
<p>And when I have many images in my folder, I want to make them all be 1600px width at once or don’t change their own size if they were created smaller intentionally (no upscale, in other words).</p>
<p>Here is the link to Shortcut import (again, the import will work only when the link is opened in <strong>Safari</strong>):</p>
<p><a href="https://www.icloud.com/shortcuts/0af8005cc9ac4207a380be445601d541" title="Click here to import the Image Resize Shortcut">β‘οΈ <strong>Click here to import the Image Resize Shortcut</strong> β¬
οΈ</a></p>
<p>Here is how the Image Resize Shortcut looks if you want to create it from scratch:</p>
<figure>
<img loading="lazy"
src="image-resize-shortcut-optimized.png"
alt="Image Resizing Shortcut"width="800"
height="1176.98"
/> <figcaption>
<p>Image Resizing Shortcut</p>
</figcaption>
</figure>
<h2 id="fun-with-shortcuts">Fun with Shortcuts</h2>
<p>I love the way Apple works on routine automation. This Shortcuts app, ported from iOS, brings a lot of cool and fun possibilities to Mac.</p>
<p>Do you use Shortcuts? What is your favorite? I would love to know!</p>
Some Techniques to Enhance Your Terraform Proficiency
2022-01-15T23:59:51Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2022/01/16/some-techniques-to-enhance-your-terraform-proficiency/
Serhii Vasylenko
https://devdosvid.blog/2022/01/16/some-techniques-to-enhance-your-terraform-proficiency/cover-image.png
<p>Terraform built-in functionality is very feature-rich: functions, expressions, and meta-arguments provide many ways to shape the code and fit it to a particular use case. I want to share a few valuable practices to boost your Terraform expertise in this blog.</p>
<div class="attention">
Some code examples in this article will work with Terraform version 0.15 and onwards. But if you’re still using 0.14 or lower, here’s another motivation for you to upgrade.
</div>
<h2 id="conditional-resource-creation-or-how-to-implement-the-if-else-statement-in-terraform">Conditional resource creation or how to implement the “if else” statement in Terraform</h2>
<figure>
<img loading="lazy"
src="condiitonal-resource-creation.png"width="400"
height="400.00"
/>
</figure>
<p>With Terraform, you can have a conditional module or a resource creation by implementing the ternary operator β so-called Conditional Expressions.</p>
<p>Let’s start from the most popular one: whether to create a resource depending on some fact, e.g., the value of a variable.</p>
<p>Terraform meta-argument <code>count</code> helps to describe that kind of resource creation logic.</p>
<p>Here is how it may look like:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">data</span> <span style="color:#a5d6ff">"aws_ssm_parameter"</span> <span style="color:#a5d6ff">"ami_id"</span> {
</span></span><span style="display:flex;"><span> count = var.ami_channel =<span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">""</span> <span style="color:#ff7b72;font-weight:bold">?</span> <span style="color:#a5d6ff">0</span> <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">1</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> name = local.ami_channels[var.ami_channel]
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The notation <code>var.ami_channel == "" ? 0 : 1</code> is called <em>conditional expression</em> and means the following: if my variable is empty (<code>var.ami_channel == ""</code> β hence, true) then set the count to 0, otherwise set to 1.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>condition ? true_val : false_val
</span></span></code></pre></div><p>In this illustration, I want to get the AMI ID from the SSM Parameter only if the AMI channel (e.g., beta or alpha) is specified. Otherwise, providing that the <code>ami_channel</code> variable is an empty string by default (""), the data source should not be created.</p>
<p>When following this method, keep in mind that the resource address will contain the index identifier. So when I need to use the value of the SSM parameter from our example, I need to reference it the following way:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>ami_id = data.aws_ssm_parameter.ami_id[<span style="color:#a5d6ff">0</span>].value
</span></span></code></pre></div><p>The <code>count</code> meta-argument can also be used when you need to conditionally create a Terraform module.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">module</span> <span style="color:#a5d6ff">"bucket"</span> {
</span></span><span style="display:flex;"><span> count = var.create_bucket =<span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#79c0ff">true</span> <span style="color:#ff7b72;font-weight:bold">?</span> <span style="color:#a5d6ff">1</span> <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">0</span>
</span></span><span style="display:flex;"><span> source = <span style="color:#a5d6ff">"./modules/s3_bucket"</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"my-unique-bucket"</span>
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The <code>var.create_bucket == true ? 1 : 0</code> expression can be written even shorter: <code>var.create_bucket ? 1 : 0</code> because the <code>create_bucket</code> variable has boolean type, apparently.</p>
<p>But what if you need to produce more than one instance of a resource or module? And still be able to avoid their creation.</p>
<p>Another meta-argument β <code>for_each</code> β will do the trick.</p>
<p>For example, this is how the <code>for_each</code> argument works for the conditional module creation:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">module</span> <span style="color:#a5d6ff">"bucket"</span> {
</span></span><span style="display:flex;"><span> for_each = var.bucket_names =<span style="color:#ff7b72;font-weight:bold">=</span> [] <span style="color:#ff7b72;font-weight:bold">?</span> [] <span style="color:#ff7b72;font-weight:bold">:</span> var.bucket_names
</span></span><span style="display:flex;"><span> source = <span style="color:#a5d6ff">"./modules/s3_bucket"</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>each.key<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">"</span>
</span></span><span style="display:flex;"><span> enable_encryption = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In this illustration, I also used a conditional expression that makes Terraform iterate through the set of values of <code>var.bucket_names</code> if it’s not empty and create several modules. Otherwise, do not iterate at all and do not create anything.</p>
<p>The same can be done for the resources. For example, when you need to create an arbitrary number of security group rules, e.g., to allowlist some IPs for your bastion host:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_security_group_rule"</span> <span style="color:#a5d6ff">"allowlist"</span> {
</span></span><span style="display:flex;"><span> for_each = var.cidr_blocks =<span style="color:#ff7b72;font-weight:bold">=</span> [] <span style="color:#ff7b72;font-weight:bold">?</span> [] <span style="color:#ff7b72;font-weight:bold">:</span> var.cidr_blocks
</span></span><span style="display:flex;"><span> type = <span style="color:#a5d6ff">"ingress"</span>
</span></span><span style="display:flex;"><span> from_port = <span style="color:#a5d6ff">22</span>
</span></span><span style="display:flex;"><span> to_port = <span style="color:#a5d6ff">22</span>
</span></span><span style="display:flex;"><span> protocol = <span style="color:#a5d6ff">"tcp"</span>
</span></span><span style="display:flex;"><span> cidr_blocks = [each.value]
</span></span><span style="display:flex;"><span> security_group_id = aws_security_group.bastion.id
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And just like with the <code>count</code> meta-argument, with the <code>for_each</code>, resource addresses will have the identifier named by the values provided to <code>for_each</code>.</p>
<p>For example, here is how I would reference a resource created in the module with <code>for_each</code> described earlier:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>bucket_name = module.bucket[<span style="color:#a5d6ff">"photos"</span>].name
</span></span></code></pre></div><div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<h2 id="conditional-resource-arguments-attributes-setting">Conditional resource arguments (attributes) setting</h2>
<figure>
<img loading="lazy"
src="conditional-resource-argument.png"width="400"
height="400.00"
/>
</figure>
<p>Now let’s go deeper and see how resource arguments can be conditionally set (or not).</p>
<p>First, let’s review the conditional argument value setting with the <code>null</code> data type:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_launch_template"</span> <span style="color:#a5d6ff">"this"</span> {
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"my-launch-template"</span>
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span> key_name = var.use_default_keypair <span style="color:#ff7b72;font-weight:bold">?</span> var.keypair_name <span style="color:#ff7b72;font-weight:bold">:</span> null
</span></span><span style="display:flex;"><span> ...
</span></span></code></pre></div><p>Here I want to skip the usage of the EC2 Key Pair for the Launch Template in some instances and Terraform allows me to write the conditional expression that will set the <code>null</code> value for the argument. It means the <em>absence</em> or <em>omission</em> and Terraform would behave the same as if you did not specify the argument at all.</p>
<p>Dynamic blocks are another case where conditional creation suits best. Take a look at the following piece of CloudFront resource code where I want to either describe the configuration for the custom error response or omit that completely:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudfront_distribution"</span> <span style="color:#a5d6ff">"cdn"</span> {
</span></span><span style="display:flex;"><span> enabled = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span> dynamic <span style="color:#a5d6ff">"custom_error_response"</span> {
</span></span><span style="display:flex;"><span> for_each = var.custom_error_response =<span style="color:#ff7b72;font-weight:bold">=</span> null <span style="color:#ff7b72;font-weight:bold">?</span> [] <span style="color:#ff7b72;font-weight:bold">:</span> [var.custom_error_response]
</span></span><span style="display:flex;"><span> iterator = cer
</span></span><span style="display:flex;"><span> content {
</span></span><span style="display:flex;"><span> error_code = lookup(cer.value, <span style="color:#a5d6ff">"error_code"</span>, null)
</span></span><span style="display:flex;"><span> error_caching_min_ttl = lookup(cer.value, <span style="color:#a5d6ff">"error_caching_min_ttl"</span>, null)
</span></span><span style="display:flex;"><span> response_code = lookup(cer.value, <span style="color:#a5d6ff">"response_code"</span>, null)
</span></span><span style="display:flex;"><span> response_page_path = lookup(cer.value, <span style="color:#a5d6ff">"response_page_path"</span>, null)
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The <code>custom_error_response</code> variable is <code>null</code> by default, but it has the <code>object</code> type, and users can assign the variable with the required nested specifications if needed. And when they do it, Terraform will add the <code>custom_error_response</code> block to the resource configuration. Otherwise, it will be omitted entirely.</p>
<h2 id="convert-types-in-terraform-with-ease">Convert types in Terraform with ease</h2>
<p><figure>
<img loading="lazy"
src="types-converstion.png"width="400"
height="400.00"
/>
</figure>
Ok, let’s move to the less conditional things now π
</p>
<p>Terraform has several type conversion functions: <code>tobool()</code>, <code>tolist()</code>,<code>tomap()</code>, <code>tonumber()</code>, <code>toset()</code>, and <code>tostring()</code>. Their purpose is to convert the input values to the compatible types.</p>
<p>For example, suppose I need to pass the set to the <code>for_each</code> (it accepts only sets and maps types of value), but I got the list as an input; let’s say I got it as an output from another module. In such a case, I would do something like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>for_each = toset(var.remote_access_ports)
</span></span></code></pre></div><p>However, I can make my code cleaner and avoid the explicit conversion β I just need to define the value type in the configuration block of the <code>my_list</code> variable. Terraform will do the conversion automatically when the value is assigned.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">variable</span> <span style="color:#a5d6ff">"remote_access_ports"</span> {
</span></span><span style="display:flex;"><span> description = <span style="color:#a5d6ff">"Ports for remote access"</span>
</span></span><span style="display:flex;"><span> type = set(string)
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>While Terraform can do a lot of implicit conversions for you, explicit type conversions are practical during values normalization or when you need to calculate some complex value for a variable. For example, the Local Values, known as <code>locals</code>, are the most suitable place for doing that.</p>
<p>By the way, although there is a <code>tolist()</code> function, there is no such thing as the <code>tostring()</code> function. But what if you need to convert the list to string in Terraform?</p>
<p>The <code>one()</code> function can help here: it takes a list, set, or tuple value with either zero or one element and returns either <code>null</code> or that one element in the form of string.</p>
<p>It’s useful in cases when a resource created using conditional expression is represented as either a zero- or one-element list, and you need to get a single value which may be either <code>null</code> or <code>string</code>, for example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_kms_key"</span> <span style="color:#a5d6ff">"main"</span> {
</span></span><span style="display:flex;"><span> count = var.ebs_encrypted <span style="color:#ff7b72;font-weight:bold">?</span> <span style="color:#a5d6ff">1</span> <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">0</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> enable_key_rotation = <span style="color:#79c0ff">true</span>
</span></span><span style="display:flex;"><span> tags = var.tags
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">
</span></span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_kms_alias"</span> <span style="color:#a5d6ff">"main"</span> {
</span></span><span style="display:flex;"><span> count = var.ebs_encrypted <span style="color:#ff7b72;font-weight:bold">?</span> <span style="color:#a5d6ff">1</span> <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">0</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> name = <span style="color:#a5d6ff">"alias/encrypt-ebs"</span>
</span></span><span style="display:flex;"><span> target_key_id = one(aws_kms_key.main[<span style="color:#ff7b72;font-weight:bold">*</span>]key_id)
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="write-yaml-or-json-as-terraform-code-hcl">Write YAML or JSON as Terraform code (HCL)</h2>
<p><figure>
<img loading="lazy"
src="write-yaml-json-as-terraform-code.png"width="400"
height="400.00"
/>
</figure>
Sometimes you need to supply JSON or YAML files to the services you manage with Terraform. For example, if you want to create something with CloudFormation using Terraform (and I am not kidding). Sometimes the AWS Terraform provider does not support the needed resource, and you want to maintain the whole infrastructure code using only one tool.</p>
<p>Instead of maintaining another file in JSON or YAML format, you can embed JSON or YAML code management into HCL by taking benefit of the <code>jsonencode()</code> or <code>yamlencode()</code> functions.</p>
<p>The attractiveness of this approach is that you can reference other Terraform resources or their attributes right in the code of your object, and you have more freedom in terms of the code syntax and its formatting comparable to native JSON or YAML.</p>
<p>Here is how it looks like:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>locals {
</span></span><span style="display:flex;"><span> some_string = <span style="color:#a5d6ff">"ult"</span>
</span></span><span style="display:flex;"><span> myjson_object = jsonencode({
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"Hashicorp Products"</span><span style="color:#ff7b72;font-weight:bold">:</span> {
</span></span><span style="display:flex;"><span> Terra<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"form"</span>
</span></span><span style="display:flex;"><span> Con<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"sul"</span>
</span></span><span style="display:flex;"><span> Vag<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"rant"</span>
</span></span><span style="display:flex;"><span> Va<span style="color:#ff7b72;font-weight:bold">:</span> local.some_string
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> })
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The value of the <code>myjson_object</code> local variable would look like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Hashicorp Products"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Con"</span>: <span style="color:#a5d6ff">"sul"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Terra"</span>: <span style="color:#a5d6ff">"form"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Va"</span>: <span style="color:#a5d6ff">"ult"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Vag"</span>: <span style="color:#a5d6ff">"rant"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And here is a piece of real-world example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>locals {
</span></span><span style="display:flex;"><span> cf_template_body = jsonencode({
</span></span><span style="display:flex;"><span> Resources <span style="color:#ff7b72;font-weight:bold">:</span> {
</span></span><span style="display:flex;"><span> DedicatedHostGroup <span style="color:#ff7b72;font-weight:bold">:</span> {
</span></span><span style="display:flex;"><span> Type <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"AWS::ResourceGroups::Group"</span>
</span></span><span style="display:flex;"><span> Properties <span style="color:#ff7b72;font-weight:bold">:</span> {
</span></span><span style="display:flex;"><span> Name <span style="color:#ff7b72;font-weight:bold">:</span> var.service_name
</span></span><span style="display:flex;"><span> Configuration <span style="color:#ff7b72;font-weight:bold">:</span> [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> Type <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"AWS::EC2::HostManagement"</span>
</span></span><span style="display:flex;"><span> Parameters <span style="color:#ff7b72;font-weight:bold">:</span> [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> Name <span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"auto-allocate-host"</span>
</span></span><span style="display:flex;"><span> Values <span style="color:#ff7b72;font-weight:bold">:</span> [var.auto_allocate_host]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span> ...
</span></span></code></pre></div><h2 id="create-custom-file-templates-in-terraform">Create custom file templates in Terraform</h2>
<p><figure>
<img loading="lazy"
src="templatize-stuff.png"width="400"
height="400.00"
/>
</figure>
The last case in this blog but not the least by its efficacy β render source file content as a template in Terraform.</p>
<p>Let’s review the following scenario: you launch an EC2 instance and want to supply it with a bash script (via the user-data parameter) for some additional configuration at launch.</p>
<p>Suppose we have the following bash script <code>instance-init.sh</code> that sets the hostname and registers our instance in a monitoring system:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#8b949e;font-weight:bold;font-style:italic">#!/bin/bash
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-weight:bold;font-style:italic"></span>
</span></span><span style="display:flex;"><span>hostname example.com
</span></span><span style="display:flex;"><span>bash /opt/system-init/register-monitoring.sh
</span></span></code></pre></div><p>But what if you want to set a different hostname per instance, and some instances should not be registered in the monitoring system?</p>
<p>In such a case, here is how the script file content will look:</p>
<pre tabindex="0"><code class="language-gotemplate" data-lang="gotemplate">#!/bin/bash
hostname ${system_hostname}
%{ if register_monitoring }
bash /opt/system-init/register-monitoring.sh
%{endif}
</code></pre><p>And when you supply this file as an argument for the EC2 instance resource in Terraform, you will use the <code>templatefile()</code> function to make the magic happen:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_instance"</span> <span style="color:#a5d6ff">"web"</span> {
</span></span><span style="display:flex;"><span> ami = var.my_ami_id
</span></span><span style="display:flex;"><span> instance_type = var.instance_type
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span> user_data = templatefile(<span style="color:#a5d6ff">"</span><span style="color:#a5d6ff">${</span>path.module<span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/instance-init.tftpl"</span>, {
</span></span><span style="display:flex;"><span> system_hostname = var.system_hostname
</span></span><span style="display:flex;"><span> register_monitoring = var.add_to_monitoring
</span></span><span style="display:flex;"><span> })
</span></span><span style="display:flex;"><span> ...
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And of course, you can create a template from any file type. The only requirement here is that the template file must exist on the disk at the beginning of the Terraform execution.</p>
<h2 id="key-takeaways">Key takeaways</h2>
<p>Terraform is far beyond the standard resource management operations. With the power of built-in functions, you can write more versatile code and reusable Terraform modules.</p>
<p>β
Use <a href="https://www.terraform.io/language/expressions/conditionals">conditional expressions</a> with <a href="https://www.terraform.io/language/meta-arguments/count">count</a> and <a href="https://www.terraform.io/language/meta-arguments/for_each">for_each</a> meta-arguments, when the creation of a resource depends on some context or user input.</p>
<p>β
Take advantage of <a href="https://www.terraform.io/language/expressions/types#type-conversion">implicit type conversion</a> when working with input variables and their values to keep your code cleaner.</p>
<p>β
Embed YAML and JSON-based objects right into your Terraform code using built-in <a href="https://www.terraform.io/language/functions/jsonencode">encoding</a> <a href="https://www.terraform.io/language/functions/yamlencode">functions</a>.</p>
<p>β
And when you need to pass some files to the managed service, you can treat them as <a href="https://www.terraform.io/language/functions/templatefile">templates</a> and make them multipurpose.</p>
<p>Thank you for reading down to this point! π€</p>
<p>If you have some favorite Terraform tricks β I would love to know!</p>
Guide to Using Terraform in CI/CD
2021-11-24T20:20:45Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/11/24/guide-to-using-terraform-in-ci/cd/
Serhii Vasylenko
https://devdosvid.blog/2021/11/24/guide-to-using-terraform-in-ci/cd/cover-image.png
<p>Terraform by itself automates a lot of things: it creates, changes, and versions your cloud resources. Although many teams run Terraform locally (sometimes with wrapper scripts), running Terraform in CI/CD can boost the organization’s performance and ensure consistent deployments.</p>
<p>In this article, I would like to review different approaches to integrating Terraform into generic deployment pipelines.</p>
<h1 id="where-to-store-the-terraform-code">Where to store the Terraform code</h1>
<p>Storing Terraform code in the same repository as the application code or maintaining a separate repository for the infrastructure?</p>
<p>This question has no strict and clear answer, but here are some insights that may help you decide:</p>
<ul>
<li>The Terraform and application code coupled together represent one unit, so it’s simple to maintain by one team;</li>
<li>Conversely, if you have a dedicated team that manages infrastructure (e.g., platform team), a separate repository for infrastructure is more convenient because it’s a standalone project in that case.</li>
<li>When infrastructure code is stored with the application, sometimes you have to deal with additional rules for the pipeline to separate triggers for these code parts. But sometimes (e.g., serverless apps) changes to either part (app/infra) should trigger the deployment.</li>
</ul>
<div class="attention">
There is no right or wrong approach, but whichever you choose, remember to follow the <strong>Donβt Repeat Yourself (DRY)</strong> principle: make the infrastructure code modular by logically grouping resources into higher abstractions and reusing these modules.
</div>
<h1 id="preparing-terraform-execution-environment">Preparing Terraform execution environment</h1>
<p>Running Terraform locally generally means that all dependencies are already in-place: you have the binary installed and present in the user’s <code>PATH</code> and perhaps even some providers already stored in the <code>.terraform</code> directory.</p>
<p>But when you shift Terraform runs from your local machine to stateless pipelines, this is not the case. However, you can still have a pre-built environment β this will speed up the pipeline execution and provide control over the process.</p>
<p>Docker image with a Terraform binary is one of the popular solutions that address this. Once created, you can execute Terraform within a container context with configuration files mounted as a Docker volume.</p>
<p>You can use the official <a href="https://hub.docker.com/r/hashicorp/terraform/">image from Hashicorp</a>, but sometimes it makes sense to maintain your own Docker images with additional tools you may need. For instance, you can bake the <code>tfsec</code> tool into the image to use it for security inspection and have it ready inside the Docker container without the need to install it every time.</p>
<p>Here is an example of a Dockerfile that builds an image with a custom Terraform version (you can override it as a build argument) and a <code>tfsec</code> tool. This example also shows how to verify the installed Terraform binary to make sure it’s signed by HashiCorp before we run it.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-dockerfile" data-lang="dockerfile"><span style="display:flex;"><span><span style="color:#ff7b72">FROM</span><span style="color:#a5d6ff"> alpine:3.14</span><span style="color:#f85149">
</span></span></span><span style="display:flex;"><span><span style="color:#f85149"></span><span style="color:#ff7b72">ARG</span> <span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">1</span>.0.11<span style="color:#f85149">
</span></span></span><span style="display:flex;"><span><span style="color:#f85149"></span><span style="color:#ff7b72">ARG</span> <span style="color:#79c0ff">TFSEC_VERSION</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">0</span>.59.0<span style="color:#f85149">
</span></span></span><span style="display:flex;"><span><span style="color:#f85149"></span><span style="color:#ff7b72">RUN</span> apk add --no-cache --virtual .sig-check gnupg<span style="color:#f85149">
</span></span></span><span style="display:flex;"><span><span style="color:#f85149"></span><span style="color:#ff7b72">RUN</span> wget -O /usr/bin/tfsec https://github.com/aquasecurity/tfsec/releases/download/v<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TFSEC_VERSION</span><span style="color:#a5d6ff">}</span>/tfsec-linux-amd64 <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> chmod +x /usr/bin/tfsec<span style="color:#f85149">
</span></span></span><span style="display:flex;"><span><span style="color:#f85149"></span><span style="color:#ff7b72">RUN</span> cd /tmp <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> wget <span style="color:#a5d6ff">"https://releases.hashicorp.com/terraform/</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">/terraform_</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">_linux_amd64.zip"</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> wget https://keybase.io/hashicorp/pgp_keys.asc <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> gpg --import pgp_keys.asc <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> gpg --fingerprint --list-signatures <span style="color:#a5d6ff">"HashiCorp Security"</span> | grep -q <span style="color:#a5d6ff">"C874 011F 0AB4 0511 0D02 1055 3436 5D94 72D7 468F"</span> <span style="color:#ff7b72;font-weight:bold">||</span> exit <span style="color:#a5d6ff">1</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> gpg --fingerprint --list-signatures <span style="color:#a5d6ff">"HashiCorp Security"</span> | grep -q <span style="color:#a5d6ff">"34365D9472D7468F"</span> <span style="color:#ff7b72;font-weight:bold">||</span> exit <span style="color:#a5d6ff">1</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> wget https://releases.hashicorp.com/terraform/<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>/terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_SHA256SUMS <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> wget https://releases.hashicorp.com/terraform/<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>/terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_SHA256SUMS.sig <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> gpg --verify terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_SHA256SUMS.sig terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_SHA256SUMS <span style="color:#ff7b72;font-weight:bold">||</span> exit <span style="color:#a5d6ff">1</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> sha256sum -c terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_SHA256SUMS 2>&<span style="color:#a5d6ff">1</span> | grep -q <span style="color:#a5d6ff">"terraform_</span><span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span><span style="color:#a5d6ff">_linux_amd64.zip: OK"</span> <span style="color:#ff7b72;font-weight:bold">||</span> exit <span style="color:#a5d6ff">1</span> <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> unzip terraform_<span style="color:#a5d6ff">${</span><span style="color:#79c0ff">TERRAFORM_VERSION</span><span style="color:#a5d6ff">}</span>_linux_amd64.zip -d /bin <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span> <span style="color:#ff7b72;font-weight:bold">&&</span> rm -rf /tmp/* <span style="color:#ff7b72;font-weight:bold">&&</span> apk del .sig-check<span style="color:#f85149">
</span></span></span></code></pre></div><p>But the main functionality of Terraform is delivered by provider plugins. It takes time to download the provider: for example, the AWS provider is about 250MB, and in a large scale, with hundreds of Terraform runs per day, this makes a difference.</p>
<p>There are two common ways to deal with it: either use a shared cache available to your pipeline workloads or bake provider binaries into the runtime environment (i.e., Docker image).</p>
<p>The critical element for both approaches is the configuration of the plugin cache directory path. By default, Terraform looks for plugins and downloads them in the <code>.terraform</code> directory, which is local to the main project directory. But you can override this, and you can leverage the <code>TF_PLUGIN_CACHE_DIR</code> environment variable to do that.</p>
<p>If supported by your CI/CD tool, the shared cache can significantly reduce the operational burden because all your pipeline runtime environments can use it to get the needed provider versions.</p>
<p>So all you have to do is to maintain the provider versions in the shared cache and instruct Terraform to use it:</p>
<ul>
<li>Mount the cache directory to the pipeline runtime (i.e., docker container) and specify its internal path</li>
<li>Set the value of the <code>TF_PLUGIN_CACHE_DIR</code> environment variable accordingly</li>
</ul>
<p>On the other hand, you can bake the provider binaries into the Docker image and inject the value for the <code>TF_PLUGIN_CACHE_DIR</code> environment variable right into the Dockerfile.</p>
<div class="attention">
This approach takes more operational effort <strong>but makes the Terraform environment self-sufficient and stateless</strong>. It also allows you to set strict boundaries around permitted provider versions as a security measure.
</div>
<h1 id="planning-and-applying-changes">Planning and Applying changes</h1>
<p>Now let’s review the ways to automate planning and applying of changes. Although <code>terraform apply</code> can do both, it’s sometimes useful to separate these actions.</p>
<h2 id="initialization">Initialization</h2>
<p>CI/CD pipelines generally run in stateless environments. Thus, every subsequent run of Terraform looks like a fresh start, so the project needs to be initialized before other actions can be performed.</p>
<p>The usage of the <code>init</code> command in CI/CD slightly differs from its common local usage:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> terraform init -input<span style="color:#ff7b72;font-weight:bold">=</span>false
</span></span></code></pre></div><p>The <code>-input=false</code> option prevents Terraform CLI from asking for user actions (it will throw an error if the input was required).</p>
<p><em>Also, there is <code>-no-color</code> option that prevents the usage of color codes in a shell, so the output will look much cleaner if your CI/CD logging system cannot render the terminal formatting.</em></p>
<p>Another option of the init command that is useful in CI β is the <code>-backend-config</code>. That option allows you to override the backend configuration in your code or define it if you prefer to use partial configuration, thus creating more uniform pipelines.</p>
<p>For example, here is how you can use the same code with different roles in different environments on AWS:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> terraform init -input<span style="color:#ff7b72;font-weight:bold">=</span>false <span style="color:#79c0ff">\
</span></span></span><span style="display:flex;"><span><span style="color:#79c0ff"></span>-backend-config<span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"role_arn=arn:aws:iam::012345678901:role/QADeploymentAutomation"</span>
</span></span></code></pre></div><p>Terraform <code>init</code> produces two artifacts:</p>
<ul>
<li><code>.terraform</code> directory, which Terraform uses to manage cached provider plugins and modules, and record backend information</li>
<li><code>.terraform.lock.hcl</code> file, which Terraform uses to track provider dependencies</li>
</ul>
<p>They both must be present in the project directory to successfully run the subsequent plan and apply commands.</p>
<p>However, I suggest checking in <code>.terraform.lock.hcl</code> to your repository as suggested by HashiCorp (<a href="https://www.terraform.io/docs/language/dependency-lock.html">Dependency Lock File</a>): this way you will be able to control dependencies more thoroughly, and you will not worry about transferring this file between build stages.</p>
<h2 id="plan">Plan</h2>
<p>The <code>terraform plan</code> command helps you validate the changes manually. However, there are ways to use it in automation as well.</p>
<p>By default, Terraform prints the plan output in a human-friendly format but also supports machine-readable JSON. With additional command-line options, you can extend your CI experience.</p>
<p>For example, you can use your validation conditions to decide whether to apply the changes automatically; or you can parse the plan details and integrate the summary into a Pull Request description. Letβs review a simple example that illustrates it.</p>
<p>First, you need to save the plan output to the file:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> terraform plan -input<span style="color:#ff7b72;font-weight:bold">=</span>false -compact-warnings -out<span style="color:#ff7b72;font-weight:bold">=</span>plan.file
</span></span></code></pre></div><p>The main point here is the <code>-out</code> option β it tells Terraform to save its output into a binary plan file, and we will talk about it in the next paragraph.</p>
<p>The <code>-compact-warnings</code> option suppresses the warning-level messages produced by Terraform.</p>
<p>Also, the <code>plan</code> command has the <code>-detailed-exitcode</code> option that returns detailed exit codes when the command exits. For example, you can leverage this in a script that wraps Terraform and adds more conditional logic to its execution, because CIs will generally fail the pipeline on a commandβs non-zero exit code. However, that may add complexity to the pipeline logic.</p>
<p>So if you need to get detailed info about the plan, I suggest parsing the plan output.</p>
<p>When you have a plan file, you can read it in JSON format and parse it. Here is a code snippet that illustrates that:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> terraform show -json plan.file| jq -r <span style="color:#a5d6ff">'([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}'</span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72;font-weight:bold">{</span>
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"create"</span>: 1,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"update"</span>: 0,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"delete"</span>: <span style="color:#a5d6ff">0</span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72;font-weight:bold">}</span>
</span></span></code></pre></div><p>Another way to see the information about changes, is to run the <code>plan</code> command with <code>-json</code> option and parse its output to stdout (available starting from Terraform 1.0.5):</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> terraform plan -json|jq <span style="color:#a5d6ff">'select( .type == "change_summary")|."@message"'</span>
</span></span><span style="display:flex;"><span><span style="color:#a5d6ff">"Plan: 1 to add, 0 to change, 0 to destroy."</span>
</span></span></code></pre></div><p><div class="attention">
This technique can make your Pull Request messages more informative and improve your collaboration with teammates.
</div>
You can write a custom script/function that sends a Pull Request comment to VCS using its API. Or you can try the existing features of your VCS: with GitHub Actions, you can use the <a href="https://github.com/marketplace/actions/terraform-pr-commenter">Terraform PR Commenter</a> or similar action to achieve that; for GitLab, there is a built-in functionality that integrates plan results into the Merge Request β <a href="https://docs.gitlab.com/ee/user/infrastructure/iac/mr_integration.html">Terraform integration in Merge Requests</a>.</p>
<p>You can find more information about the specification of the JSON output here β <a href="https://www.terraform.io/docs/internals/json-format.html">Terraform JSON Output Format</a>.</p>
<h2 id="apply">Apply</h2>
<p>When the plan file is ready, and the proposed changes are expected and approved, it’s time to <code>apply</code> them.</p>
<p>Here is how the <code>apply</code> command may look like in automation:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>terraform apply -input<span style="color:#ff7b72;font-weight:bold">=</span>false -compact-warnings plan.file
</span></span></code></pre></div><p>Here, the <code>plan.file</code> is the file we got from the previous plan step.</p>
<p>Alternatively, you might want to omit the planning phase at all. In that case, the following command will apply the configuration immediately, without the need for a plan:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>terraform apply -input<span style="color:#ff7b72;font-weight:bold">=</span>false -compact-warnings -auto-approve
</span></span></code></pre></div><p>Here, the <code>-auto-approve</code> option tells Terraform to create the plan implicitly and skip the interactive approval of that plan before applying.</p>
<p>Whichever way you choose, keep in mind the destructive nature of the apply command. Hence, the fully automated apply of configuration generally works well with environments that tolerate unexpected downtimes, such as development or testing. Whereas plan review is recommended for production-grade environments, and in that case, the <code>apply</code> job is configured for a manual trigger.</p>
<h1 id="dealing-with-stateless-environments">Dealing with stateless environments</h1>
<p>If you run <code>init</code>, <code>plan</code>, and <code>apply</code> commands in different environments, you need to care for some artifacts produced by Terraform:</p>
<ul>
<li>The <code>.terraform</code> directory with information about modules, providers, and the state file (even in the case of remote state).</li>
<li>The <code>.terraform.lock.hcl</code> file β the dependency lock file which Terraform uses to check the integrity of provider versions used for the project. If your VCS does not track it, you’ll need to pass that file to the <code>plan</code> and <code>apply</code> commands to make them work after <code>init</code>.</li>
<li>The output file of the <code>plan</code> command is essential for the <code>apply</code> command, so treat it as a vital artifact. This file includes a full copy of the project configuration, the state, and variables passed to the <code>plan</code> command (if any). Therefore, mind the security precautions because sensitive information may be present there.</li>
</ul>
<p>There is one shortcut, though. You can execute the <code>init</code> and <code>plan</code> commands within the same step/stage and transfer the artifacts only once β to the <code>apply</code> execution.</p>
<h1 id="using-the-command-line-and-environments-variables">Using the command-line and environments variables</h1>
<p>Last but not least, a few words about ways to maximize the advantage of variables when running Terraform in CI.</p>
<p>There are two common ways how you can pass values for the variables used in the configuration:</p>
<ol>
<li>Using a <code>-var-file</code> option with the variable definitions file β a filename ending in <code>.tfvars</code> or <code>.tfvars.json</code>. For example:
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>terraform apply -var-file<span style="color:#ff7b72;font-weight:bold">=</span>development.tfvars -input<span style="color:#ff7b72;font-weight:bold">=</span>false -no-color -compact-warnings -auto-approve
</span></span></code></pre></div>Also, Terraform can automatically load the variables from files named exactly <code>terraform.tfvars</code> or <code>terraform.tfvars.json</code>: with that approach, you donβt need to specify the tfvar file as a command option explicitly.</li>
<li>Using environment variables with the prefix <code>TF_VAR_</code>. Implicitly, Terraform always looks for the environment variables (within its process context) with that prefix, so the same “instance_type” variables from the example above can be passed as follows:
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>export <span style="color:#79c0ff">TF_VAR_instance_type</span><span style="color:#ff7b72;font-weight:bold">=</span>t3.nano
</span></span><span style="display:flex;"><span>terraform -input<span style="color:#ff7b72;font-weight:bold">=</span>false -no-color -compact-warnings -auto-approve
</span></span></code></pre></div></li>
</ol>
<p>The latter method is widely used in CI because modern CI/CD tools support the management of the environment variables for automation jobs.</p>
<p>Please refer to the following official documentation if you want to know more about variables β <a href="https://www.terraform.io/docs/language/values/variables.html">Terraform Input Variables</a>.</p>
<p>Along with that, Terraform supports several configuration parameters in the form of environment variables. These parameters are optional; however, they can simplify the automation management and streamline its code.</p>
<ul>
<li><code>TF_INPUT</code> β when set to “false” or “0”, this tells Terraform to behave the same way as with the <code>-input=false</code> flag;</li>
<li><code>TF_CLI_ARGS</code> β can contain a set of command-line options that will be passed to one or another Terraform command. Therefore, the following notation can simplify the execution of <code>apply</code> and <code>plan</code> commands by unifying their options for CI:
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>export <span style="color:#79c0ff">TF_CLI_ARGS</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#a5d6ff">"-input=false -no-color -compact-warnings"</span>
</span></span><span style="display:flex;"><span>terraform plan ...
</span></span><span style="display:flex;"><span>terraform apply ...
</span></span></code></pre></div>You can advantage this even more when using this variable as the environment configuration of stages or jobs in a CI/CD tool.</li>
<li><code>TF_IN_AUTOMATION</code> β when set to any non-empty value (e.g., “true”), Terraform stops suggesting commands run after the one you execute, hence producing less output.</li>
</ul>
<h1 id="key-takeaways">Key takeaways</h1>
<p>There are two primary outcomes from automating Terraform executions: consistent results and integrating with the code or project management solutions. Although the exact implementation of Terraform in CI may vary per project or team, try to aim the following goals when working on it:</p>
<ul>
<li>Ease of code management</li>
<li>A secure and controlled execution environment</li>
<li>Coherent runs of init, plan, apply phases</li>
<li>Leveraging of built-in Terraform capabilities</li>
</ul>
<h5 id="i-originally-wrote-this-article-for-the-spaceliftio-technical-blog-but-i-decided-to-keep-it-here-as-well-for-the-history-the-canonical-link-to-their-blog-has-been-set-accordingly">I originally wrote this article for the Spacelift.io technical blog. But I decided to keep it here as well, for the history. The canonical link to their blog has been set accordingly.</h5>
Apply Cloudfront Security Headers With Terraform
2021-11-05T12:20:58Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/11/05/apply-cloudfront-security-headers-with-terraform/
Serhii Vasylenko
https://devdosvid.blog/2021/11/05/apply-cloudfront-security-headers-with-terraform/cover-image.png
<p>In November 2021, AWS announced Response Headers Policies β native support of response headers in CloudFront. You can read the full announcement here: <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-cloudfront-introduces-response-headers-policies/">Amazon CloudFront introduces Response Headers Policies</a></p>
<p>I said “native” because previously you could set response headers either using <a href="https://devdosvid.blog/2021/05/21/configure-http-security-headers-with-cloudfront-functions.html">CloudFront Functions</a> or <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/adding-http-security-headers-using-lambdaedge-and-amazon-cloudfront/">Lambda@Edge</a>.</p>
<p>And one of the common use cases for that was to set security headers. Now you don’t need to add intermediate requests processing to modify the headers: CloudFront does that for you <strong>with no additional fee</strong>.</p>
<h2 id="manage-security-headers-as-code">Manage Security Headers as Code</h2>
<p>Starting from the <a href="https://github.com/hashicorp/terraform-provider-aws/blob/main/CHANGELOG.md#3640-november-04-2021">3.64.0</a> version of Terraform AWS provider, you can create the security headers policies and apply them for your distribution.</p>
<p>Let’s see how that looks!</p>
<p>First, you need to describe the <code>aws_cloudfront_response_headers_policy</code> resource:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudfront_response_headers_policy" "security_headers_policy"</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"my-security-headers-policy"</span>
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">security_headers_config</span> {
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">content_type_options</span> {
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">frame_options</span> {
</span></span><span style="display:flex;"><span> frame_option <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"DENY"</span>
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">referrer_policy</span> {
</span></span><span style="display:flex;"><span> referrer_policy <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"same-origin"</span>
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">xss_protection</span> {
</span></span><span style="display:flex;"><span> mode_block <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> protection <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">strict_transport_security</span> {
</span></span><span style="display:flex;"><span> access_control_max_age_sec <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"63072000"</span>
</span></span><span style="display:flex;"><span> include_subdomains <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> preload <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">content_security_policy</span> {
</span></span><span style="display:flex;"><span> content_security_policy <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"frame-ancestors 'none'; default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'"</span>
</span></span><span style="display:flex;"><span> override <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>List of security headers used:</p>
<ul>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-content-type-options">X-Content-Type-Options</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-frame-options">X-Frame-Options</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#referrer-policy">Referrer Policy</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-xss-protection">X-XSS-Protection</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#http-strict-transport-security">Strict Transport Security</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#content-security-policy">Content Security Policy</a></li>
</ul>
<p>The values for the security headers can be different, of course. However, the provided ones cover the majority of cases. And you can always get the up to date info about these headers and possible values here: <a href="https://infosec.mozilla.org/guidelines/web_security">Mozilla web Security Guidelines</a></p>
<p>Also, you could notice that provided example uses the <code>override</code> argument a lot. The <code>override</code> argument tells CloudFront to set these values for specified headers despite the values received from the origin. This way, you can enforce your security headers configuration.</p>
<p>Once you have the <code>aws_cloudfront_response_headers_policy</code> resource, you can refer to it in the code of <code>aws_cloudfront_distribution</code> resource inside cache behavior block (default or ordered). For example, in your <code>default_cache_behavior</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudfront_distribution" "test"</span> {
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">default_cache_behavior</span> {
</span></span><span style="display:flex;"><span> target_origin_id <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">aws_s3_bucket</span>.<span style="color:#ff7b72">my_origin</span>.<span style="color:#ff7b72">id</span>
</span></span><span style="display:flex;"><span> allowed_methods <span style="color:#ff7b72;font-weight:bold">=</span> [<span style="color:#a5d6ff">"GET", "HEAD", "OPTIONS"</span>]
</span></span><span style="display:flex;"><span> cached_methods <span style="color:#ff7b72;font-weight:bold">=</span> [<span style="color:#a5d6ff">"GET", "HEAD"</span>]
</span></span><span style="display:flex;"><span> viewer_protocol_policy <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"redirect-to-https"</span><span style="color:#8b949e;font-style:italic">
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic">
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"> # some arguments skipped from listing for the sake of simplicity
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span>
</span></span><span style="display:flex;"><span> response_headers_policy_id <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">aws_cloudfront_response_headers_policy</span>.<span style="color:#ff7b72">security_headers_policy</span>.<span style="color:#ff7b72">id</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> }<span style="color:#8b949e;font-style:italic">
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic">
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"> # some arguments skipped from listing for the sake of simplicity
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span>}
</span></span></code></pre></div><h3 id="security-scan-results">Security Scan Results</h3>
<p>Here is what Mozilla Observatory reports about my test CF distribution where I enabled the policy described above:</p>
<figure>
<img loading="lazy"
src="observatory-results.png"
alt="Scan summary for CloudFront distribution with security headers policy"width="800"
height="903.43"
/> <figcaption>
<p>Scan summary for CloudFront distribution with security headers policy</p>
</figcaption>
</figure>
<p>So with just minimum effort, you can greatly boost your web application security posture.</p>
<h3 id="more-to-read">More to read:</h3>
<ul>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudfront_response_headers_policy">Terraform Resource: aws_cloudfront_response_headers_policy</a></li>
<li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/creating-response-headers-policies.html">Creating response headers policies - Amazon CloudFront</a></li>
<li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/using-managed-response-headers-policies.html">Using the managed response headers policies - Amazon CloudFront</a></li>
<li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/understanding-response-headers-policies.html">Understanding response headers policies - Amazon CloudFront</a></li>
</ul>
Auto Scaling Group for your macOS EC2 Instances fleet
2021-10-23T23:00:31Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/10/24/auto-scaling-group-for-your-macos-ec2-instances-fleet/
Serhii Vasylenko
https://devdosvid.blog/2021/10/24/auto-scaling-group-for-your-macos-ec2-instances-fleet/cover-image.png
<p>Itβs been almost a year since I started using macOS EC2 instances on AWS: there were <a href="https://devdosvid.blog/2021/01/19/mac1-metal-EC2-Instance-user-experience.html">ups and downs in service offerings</a> and a lot of discoveries with <a href="https://devdosvid.blog/2021/02/01/customizing-mac1-metal-ec2-ami.html">macOS AMI build</a> automation.</p>
<p>And I like this small but so helpful update to the offerings list of the EC2 service: with mac1.metal instances, seamless integration of Apple-oriented CI/CD with other AWS infrastructure could finally happen.</p>
<p>But while management of a single mac1.metal node (or a tiny number of ones) is not a big deal (especially when <a href="https://devdosvid.blog/2021/01/20/terraforming-mac1-metal-at-AWS.html">Dedicated Host support</a> was added to Terraform provider), governing the fleet of instances is still complicated.</p>
<p>Or it has been complicated until recent days.</p>
<p>With a growing number of instances, the following challenges arise:</p>
<ul>
<li>Scale mac1.metal instances horizontally</li>
<li>Automatically allocate and release Dedicated Hosts needed for instances</li>
<li>Automatically replace unhealthy instances</li>
</ul>
<p>If you have worked with AWS before, you know that Auto Scaling Group service can solve such things.</p>
<h2 id="auto-scaling-for-macos-ec2-instances">Auto Scaling for macOS EC2 Instances</h2>
<p>So how does all that work?</p>
<p>Letβs review the diagram that illustrates the interconnection between involved services:</p>
<figure>
<img loading="lazy"
src="general-scheme_compressed.png"
alt="Services logical interconnection"width="800"
height="639"
/> <figcaption>
<p>Services logical interconnection</p>
</figcaption>
</figure>
<p>With the help of Licence Manager service and Launch Templates, you can set up EC2 Auto Scaling Group for mac1.metal and leave the automated instance provisioning to the service.</p>
<h3 id="license-configuration">License Configuration</h3>
<p>First, you need to create a License Configuration so that the Host resource group can allocate the hots.</p>
<p>Go to AWS License Manager -> Customer managed licenses -> Create customer-managed license.</p>
<p>Specify <strong>Sockets</strong> as the Licence type. You may skip setting the Number of Sockets. However, the actual limit of mac1.metal instances per account is regulated by Service Quota. The default number of mac instances allowed per account is 3. Therefore, consider <a href="https://docs.aws.amazon.com/servicequotas/latest/userguide/request-quota-increase.html">increasing</a> this to a more significant number.</p>
<figure>
<img loading="lazy"
src="license-configuration_compressed.png"
alt="Licence configuration values"width="800"
height="807"
/> <figcaption>
<p>Licence configuration values</p>
</figcaption>
</figure>
<h3 id="host-resource-group">Host resource group</h3>
<p>Second, create the Host resource group: AWS License Manager -> Host resource groups -> Create host resource group.</p>
<p>When creating the Host resource group, check β<strong>Allocate hosts automatically</strong>β and β<strong>Release hosts automatically</strong>β but leave βRecover hosts automaticallyβ unchecked. Dedicated Host does <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-hosts-recovery.html#dedicated-hosts-recovery-instances">not support host recovery</a> for mac1.metal.
However, Auto Scaling Group will maintain the desired number of instances if one fails the health check (which assumes the case of host failure as well).</p>
<p>Also, I recommend specifying βmac1β as an allowed Instance family for the sake of transparent resource management: only this instance type is permitted to allocate hosts in the group.</p>
<figure>
<img loading="lazy"
src="host-resource-group_compressed.png"
alt="Host resource group configuration values"width="800"
height="902"
/> <figcaption>
<p>Host resource group configuration values</p>
</figcaption>
</figure>
<p>Optionally, you may specify the license association here (the Host group will pick any compatible license) or select the license you created on step one.</p>
<h3 id="launch-template">Launch Template</h3>
<p>Create Launch Template: EC2 -> Launch templates -> Create launch template.</p>
<p>I will skip the description of all Launch Template parameters (but here is a nice <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html">tutorial</a>), if you donβt mind, and keep focus only on the items relevant to the current case.</p>
<p>Specify mac1.metal as the Instance type. Later, in Advanced details: find the <strong>Tenancy</strong> parameter and set it to βDedicated hostβ; for <strong>Target host by</strong> select βHost resource groupβ, and once selected the new parameter <strong>Tenancy host resource group</strong> will appear where you should choose your host group; select your license in <strong>License configurations</strong> parameter.</p>
<figure>
<img loading="lazy"
src="launch-template_compressed.png"
alt="Launch Template configuration values"width="800"
height="741"
/> <figcaption>
<p>Launch Template configuration values</p>
</figcaption>
</figure>
<h3 id="auto-scaling-group">Auto Scaling Group</h3>
<p>Finally, create the Auto Scaling Group: EC2 -> Auto Scaling groups -> Create Auto Scaling group.</p>
<p>The vital thing to note here β is the availability of the mac1.metal instance in particular AZ.</p>
<p>Mac instances are available in us-east-1 and <a href="https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-ec2-mac-instances-additional-regions/">7 more regions</a>, but not every Availability Zone in the region supports it. So you must figure out which AZ supports the needed instance type.</p>
<p>There is no documentation for that, but there is an AWS CLI command that can answer this question: <a href="https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/describe-instance-type-offerings.html">describe-instance-type-offerings β AWS CLI 2.3.0 Command Reference</a></p>
<p>Here is an example for the us-east-1 region:
<div class="code-snippet">
<details>
<summary markdown="span">Click here to see the code snippet</summary>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> aws ec2 describe-instance-type-offerings --location-type availability-zone-id --filters <span style="color:#79c0ff">Name</span><span style="color:#ff7b72;font-weight:bold">=</span>instance-type,Values<span style="color:#ff7b72;font-weight:bold">=</span>mac1.metal --region us-east-1 --output text
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>INSTANCETYPEOFFERINGS mac1.metal use1-az6 availability-zone-id
</span></span><span style="display:flex;"><span>INSTANCETYPEOFFERINGS mac1.metal use1-az4 availability-zone-id
</span></span></code></pre></div>
</details>
</div></p>
<p>Keep that nuance in mind when selecting a subnet for the mac1.metal instances.</p>
<p>When you know the AZ, specify the respective Subnet in the Auto Scaling Group settings, and you’re ready to go!</p>
<h2 id="bring-infrastructure-as-code-here">Bring Infrastructure as Code here</h2>
<p>I suggest describing all that as a code. I prefer Terraform, and its AWS provider supports the needed resources. Except one.</p>
<p>As of October 2021, resources supported :</p>
<ul>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/servicequotas_service_quota">aws_servicequotas_service_quota</a></li>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/licensemanager_license_configuration">aws_licensemanager_license_configuration</a></li>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template">aws_launch_template</a></li>
<li><a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group">aws_autoscaling_group</a></li>
</ul>
<p>The Host resource group is not yet supported by the provider, unfortunately. However, we can use CloudFormation in Terraform to overcome that: describe the Host resource group as <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudformation_stack">aws_cloudformation_stack</a> Terraform resource using CloudFormation template from a file.</p>
<p>Here is how it looks like:
<div class="code-snippet">
<details>
<summary markdown="span">Click here to see the code snippet</summary>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_licensemanager_license_configuration" "this"</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">local</span>.<span style="color:#ff7b72">full_name</span>
</span></span><span style="display:flex;"><span> license_counting_type <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"Socket"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_cloudformation_stack" "this"</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">local</span>.<span style="color:#ff7b72">full_name</span><span style="color:#8b949e;font-style:italic"> # the name of CloudFormation stack
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span> template_body <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">file</span>(<span style="color:#a5d6ff">"${path.module}/resource-group-cf-stack-template.json"</span>)
</span></span><span style="display:flex;"><span> parameters <span style="color:#ff7b72;font-weight:bold">=</span> {
</span></span><span style="display:flex;"><span> GroupName <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">local</span>.<span style="color:#ff7b72">full_name</span><span style="color:#8b949e;font-style:italic"> # the name for the Host group, passed to CloudFormation template
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span> }
</span></span><span style="display:flex;"><span> on_failure <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"DELETE"</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div>
</details>
</div></p>
<p>And the next code snippet explains the CloudFromation template (which is the <code>resource-group-cf-stack-template.json</code> file in the code snippet above)
<div class="code-snippet">
<details>
<summary markdown="span">Click here to see the code snippet</summary>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Parameters"</span> : {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"GroupName"</span> : {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Type"</span> : <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Description"</span> : <span style="color:#a5d6ff">"The name of Host Group"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Resources"</span> : {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"DedicatedHostGroup"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Type"</span>: <span style="color:#a5d6ff">"AWS::ResourceGroups::Group"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Properties"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: { <span style="color:#7ee787">"Ref"</span>: <span style="color:#a5d6ff">"GroupName"</span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Configuration"</span>: [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Type"</span>: <span style="color:#a5d6ff">"AWS::ResourceGroups::Generic"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Parameters"</span>: [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"allowed-resource-types"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"AWS::EC2::Host"</span>]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"deletion-protection"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"UNLESS_EMPTY"</span>]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Type"</span>: <span style="color:#a5d6ff">"AWS::EC2::HostManagement"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Parameters"</span>: [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"allowed-host-families"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"mac1"</span>]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"auto-allocate-host"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"true"</span>]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"auto-release-host"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"true"</span>]
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Name"</span>: <span style="color:#a5d6ff">"any-host-based-license-configuration"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Values"</span>: [<span style="color:#a5d6ff">"true"</span>]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Outputs"</span> : {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"ResourceGroupARN"</span> : {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Description"</span>: <span style="color:#a5d6ff">"ResourceGroupARN"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"Value"</span> : { <span style="color:#7ee787">"Fn::GetAtt"</span> : [<span style="color:#a5d6ff">"DedicatedHostGroup"</span>, <span style="color:#a5d6ff">"Arn"</span>] }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div>
</details>
</div></p>
<p>The <code>aws_cloudformation_stack</code> resource will export the <code>DedicatedHostGroup</code> attribute (see the code of CloudFromation template), which you will use later in the Launch Template resource.</p>
<div class="attention">
<p>If you manage an AWS Organization, I have good news: Host groups and Licenses are supported by <a href="https://docs.aws.amazon.com/ram/latest/userguide/shareable.html">Resource Access Manager</a> service.</p>
<p>Hence, you can host all mac instances in one account and share them with other accounts β it might be helpful for costs allocation, for example. Also, check out <a href="https://devdosvid.blog/2021/09/25/aws-resource-access-manager-multi-account-resource-governance/">my blog about AWS RAM</a> if you are very new to this service.</p>
<p><br>
And you can leverage the <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type_offerings">aws_ec2_instance_type_offerings</a> and <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet_ids">aws_subnet_ids</a> data sources to solve the βwhich AZ supports mac metalβ puzzle.</p>
</div>
<h2 id="costs-considerations">Costs considerations</h2>
<p>License Manager is a <a href="https://aws.amazon.com/license-manager/pricing/">free of charge service</a>, as well as <a href="https://aws.amazon.com/autoscaling/pricing/">Auto Scaling</a>, and <a href="https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-launch-templates-for-amazon-ec2-instances/">Launch Template</a>.</p>
<p>So itβs all about the price for mac1.metal Dedicated Host which is <a href="https://aws.amazon.com/ec2/dedicated-hosts/pricing/">$1.083 per hour</a> as of October 2021. However, <a href="https://docs.aws.amazon.com/savingsplans/latest/userguide/what-is-savings-plans.html">Saving Plans</a> can be applied.</p>
<p>Please note that the minimum allocation time for that type of host is 24 hours. Maybe someday AWS will change that to 1-hour minimum someday (fingers crossed).</p>
<h2 id="oh-so-asg">Oh. So. ASG.</h2>
<p>The Auto Scaling for mac1.metal opens new possibilities for CI/CD: you can integrate that to your favorite tool (GitLab, Jenkins, whatsoever) using AWS Lambda and provision new instances when your development/testing environments need that.</p>
<p>Or you can use other cool ASG stuff, such as Lifecycle hooks, to create even more custom scenarios.</p>
<p>Also, I want to say thanks (thanks, pal!) to <a href="https://github.com/hashicorp/terraform/issues/28531">OliverKoo</a>, who started digging into that back in April'21.</p>
AWS Resource Access Manager β Multi Account Resource Governance
2021-09-24T21:54:23Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/09/25/aws-resource-access-manager-multi-account-resource-governance/
Serhii Vasylenko
https://devdosvid.blog/2021/09/25/aws-resource-access-manager-multi-account-resource-governance/cover-image.png
<p>With a multi-account approach of building the infrastructure, there is always a challenge of provision and governance of the resources to subordinate accounts within the Organization. Provision resources, keep them up to date, and decommission them properly β that’s only a part of them.</p>
<p>AWS has numerous solutions that help make this process reliable and secure, and the Resource Access Manager (RAM) is one of them.
In a nutshell, the RAM service allows you to share the AWS resources you create in one AWS account with other AWS accounts. They can be your organizations’ accounts, organizational units (OU), or even third-party accounts.</p>
<p>So let’s see what the RAM is and review some of its usage examples.</p>
<h2 id="why-using-ram">Why using RAM</h2>
<p>There are several benefits of using the RAM service:</p>
<ol>
<li>
<p><strong>Reduced operational overhead</strong>: eliminate the need of provisioning the same kind of resource multiple times β RAM does that for you</p>
</li>
<li>
<p><strong>Simplified security management</strong>: AWS RAM-managed permissions (at least one per resource type) define the actions that principals with access to the resources (i.e., resource users) can perform on those resources.</p>
</li>
<li>
<p><strong>Consistent experience</strong>: you share the resource in its state and with its security configuration with an arbitrary number of accounts.</p>
<p>That plays incredibly well in the case of organization-wide sharing: new accounts get the resources automatically. And the shared resource itself looks like a native resource in the account that accepts your sharing.</p>
</li>
<li>
<p><strong>Audit and visibility</strong>: RAM integrates with the CloudWatch and CloudTrail.</p>
</li>
</ol>
<h2 id="how-to-share-a-resource">How to share a resource</h2>
<p>When you share a resource, the AWS account that owns that resource retains full ownership of the resource.</p>
<p>Sharing of the resource doesn’t change any permissions or quotas that apply to that resource. Also, you can share the resource only if you own it.</p>
<p>Availability of the shared resources scopes to the Region: the users of your shared resources can access these resources only in the same Region where resources belong.</p>
<p>Creation of resource share consists of three steps:
<figure>
<img loading="lazy"
src="ram-diagram-800.png"width="800"
height="533"
/>
</figure>
</p>
<ol>
<li>
<p>Specify the share name and the resource(s) you want to share. It can be either one resource type or several. You can also skip the resources selection and do that later.</p>
<p>It’s possible to modify the resource share later (e.g., you want to add some resources to the share).</p>
</li>
<li>
<p>Associate permissions with resource types you share. Some resources can have only one managed permission (will be attached automatically), and some can have multiple.</p>
<p>You can check the Permissions Library in the AWS RAM Console to see what managed permissions are available.</p>
</li>
<li>
<p>Select who can use the resources you share: either external or Organization account or IAM role/user. If you share the resource with third parties, they will have to accept the sharing explicitly.</p>
<p>Organization-wide resource share is accepted implicitly if resource sharing is enabled for the Organization.</p>
</li>
</ol>
<p>Finally, review the summary page of the resource share and create it.</p>
<p>Only specific actions are available to the users of shared resources. These actions mostly have the “read-only” nature and <a href="https://docs.aws.amazon.com/ram/latest/userguide/shareable.html">vary by resource type</a>.</p>
<p>Also, the RAM service is <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ram_resource_share">supported by Terraform</a>, so the resource sharing configuration may look like that, for example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_ram_resource_share" "example"</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"example"</span>
</span></span><span style="display:flex;"><span> allow_external_principals <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">false</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> tags <span style="color:#ff7b72;font-weight:bold">=</span> {
</span></span><span style="display:flex;"><span> Environment <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"Production"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_ram_resource_association" "example"</span> {
</span></span><span style="display:flex;"><span> resource_arn <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">aws_subnet</span>.<span style="color:#ff7b72">example</span>.<span style="color:#ff7b72">arn</span>
</span></span><span style="display:flex;"><span> resource_share_arn <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">aws_ram_resource_share</span>.<span style="color:#ff7b72">example</span>.<span style="color:#ff7b72">arn</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="example-use-cases">Example use cases</h2>
<p>One of the trivial but valuable examples of RAM service usage is sharing a Manged Prefix List.
Suppose you have some service user across your Organization, a self-hosted VPN server, for example. And you have a static set of IPs for that VPN: you trust these IPs and would like them to be allow-listed in your other services.
How to report these IPs to all organization accounts/users? And if the IP set changes, how to announce that change, and what should be done to reflect that change in services that depend on it, for example, Security Groups?</p>
<p>The answer is a shared <a href="https://docs.aws.amazon.com/vpc/latest/userguide/managed-prefix-lists.html#managed-prefix-lists-concepts">Managed Prefix List</a>. You create the list once in the account and share it across your Organization. Other accounts automatically get access to that list and can reference the list in their Security Groups. And when the list entry is changed, they do not need to perform any actions: their Security Groups will get the updated IPs implicitly.</p>
<p>Another everyday use case of RAM is the VPC sharing that can form the foundation of the <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-sharing-a-new-approach-to-multiple-accounts-and-vpc-management/">multi-account AWS architectures</a>.</p>
<hr>
<p>Of course, the RAM service is not the only way to organize and centralize resource management in AWS. There are Service Catalog, Control Tower, Systems Manager, Config, and others. However, the RAM is relatively simple to adopt but is capable of providing worthy outcomes.</p>
Run Ansible playbook on Mac EC2 Instances fleet with AWS Systems Manager
2021-05-27T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/05/27/run-ansible-playbook-on-mac-ec2-instances-fleet-with-aws-systems-manager/
Serhii Vasylenko
https://devdosvid.blog/2021/05/27/run-ansible-playbook-on-mac-ec2-instances-fleet-with-aws-systems-manager/cover-image.png
<p>In days of containers and serverless applications, Ansible looks not such a trendy thing.</p>
<p>But still, there are cases when it helps, and there are cases when it combines very well with brand new product offerings, such as EC2 Mac instances.</p>
<p>The <a href="https://devdosvid.blog/2021/02/01/customizing-mac1-metal-ec2-ami.html">more I use mac1.metal</a> in AWS, the more I see that Ansible becomes a bedrock of software customization in my case.</p>
<p>And when you have a large instances fleet, the AWS Systems Manager becomes your best friend (the sooner you get along together, the better).</p>
<p>So is it possible to use Ansible playbooks for mac1.metal on a big scale, with the help of AWS Systems Manager?</p>
<h2 id="not-available-out-of-the-box">(Not) Available out of the box</h2>
<p>AWS Systems Manager (SSM hereafter) has a pre-defined, shared Document that allows running Ansible playbooks.</p>
<p>Itβs called βAWS-RunAnsiblePlaybook,β and you can find it in AWS SSM β Documents β Owned by Amazon.</p>
<p>However, this Document is not quite βfriendlyβ to macOS. When the SSM agent calls Ansible on the Mac EC2 instance, it does not recognize the Ansible installed with Homebrew (de-facto most used macOS package manager).</p>
<p>So if you try to run a command on the mac1.metal instance using this Document, you will get the following error:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>Ansible is not installed. Please install Ansible and rerun the command.
</span></span></code></pre></div><p>The root cause is trivial: the path to Ansible binary is not present on the list of paths available to the SSM agent by default.</p>
<p>There are several ways to solve that, but I believe that the most convenient one would be to create your custom Document β a slightly adjusted version of the default one provided by AWS.</p>
<h2 id="creating-own-ssm-document-for-ansible-installed-with-homebrew">Creating own SSM Document for Ansible installed with Homebrew</h2>
<p>All you need to do is clone the Document provided by AWS and change its code a little β replace the callouts of <code>ansible</code> with the full path to the binary.</p>
<p>Navigate to AWS SSM β Documents β Owned by Amazon and type <code>AWS-RunAnsiblePlaybook</code> in the search field.</p>
<p>Select the Document by pressing the circle on its top-right corner and then click Actions β Clone document.</p>
<figure>
<img loading="lazy"
src="aws_ssm_document_clone.png"width="800"
height="476.68"
/>
</figure>
<p>Give the new SSM Document a name, e.g., <code>macos-arbitrary-ansible-playbook</code>, and change the <code>ansible</code> callouts (at the end of the code) with the full path to the ansible symlink made by Homebrew which is <code>/usr/local/bin/ansible</code></p>
<p>Here is the complete source code of the Document with adjusted Ansible path:</p>
<div class="code-snippet">
<details>
<summary markdown="span">Click here to see the code snippet</summary>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"schemaVersion"</span>: <span style="color:#a5d6ff">"2.0"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">"Use this document to run arbitrary Ansible playbooks on macOS EC2 instances. Specify either YAML text or URL. If you specify both, the URL parameter will be used. Use the extravar parameter to send runtime variables to the Ansible execution. Use the check parameter to perform a dry run of the Ansible execution. The output of the dry run shows the changes that will be made when the playbook is executed."</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"parameters"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"playbook"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"type"</span>: <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">"(Optional) If you don't specify a URL, then you must specify playbook YAML in this field."</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"default"</span>: <span style="color:#a5d6ff">""</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"displayType"</span>: <span style="color:#a5d6ff">"textarea"</span>
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"playbookurl"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"type"</span>: <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">"(Optional) If you don't specify playbook YAML, then you must specify a URL where the playbook is stored. You can specify the URL in the following formats: http://example.com/playbook.yml or s3://examplebucket/plabook.url. For security reasons, you can't specify a URL with quotes."</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"default"</span>: <span style="color:#a5d6ff">""</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"allowedPattern"</span>: <span style="color:#a5d6ff">"^\\s*$|^(http|https|s3)://[^']*$"</span>
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"extravars"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"type"</span>: <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">"(Optional) Additional variables to pass to Ansible at runtime. Enter a space separated list of key/value pairs. For example: color=red or fruits=[apples,pears]"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"default"</span>: <span style="color:#a5d6ff">"foo=bar"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"displayType"</span>: <span style="color:#a5d6ff">"textarea"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"allowedPattern"</span>: <span style="color:#a5d6ff">"^((^|\\s)\\w+=(\\S+|'.*'))*$"</span>
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"check"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"type"</span>: <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">" (Optional) Use the check parameter to perform a dry run of the Ansible execution."</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"allowedValues"</span>: [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"True"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"False"</span>
</span></span><span style="display:flex;"><span> ],
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"default"</span>: <span style="color:#a5d6ff">"False"</span>
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"timeoutSeconds"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"type"</span>: <span style="color:#a5d6ff">"String"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"description"</span>: <span style="color:#a5d6ff">"(Optional) The time in seconds for a command to be completed before it is considered to have failed."</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"default"</span>: <span style="color:#a5d6ff">"3600"</span>
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> },
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"mainSteps"</span>: [
</span></span><span style="display:flex;"><span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"action"</span>: <span style="color:#a5d6ff">"aws:runShellScript"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"name"</span>: <span style="color:#a5d6ff">"runShellScript"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"inputs"</span>: {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"timeoutSeconds"</span>: <span style="color:#a5d6ff">"{{ timeoutSeconds }}"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"runCommand"</span>: [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"#!/bin/bash"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"/usr/local/bin/ansible --version"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"if [ $? -ne 0 ]; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo \"Ansible is not installed. Please install Ansible and rerun the command\" >&2"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" exit 1"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"execdir=$(dirname $0)"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"cd $execdir"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"if [ -z '{{playbook}}' ] ; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" if [[ \"{{playbookurl}}\" == http* ]]; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" wget '{{playbookurl}}' -O playbook.yml"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" if [ $? -ne 0 ]; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo \"There was a problem downloading the playbook. Make sure the URL is correct and that the playbook exists.\" >&2"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" exit 1"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" elif [[ \"{{playbookurl}}\" == s3* ]] ; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" aws --version"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" if [ $? -ne 0 ]; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo \"The AWS CLI is not installed. The CLI is required to process Amazon S3 URLs. Install the AWS CLI and run the command again.\" >&2"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" exit 1"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" aws s3 cp '{{playbookurl}}' playbook.yml"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" if [ $? -ne 0 ]; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo \"Error while downloading the document from S3\" >&2"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" exit 1"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" else"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo \"The playbook URL is not valid. Verify the URL and try again.\""</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"else"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" echo '{{playbook}}' > playbook.yml"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"fi"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"if [[ \"{{check}}\" == True ]] ; then"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" /usr/local/bin/ansible-playbook -i \"localhost,\" --check -c local -e \"{{extravars}}\" playbook.yml"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"else"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">" /usr/local/bin/ansible-playbook -i \"localhost,\" -c local -e \"{{extravars}}\" playbook.yml"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"fi"</span>
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span> ]
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div>
</details>
</div>
<h2 id="applying-ansible-playbook-to-the-fleet-of-mac1metal">Applying Ansible playbook to the fleet of mac1.metal</h2>
<p>Letβs give our new SSM Document a try! (I suppose you have at least one mac1 instance running, right?)</p>
<p>In AWS SSM, go to the Run Command feature, then click on the Run Command button.</p>
<p>On the new panel, type the name of your Document (<code>macos-arbitrary-ansible-playbook</code> in this example) in the search field and press enter.</p>
<p>Select the Document, and youβll see its parameters and settings.</p>
<p>The rest is self-explanatory. Enter either a playbook code or a link to the source file, add extra variables if needed, and select the target host or a filtered bunch (I like that feature with tags filtering!). Finally, click on the βRunβ orange button to apply your playbook.</p>
<p>Thatβs it! Now you can make all your ansible-playbook dreams come true! π</p>
Configure HTTP Security headers with CloudFront Functions
2021-05-21T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/05/21/configure-http-security-headers-with-cloudfront-functions/
Serhii Vasylenko
https://devdosvid.blog/2021/05/21/configure-http-security-headers-with-cloudfront-functions/cover-image.png
<div class="updatenotice">
<p>In November 2021, AWS has added this functionality as a native CloudFront feature.</p>
<p>I suggest switching to the native implementation. I have described how to configure Security Response Headers for CloudFront in the following article:</p>
<p><a href="https://devdosvid.blog/2021/11/05/apply-cloudfront-security-headers-with-terraform/">Apply Cloudfront Security Headers With Terraform</a></p>
</div>
<p>A couple of weeks ago, AWS released CloudFront Functions β a βtrue edgeβ compute capability for the CloudFront.</p>
<p>It is βtrue edgeβ because Functions work on 200+ edge locations (<a href="https://aws.amazon.com/cloudfront/features/?whats-new-cloudfront.sort-by=item.additionalFields.postDateTime&whats-new-cloudfront.sort-order=desc#Edge_Computing">link to doc</a>) while its predecessor, the Lambda@Edge, runs on a small number of regional edge caches.</p>
<p>One of the use cases for Lambda@Edge was adding security HTTP headers (itβs even listed on the <a href="https://aws.amazon.com/lambda/edge/">product page</a>), and now there is one more way to make it using CloudFront Functions.</p>
<div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<h2 id="what-are-security-headers-and-why-it-matters">What are security headers, and why it matters</h2>
<p>Security Headers are one of the web security pillars.</p>
<p>They specify security-related information of communication between a web application (i.e., website) and a client (i.e., browser) and protect the web app from different types of attacks. Also, HIPAA and PCI, and other security standard certifications generally include these headers in their rankings.</p>
<p>We will use CloudFront Functions to set the following headers:</p>
<ul>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#content-security-policy">Content Security Policy</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#http-strict-transport-security">Strict Transport Security</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-content-type-options">X-Content-Type-Options</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-xss-protection">X-XSS-Protection</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#x-frame-options">X-Frame-Options</a></li>
<li><a href="https://infosec.mozilla.org/guidelines/web_security#referrer-policy">Referrer Policy</a></li>
</ul>
<p>You can find a short and detailed explanation for each security header on <a href="https://infosec.mozilla.org/guidelines/web_security">Web Security cheatsheet made by Mozilla</a></p>
<h2 id="cloudfront-functions-overview">CloudFront Functions overview</h2>
<p>In a nutshell, CloudFront Functions allow performing simple actions against HTTP(s) request (from the client) and response (from the CloudFront cache at the edge). Functions take less than one millisecond to execute, support JavaScript (ECMAScript 5.1 compliant), and cost $0.10 per 1 million invocations.</p>
<p>Every CloudFront distribution has one (default) or more Cache behaviors, and Functions can be associated with these behaviors to execute upon a specific event.</p>
<p>That is how the request flow looks like in general, and here is where CloudFront Functions execution happens:</p>
<figure>
<img loading="lazy"
src="request_flow.png"width="800"
height="171.67"
/>
</figure>
<p>CloudFront Functions support Viewer Request (after CloudFront receives a request from a client) and Viewer Response (before CloudFront forwards the response to the client) events.</p>
<p>You can read more about the events types and their properties here β <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-cloudfront-trigger-events.html">CloudFront Events That Can Trigger a Lambda Function - Amazon CloudFront</a>.</p>
<p>Also, the CloudFront Functions allow you to manage and operate the code and lifecycle of the functions directly from the CloudFront web interface.</p>
<h2 id="solution-overview">Solution overview</h2>
<p>CloudFront distribution should exist before Function creation so you could associate the Function with the distribution.</p>
<p>Creation and configuration of the CloudFront Function consist of the following steps:</p>
<h3 id="create-function">Create Function</h3>
<p>In the AWS Console, open CloudFront service and lick on the Functions on the left navigation bar, then click Create function button.
<figure>
<img loading="lazy"
src="create_function.png"width="800"
height="255.83"
/>
</figure>
Enter the name of your Function (e.g., βsecurity-headersβ) and click Continue.</p>
<h3 id="build-function">Build Function</h3>
<p>On the function settings page, you will see four tabs with the four lifecycle steps: Build, Test, Publish, Associate.</p>
<p>Paste the function code into the editor and click βSave.β</p>
<figure>
<img loading="lazy"
src="function_editor.png"width="800"
height="467.84"
/>
</figure>
<p>Here is the source code of the function:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-javascript" data-lang="javascript"><span style="display:flex;"><span><span style="color:#ff7b72">function</span> handler(event) {
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">var</span> response <span style="color:#ff7b72;font-weight:bold">=</span> event.response;
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">var</span> headers <span style="color:#ff7b72;font-weight:bold">=</span> response.headers;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'strict-transport-security'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> { value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">'max-age=63072000; includeSubdomains; preload'</span>};
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'content-security-policy'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> { value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">"default-src 'none'; img-src 'self'; script-src 'self'; style-src 'self'; object-src 'none'; frame-ancestors 'none'"</span>};
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'x-content-type-options'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> { value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">'nosniff'</span>};
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'x-xss-protection'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> {value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">'1; mode=block'</span>};
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'referrer-policy'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> {value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">'same-origin'</span>};
</span></span><span style="display:flex;"><span>headers[<span style="color:#a5d6ff">'x-frame-options'</span>] <span style="color:#ff7b72;font-weight:bold">=</span> {value<span style="color:#ff7b72;font-weight:bold">:</span> <span style="color:#a5d6ff">'DENY'</span>};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">return</span> response;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h3 id="test-function">Test Function</h3>
<p>Open the βTestβ tab β letβs try our function first before it becomes live!</p>
<p>Select Viewer Response event type and Development Stage, then select βViewer response with headersβ as a Sample test event (you will get a simple set of headers automatically).</p>
<p>Now click the blue βTestβ button and observe the output results:</p>
<ul>
<li>Compute utilization represents the relative amount of time (on a scale between 0 and 100) your function took to run</li>
<li>Check the Response headers tab and take a look at how the function added custom headers.</li>
</ul>
<figure>
<img loading="lazy"
src="function_test.png"width="800"
height="624.03"
/>
</figure>
<h3 id="publish-function">Publish Function</h3>
<p>Letβs publish our function. To do that, open the Publish tab and click on the blue button βPublish and update.β
<figure>
<img loading="lazy"
src="function_publish.png"width="800"
height="257.24"
/>
</figure>
</p>
<h3 id="associate-your-function-with-cloudfront-distribution">Associate your Function with CloudFront distribution</h3>
<p>Now, you can associate the function with the CloudFront distribution.</p>
<p>To do so, open the Associate tab, select the distribution and event type (Viewer Response), and select the Cache behavior of your distribution which you want to use for the association.</p>
<figure>
<img loading="lazy"
src="function_associate.png"width="800"
height="366.08"
/>
</figure>
<p>Once you associate the function with the CloudFront distribution, you can test it in live mode.</p>
<p>I will use curl here to demonstrate it:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>> curl -i https://d30i87a4ss9ifz.cloudfront.net
</span></span><span style="display:flex;"><span>HTTP/2 <span style="color:#a5d6ff">200</span>
</span></span><span style="display:flex;"><span>content-type: text/html
</span></span><span style="display:flex;"><span>content-length: <span style="color:#a5d6ff">140</span>
</span></span><span style="display:flex;"><span>date: Sat, <span style="color:#a5d6ff">22</span> May <span style="color:#a5d6ff">2021</span> 00:22:18 GMT
</span></span><span style="display:flex;"><span>last-modified: Tue, <span style="color:#a5d6ff">27</span> Apr <span style="color:#a5d6ff">2021</span> 23:07:14 GMT
</span></span><span style="display:flex;"><span>etag: <span style="color:#a5d6ff">"a855a3189f8223db53df8a0ca362dd62"</span>
</span></span><span style="display:flex;"><span>accept-ranges: bytes
</span></span><span style="display:flex;"><span>server: AmazonS3
</span></span><span style="display:flex;"><span>via: 1.1 50f21cb925e6471490e080147e252d7d.cloudfront.net <span style="color:#ff7b72;font-weight:bold">(</span>CloudFront<span style="color:#ff7b72;font-weight:bold">)</span>
</span></span><span style="display:flex;"><span>content-security-policy: default-src <span style="color:#a5d6ff">'none'</span>; img-src <span style="color:#a5d6ff">'self'</span>; script-src <span style="color:#a5d6ff">'self'</span>; style-src <span style="color:#a5d6ff">'self'</span>; object-src <span style="color:#a5d6ff">'none'</span>; frame-ancestors <span style="color:#a5d6ff">'none'</span>
</span></span><span style="display:flex;"><span>strict-transport-security: max-age<span style="color:#ff7b72;font-weight:bold">=</span>63072000; includeSubdomains; preload
</span></span><span style="display:flex;"><span>x-xss-protection: 1; <span style="color:#79c0ff">mode</span><span style="color:#ff7b72;font-weight:bold">=</span>block
</span></span><span style="display:flex;"><span>x-frame-options: DENY
</span></span><span style="display:flex;"><span>referrer-policy: same-origin
</span></span><span style="display:flex;"><span>x-content-type-options: nosniff
</span></span><span style="display:flex;"><span>x-cache: Miss from cloudfront
</span></span><span style="display:flex;"><span>x-amz-cf-pop: WAW50-C1
</span></span><span style="display:flex;"><span>x-amz-cf-id: ud3qH8rLs7QmbhUZ-DeupGwFhWLpKDSD59vr7uWC65Hui5m2U8o2mw<span style="color:#ff7b72;font-weight:bold">==</span>
</span></span></code></pre></div><p>You can also test your results here β <a href="https://observatory.mozilla.org/">Mozilla Observatory</a></p>
<p><figure>
<img loading="lazy"
src="scan_result-1.png"width="800"
height="345.58"
/>
</figure>
<figure>
<img loading="lazy"
src="scan_result-2.png"width="800"
height="626.15"
/>
</figure>
</p>
<h2 id="read-more">Read more</h2>
<p>That was a simplified overview of the CloudFront Functions capabilities.</p>
<p>But if you want to get deeper, here is a couple of useful links to start:</p>
<ul>
<li>Another overview from AWS β <a href="https://aws.amazon.com/blogs/aws/introducing-cloudfront-functions-run-your-code-at-the-edge-with-low-latency-at-any-scale">CloudFront Functions Launch Blog</a></li>
<li>More about creating, testing, updating and publishing of CloudFront Functions β <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/managing-functions.html">Managing functions in CloudFront Functions - Amazon CloudFront</a></li>
</ul>
<h2 id="so-what-to-choose">So what to choose?</h2>
<p>CloudFront Functions are simpler than Lambda@Edge and run faster with minimal latency and minimal time penalty for your web clients.</p>
<p>Lambda@Edge takes more time to invoke, but it can run upon Origin Response event so that CloudFront can cache the processed response (including headers) and return it faster afterward.</p>
<p>But again, the CloudFront Functions invocations are much cheaper (6x times) than Lambda@Edge, and you do not pay for the function execution duration.</p>
<p>The final decision would also depend on the dynamic/static nature of the content you have at your origin.</p>
<p>To make a wise and deliberate decision, try to analyze your use case using these two documentation articles:</p>
<ul>
<li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/edge-functions.html">Choosing between CloudFront Functions and Lambda@Edge</a></li>
<li><a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-how-to-choose-event.html">How to Decide Which CloudFront Event to Use to Trigger a Lambda Function</a></li>
</ul>
Using TinyPNG Image Compression From MacOS Finder Contextual Menu
2021-02-14T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/02/14/using-tinypng-image-compression-from-macos-finder-contextual-menu/
Serhii Vasylenko
https://devdosvid.blog/2021/02/14/using-tinypng-image-compression-from-macos-finder-contextual-menu/cover-image.png
<p>I just wanted to compress one image, but went to far…</p>
<p>or “How to add TinyPNG image compression to your macOS Finder contextual menu.”</p>
<h1 id="what-is-it-and-how-it-works">What is it and how it works</h1>
<p>You select needed files or folders, then right-click on them, click on the Services menu item and choose TinyPNG.</p>
<p>After a moment, the new optimized versions of images will appear near to original files.</p>
<p>If you selected a folder along with the files, the script would process all <code>png</code> and <code>jpeg</code> files in it.</p>
<video class="animation" autoplay loop muted playsinline>
<source src="context_menu_full_compressed.webm" type="video/webm">
</video>
<h1 id="prerequisites">Prerequisites</h1>
<p>You need to register at TinyPNG and get your API key here β <a href="https://tinypng.com/developers">Developer API</a>.</p>
<p>They sometimes block some countries (for example, Ukraine) from registration; in that case, try to use a web-proxy or VPN.</p>
<h1 id="how-to-create-quick-action-workflow">How to create Quick Action Workflow</h1>
<p>Open Automator application. If you never used this app before, please read about it on the official <a href="https://support.apple.com/guide/automator/create-a-workflow-aut7cac58839/2.10/mac/11.0">user guide website</a>.</p>
<p>On the New Action screen, chose <strong>Quick Action</strong></p>
<figure>
<img loading="lazy"
src="quick_action_compressed.png"width="800"
height="695.62"
/>
</figure>
<p>After you click the “Choose” button, you’ll see the workflow configuration window.</p>
<h1 id="workflow-configuration">Workflow configuration</h1>
<p>Find the <strong>Run Shell Script</strong> action on the Utilities list in Library on the left, and drag it onto the right side of the panel.</p>
<p>Set the following workflow configuration options as described below:</p>
<p><strong>Workflow receives current</strong> <code>files and folders</code> <strong>in</strong> <code>Finder</code></p>
<p><strong>Shell</strong> <code>/bin/zsh</code></p>
<p><strong>Pass input</strong> <code>as arguments</code></p>
<p>Click the <strong>Option</strong> button at the bottom of the Action window and <strong>Uncheck</strong> <code>Show this action when the workflow runs.</code></p>
<figure>
<img loading="lazy"
src="run_shell_script_compressed.png"width="800"
height="702.36"
/>
</figure>
<p>Put the following script into the <strong>Run Shell Script</strong> window, replacing the <em>YOUR_API_KEY_HERE</em> string with your API key obtained from TinyPNG.</p>
<script src="https://gist.github.com/vasylenko/13cb423aa83265e79ac5ad900195603f.js"></script>
<h2 id="utilities-used-in-the-script--explained">Utilities used in the script β explained</h2>
<p><code>curl</code> β used to make web requests (like your browser does)</p>
<p><code>grep</code> β used to parse the response for the needed header (i.e., field) with the file download link</p>
<p><code>cut</code> β used to extract the URL from the parsed result</p>
<p><code>sed</code> β used to remove the trailing “carriage return” symbol at the end of extracted string</p>
<p>The response body also contains a JSON object that includes the download URL; you can parse it with <code>jq</code>, for example. But I intentionally refused to use the <code>jq</code> tool because it is not pre-installed in MacOS.</p>
<h1 id="conclusion">Conclusion</h1>
<p>It is simple, and it does its job fine. And you don’t need to install anything to make it work.</p>
<p>To make this a bit fancier, you might also like to add a “Display Notification” (from the Utilities library on the left) after the “Run Shell Script”. The action will display a notification once image processing is completed.</p>
<p>Thank you for reading!</p>
Customizing mac1.metal EC2 AMI β new guts, more glory
2021-02-01T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/02/01/customizing-mac1.metal-ec2-ami-new-guts-more-glory/
Serhii Vasylenko
https://devdosvid.blog/2021/02/01/customizing-mac1.metal-ec2-ami-new-guts-more-glory/cover-image.png
<p>I guess macOS was designed for a user, not for the ops or engineers, so this is why its customization and usage for CI/CD are not trivial (compared to something Linux-based). A smart guess, huh?</p>
<h1 id="configuration-management">Configuration Management</h1>
<p>Native Apple’s Mobile device management (a.k.a MDM) and Jamf is probably the most potent combination for macOS configuration. But as much as it’s mighty, it is a cumbersome combination, and Jamf is not free.</p>
<p>Then we have Ansible, Chef, Puppet, SaltStack β they all are good with Linux, but what about macOS?</p>
<p>I tried to search for use cases of mentioned CM tools for macOS. However, I concluded that they wrap the execution of native macOS command-line utilities most of the time.</p>
<p>And if you search for the ‘macos’ word in Chef Supermarket or Puppet Forge, you won’t be impressed by the number of actively maintained packages. Although, here is a motivating article about using Chef <a href="https://pspdfkit.com/blog/2016/chef-on-macos/">automating-macos-provisioning-with-chef</a> if you prefer it. I could not find something similar and fresh for Puppet, so I am sorry, Puppet fans.</p>
<p>That is why I decided to follow the KISS principle and chose Ansible.</p>
<p>It’s easy to write and read the configuration, it allows to group tasks and to add execution logic <del>, and it feels more DevOps executing shell commands inside Ansible tasks instead of shell scripts; I know you know that π</del></p>
<p>By the way, Ansible Galaxy does not have many management packages for macOS, either. But thankfully, it has the basics:</p>
<ul>
<li><a href="https://docs.ansible.com/ansible/latest/collections/community/general/homebrew_module.html#ansible-collections-community-general-homebrew-module">homebrew</a> with <a href="https://docs.ansible.com/ansible/latest/collections/community/general/homebrew_cask_module.html#ansible-collections-community-general-homebrew-cask-module">homebrew_cask</a> and <a href="https://docs.ansible.com/ansible/latest/collections/community/general/homebrew_tap_module.html#ansible-collections-community-general-homebrew-tap-module">homebrew_tap</a> β to install software</li>
<li><a href="https://docs.ansible.com/ansible/latest/collections/community/general/launchd_module.html#ansible-collections-community-general-launchd-module">launchd</a> β to manage services</li>
<li><a href="https://docs.ansible.com/ansible/latest/collections/community/general/osx_defaults_module.html#ansible-collections-community-general-osx-defaults-module">osx_defaults</a> β to manage some user settings (not all!)</li>
</ul>
<p>I used Ansible to build the macOS AMI for CI/CD, so here are some tips for such a case.</p>
<p><em>Some values are hardcoded intentionally in the code examples for the sake of simplicity and easy reading. You would probably want to parametrize them.</em></p>
<h2 id="xcode-installation-example">Xcode installation example</h2>
<p>The following tasks will help you to automate the basics.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Install Xcode</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"xip --expand Xcode.xip"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">args</span>:<span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">chdir</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">/Applications</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"></span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Accept License Agreement</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -license accept"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"></span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Accept License Agreement</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -runFirstLaunch"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"></span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Switch into newly installed Xcode context</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"xcode-select --switch /Applications/Xcode.app/Contents/Developer"</span><span style="color:#6e7681">
</span></span></span></code></pre></div><h2 id="example-of-software-installation-with-brew">Example of software installation with Brew</h2>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Install common build software</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">community.general.homebrew</span>:<span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"{{ item }}"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">state</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">latest</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">loop</span>:<span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span>- <span style="color:#a5d6ff">swiftlint</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span>- <span style="color:#a5d6ff">swiftformat</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span>- <span style="color:#a5d6ff">wget</span><span style="color:#6e7681">
</span></span></span></code></pre></div><h2 id="screensharing-remote-desktop-configuration-example">ScreenSharing (remote desktop) configuration example</h2>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Turn On Remote Management</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"./kickstart -activate -configure -allowAccessFor -specifiedUsers"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">args</span>:<span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">chdir</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"></span>- <span style="color:#7ee787">name</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">Enable Remote Management for CI user</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">shell</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">"./kickstart -configure -users ec2-user -access -on -privs -all"</span><span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">args</span>:<span style="color:#6e7681">
</span></span></span><span style="display:flex;"><span><span style="color:#6e7681"> </span><span style="color:#7ee787">chdir</span>:<span style="color:#6e7681"> </span><span style="color:#a5d6ff">/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/</span><span style="color:#6e7681">
</span></span></span></code></pre></div><p>Shell rulez, yes.</p>
<h1 id="building-the-ami">Building the AMI</h1>
<video class="animation" autoplay loop muted playsinline>
<source src="ami-build.webm" type="video/webm">
</video>
<p><a href="https://www.packer.io/docs/builders/amazon/ebs">Packer by HashiCorp</a>, of course.</p>
<p>I would love to compare Packer with EC2 Image Builder, but it <a href="https://docs.aws.amazon.com/imagebuilder/latest/userguide/what-is-image-builder.html#image-builder-os">does not support macOS</a> yet (as of Feb'21).</p>
<p>Packer configuration is straightforward, so I want to highlight only the things specific to the “mac1.metal” use case.</p>
<h2 id="timeouts">Timeouts</h2>
<p>As I mentioned in the <a href="https://devdosvid.blog/2021/01/19/mac1-metal-EC2-Instance-user-experience.html">previous article</a>, the creation and deletion time of the “mac1.metal” Instance is significantly bigger than Linux. That is why you should raise the polling parameters for the builder.</p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span><span style="color:#a5d6ff">"aws_polling"</span><span style="color:#f85149">:</span> {
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"delay_seconds"</span>: <span style="color:#a5d6ff">30</span>,
</span></span><span style="display:flex;"><span> <span style="color:#7ee787">"max_attempts"</span>: <span style="color:#a5d6ff">60</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And it would be best if you also increased the SSH timeout:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span> <span style="color:#a5d6ff">"ssh_timeout"</span><span style="color:#f85149">:</span> <span style="color:#a5d6ff">"1h"</span>
</span></span></code></pre></div><p>Fortunately, Packer’s AMI builder does not require an explicit declaration of the Dedicated Host ID. So you can just reference the same subnet where you allocated the Host, assuming you did it with the enabled “Auto placement” parameter during the host creation.</p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span> <span style="color:#a5d6ff">"tenancy"</span><span style="color:#f85149">:</span> <span style="color:#a5d6ff">"host"</span><span style="color:#f85149">,</span>
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"subnet_id"</span><span style="color:#f85149">:</span> <span style="color:#a5d6ff">"your-subnet-id"</span>
</span></span></code></pre></div><h2 id="provisioning">Provisioning</h2>
<p>Packer has <a href="https://www.packer.io/docs/provisioners/ansible">Ansible Provisioner</a> that I used for the AMI. Its documentation is also very clean and straightforward.</p>
<p>But it is still worth mentioning that if you want to parametrize the Ansible playbook, then the following configuration example will be handy:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-json" data-lang="json"><span style="display:flex;"><span> <span style="color:#a5d6ff">"extra_arguments"</span><span style="color:#f85149">:</span> [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"--extra-vars"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"your-variable-foo=your-value-bar]"</span>
</span></span><span style="display:flex;"><span> ]<span style="color:#f85149">,</span>
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"ansible_env_vars"</span><span style="color:#f85149">:</span> [
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"ANSIBLE_PYTHON_INTERPRETER=auto_legacy_silent"</span>,
</span></span><span style="display:flex;"><span> <span style="color:#a5d6ff">"ANSIBLE_OTHER_ENV_VARIABLE=other_value"</span>
</span></span><span style="display:flex;"><span> ]
</span></span></code></pre></div><h1 id="configuration-at-launch">Configuration at launch</h1>
<p>If you’re familiar with AWS EC2, you probably know what the Instance <code>user data</code> is.</p>
<p>A group of AWS developers made something similar for the macOS: <a href="https://github.com/aws/ec2-macos-init">EC2 macOS Init</a>.</p>
<p>It does not support <code>cloud-init</code> as on Linux-based Instances, but it can run shell scripts, which is quite enough.</p>
<p>EC2 macOS Init utility is a Launch Daemon (macOS terminology) that runs on behalf of the <code>root</code> user at system boot. It executes the commands according to the so-called Priority Groups, or the sequence in other words.</p>
<p>The number of the group corresponds to the execution order. You can put several tasks into a single Priority Group, and the tool will execute them simultaneously.</p>
<p>EC2 macOS Init uses a human-readable configuration file in <code>toml</code> format.</p>
<p>Example:</p>
<pre tabindex="0"><code>[[Module]]
Name = "Create-some-folder"
PriorityGroup = 3
FatalOnError = false
RunPerInstance = true
[Module.Command]
Cmd = ["mkdir", "/Users/ec2-user/my-directory"]
RunAsUser = "ec2-user"
EnvironmentVars = ["MY_VAR_FOO=myValueBar"]
</code></pre><p>I should clarify some things here.</p>
<p>Modules β a set of pre-defined modules for different purposes. It is something similar to the Ansible modules.</p>
<p>You can find the list of available modules here <a href="https://github.com/aws/ec2-macos-init/tree/master/lib/ec2macosinit">ec2-macos-init/lib/ec2macosinit</a></p>
<p>The <code>RunPerInstance</code> directive controls whether a module should run. There are three of such directives, and here is what they mean:</p>
<ul>
<li><code>RunPerBoot</code> β module will run at every system boot</li>
<li><code>RunPerInstance</code> β module will run once for the Instance. Each Instance has a unique ID; the init tool fetches it from the AWS API before the execution and keeps its execution history per Instance ID. When you create a new Instance from the AMI, it will have a unique ID, and the module will run again.</li>
<li><code>RunOnce</code> β module will run only once, despite the instance ID change</li>
</ul>
<p>I mentioned the execution history above. When EC2 macOS Init runs on the Instance first time, it creates a unique directory with the name per Instance ID to store the execution history and user data copy.</p>
<p><code>RunPerInstance</code> and <code>RunOnce</code> directives depend on the execution history, and modules with those directives will run again on the next boot if the previous execution failed. It was not obvious to me why RunOnce keeps repeating itself every boot until I dug into <a href="https://github.com/aws/ec2-macos-init/blob/master/lib/ec2macosinit/module.go#L110">the source code</a>.</p>
<p>Finally, there is a module for user data. It runs at the end by default (priority group #4) and pulls the user data script from AWS API before script execution.</p>
<p>I suggest looking into the default <a href="https://github.com/aws/ec2-macos-init/blob/master/configuration/init.toml">init.toml</a> configuration file to get yourself more familiar with the capabilities of the tool.</p>
<p>The init tool can also clear its history, which is useful for the new AMI creation.</p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>ec2-macos-init clean -all
</span></span></code></pre></div><p>And you can run the init manually for debugging purposes.</p>
<p>Example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>ec2-macos-init run
</span></span></code></pre></div><p>You can also combine the EC2 macOS Init actions (made by modules) with your script in user data for more accurate nontrivial configurations.</p>
<h1 id="wrapping-up">Wrapping up</h1>
<p>As a whole, building and operating macOS-based AMI does not differ from AMI management for other platforms.</p>
<p>There are the same principle stages: prepare, clear, build, execute deployment script (if necessary). Though, the particular implementation of each step has its nuances and constraints.</p>
<p>So the whole process may look as follows:</p>
<ul>
<li>Provision and configure needed software with Ansible playbook</li>
<li>Clean-up system logs and EC2 macOS Init history (again, with Ansible task)</li>
<li>Create the AMI</li>
<li>Add more customizations at launch with EC2 macOS Init modules and user data (that also executes your Ansible playbook or shell commands)</li>
</ul>
<p>Getting into all this was both fun and interesting. Sometimes painful, though. π</p>
<p>I sincerely hope this article was helpful to you. Thank you for reading!</p>
Terraforming mac1.metal at AWS
2021-01-20T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/01/20/terraforming-mac1.metal-at-aws/
Serhii Vasylenko
https://devdosvid.blog/2021/01/20/terraforming-mac1.metal-at-aws/cover-image.jpg
<div class="updatenotice">
Updated on the 23rd of October, 2021: Terraform AWS provider now <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ec2_host">supports</a> Dedicated Hosts natively
</div>
<p>In November 2021, AWS <a href="https://aws.amazon.com/blogs/aws/new-use-mac-instances-to-build-test-macos-ios-ipados-tvos-and-watchos-apps/">announced</a> the support for Mac mini instances.</p>
<p>I believe this is huge, even despite the number of constraints this solution has. This offering opens the door to seamless macOS CI/CD integration into existing AWS infrastructure.</p>
<p>So here is a quick-start example of creating the dedicated host and the instance altogether using Terraform.</p>
<p>I intentionally used some hardcoded values for the sake of simplicity in the example.</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_ec2_host" "example_host"</span> {
</span></span><span style="display:flex;"><span> instance_type <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"mac1.metal"</span>
</span></span><span style="display:flex;"><span> availability_zone <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"us-east-1a"</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">resource</span> <span style="color:#a5d6ff">"aws_instance" "example_instance"</span> {
</span></span><span style="display:flex;"><span> ami <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">data</span>.<span style="color:#ff7b72">aws_ami</span>.<span style="color:#ff7b72">mac1metal</span>.<span style="color:#ff7b72">id</span>
</span></span><span style="display:flex;"><span> host_id <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">aws_ec2_host</span>.<span style="color:#ff7b72">example_host</span>.<span style="color:#ff7b72">id</span>
</span></span><span style="display:flex;"><span> instance_type <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"mac1.metal"</span>
</span></span><span style="display:flex;"><span> subnet_id <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">data</span>.<span style="color:#ff7b72">aws_subnet</span>.<span style="color:#ff7b72">example_subnet</span>.<span style="color:#ff7b72">id</span>
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">data</span> <span style="color:#a5d6ff">"aws_subnet" "example_subnet"</span> {
</span></span><span style="display:flex;"><span> availability_zone <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"us-east-1a"</span>
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">filter</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"tag:Tier"</span><span style="color:#8b949e;font-style:italic"> # you should omit this filter if you don't distinguish your subnets on private and public
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span> values <span style="color:#ff7b72;font-weight:bold">=</span> [<span style="color:#a5d6ff">"private"</span>]
</span></span><span style="display:flex;"><span> }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#ff7b72">data</span> <span style="color:#a5d6ff">"aws_ami" "mac1metal"</span> {
</span></span><span style="display:flex;"><span> owners <span style="color:#ff7b72;font-weight:bold">=</span> [<span style="color:#a5d6ff">"amazon"</span>]
</span></span><span style="display:flex;"><span> most_recent <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#ff7b72">true</span>
</span></span><span style="display:flex;"><span> <span style="color:#ff7b72">filter</span> {
</span></span><span style="display:flex;"><span> name <span style="color:#ff7b72;font-weight:bold">=</span> <span style="color:#a5d6ff">"name"</span>
</span></span><span style="display:flex;"><span> values <span style="color:#ff7b72;font-weight:bold">=</span> [<span style="color:#a5d6ff">"amzn-ec2-macos-11*"</span>]<span style="color:#8b949e;font-style:italic"> # get latest BigSur AMI
</span></span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"></span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Simple as that, yes. Now, you can integrate it into your CI system and have the Mac instance with the underlying host in a bundle.
<div class="attention">
Pro tip: you can leverage the <code>aws_ec2_instance_type_offerings</code> <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ec2_instance_type_offerings">Data Source</a> and use its output with <code>aws_subnet</code> source to avoid availability zone hardcoding.
</div></p>
<p>To make the code more uniform and reusable, you can wrap it into a <a href="https://devdosvid.blog/2020/09/09/terraform-modules-explained.html">Terraform module</a> that accepts specific parameters (such as <code>instance_type</code> or <code>availability_zone</code>) as input variables.</p>
mac1.metal and mac2.metal EC2 Instances β user experience
2021-01-19T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2021/01/19/mac1.metal-and-mac2.metal-ec2-instances-user-experience/
Serhii Vasylenko
https://devdosvid.blog/2021/01/19/mac1.metal-and-mac2.metal-ec2-instances-user-experience/cover-image.jpg
<p>This is the review of EC2 Mac instances, <strong>mac1.metal</strong> and <strong>mac2.metal</strong> β the new EC2 instance types that enables macOS workloads on AWS.</p>
<div class="updatenotice">
<strong>Updated in June 2022</strong>: new information added about the offering β more cool stuff π€©
</div>
<p>AWS announced EC2 macOS instances based on the Intel CPU on 30 November 2020.</p>
<p>After a year and a half, the M1 Mac Instances arrived (7 July 2022).</p>
<p>Some basic information about the Mac EC2 first:</p>
<ul>
<li>
<p>The <strong>mac1.metal</strong> instances are Intel-based</p>
<p>12 vCPU, 32 GiB RAM | 10 Gbps Network and 8 Gbps EBS bandwidth</p>
</li>
<li>
<p>The <strong>mac2.metal</strong> instances are powered by M1 Apple Silicon processors.</p>
<p>8 vCPU, 16 GiB RAM, 16 core Apple Neural Engine | 10 Gbps Network and 8 Gbps EBS bandwidth</p>
</li>
<li>
<p>The Instance must be placed onto a <a href="https://aws.amazon.com/ec2/dedicated-hosts/">Dedicated Host</a> because these are physical Apple Mac minis.</p>
</li>
<li>
<p>AWS has integrated the <a href="https://aws.amazon.com/ec2/nitro/">Nitro System</a> to make Macs work as EC2 instances and connect them with many other services.</p>
<p>Mac minis are connected to the AWS Nitro via Thunderbolt, just a fun fact.</p>
</li>
<li>
<p>You don’t pay anything for the Instance itself, but you pay for the Dedicated Host leasing, and the minimum lease time is 24 hours.</p>
</li>
</ul>
<div class="substack-embedded-container">
<h3>Subscribe to blog updates!</h3>
<iframe title="Substack" class="substack-embedded-iframe" src="https://devdosvid.substack.com/embed" height="250"
loading="lazy"></iframe>
</div>
<h2 id="ec2-mac-instance-prices-june-2022">EC2 Mac Instance Prices (June 2022)</h2>
<p>On-demand pricing (us-east-1, North Virginia:</p>
<ul>
<li>mac1.metal costs 1.083 USD per hour or about 780 USD per month</li>
<li>mac2.metal costs 0.65 USD per hour or about 470 USD per month</li>
</ul>
<div class="attention">
The mac2.metal costs 40% less compared to the mac1.metal
</div>
<p>Since the minimal leas time for the mac*.metal dedicated host is 24 hours, the first launch of the Instance is always costly, mind that while testing.</p>
<p>One day of mac1.metal usage costs 26 USD</p>
<p>One day of mac2.metal usage costs 15.6 USD</p>
<p>To save yourself some money, you can use <a href="https://aws.amazon.com/savingsplans/compute-pricing/">Savings Plans</a>, both Instance and Compute, and save up to 44% off On-Demand pricing.</p>
<p>For example, with the one-year commitment, partial 50% upfront payment, and the Instance Savings pricing model, you can get the 20% lower price per hour:</p>
<ul>
<li>mac1.metal β 0.867 USD</li>
<li>mac2.metal β 0.52 USD</li>
</ul>
<p>Feel free to play with the numbers in the <a href="https://calculator.aws/#/createCalculator/EC2DedicatedHosts">Dedicated Host Pricing Calculator</a></p>
<h2 id="supported-operating-systems-june-2022">Supported Operating Systems (June 2022)</h2>
<ul>
<li>macOS Mojave 10.14.x (mac1.metal only)</li>
<li>macOS Catalina 10.15.x (mac1.metal only)</li>
<li>macOS Big Sur 11.x</li>
<li>macOS Monterey 12.x</li>
</ul>
<h2 id="what-can-it-do">What can it do</h2>
<p>Here is a list of some features that the mac1.metal and mac2.metal instances have:</p>
<ul>
<li>
<p>It lives in your VPC because it is an EC2 Instance, so you can access many other services.</p>
</li>
<li>
<p>For EBS, it supports the attachment of up to 16 volumes for mac1 and 10 for mac2.</p>
</li>
<li>
<p>It supports SSM Agent and Session Manager.</p>
</li>
<li>
<p>It has several AWS tools pre-installed: AWS CLI, SSM Agent, EFS Utils, and more.</p>
</li>
<li>
<p>It has pre-installed Enhanced Network Interface drivers. My test upload/download to S3 was about 300GB/s.</p>
</li>
<li>
<p>It can report CPU metrics to CloudWatch.</p>
</li>
<li>
<p>It supports <a href="https://devdosvid.blog/2021/10/24/auto-scaling-group-for-your-macos-ec2-instances-fleet/">AutoScaling</a> π</p>
</li>
<li>
<p>And you can share the instances using <a href="https://devdosvid.blog/2021/09/25/aws-resource-access-manager-multi-account-resource-governance/">AWS Resource Access Manager</a>.</p>
<p>For example, you can have a dedicated AWS account used solely as a MacOS-based farm in your organization where instances are shared with other accounts.</p>
</li>
<li>
<p>Although there is a local SSD disk available, EC2 Mac can boot only from the EBS</p>
</li>
</ul>
<div class="attention">
<p>The built-it physical SSD is still there and yours to use: build-cache, temporary storage, etc.</p>
<p>However, AWS does not manage or support the Apple hardware’s internal SSD. So there is no guarantee for data persistency.</p>
</div>
<h2 id="what-cant-it-do">What can’t it do</h2>
<ul>
<li>It can’t recognize the attached EBS if you connected it while the instance was running β you must reboot the instance to make it visible.</li>
<li>It does not recognize the live resize of EBS either β you must reboot the instance so resize change can take effect.</li>
<li>And the same relates to the Elastic Network Interfaces β attach and reboot the instance to apply it.</li>
<li>It does not support several services that rely on additional custom software, such as “EC2 Instance Connect” and “AWS Inspect.” But I think that AWS will add macOS distros for those soon.</li>
</ul>
<p>As of July 2022, mac2.metal is not supported by Host Resource Groups. Therefore you cannot use mac2.metal Instances in Auto Scaling Groups. But AWS support says they are working on that, so fingers crossed!</p>
<h2 id="launching-the-instance">Launching the Instance</h2>
<figure>
<img loading="lazy"
src="launch.png"width="400"
height="200"
/>
</figure>
<p>Jeff Bar <a href="https://aws.amazon.com/blogs/aws/new-use-mac-instances-to-build-test-macos-ios-ipados-tvos-and-watchos-apps/">published</a> an excellent how-to about kickstart of the “mac1.metal”, so I will focus on things he did not mention.</p>
<p>Once you allocated the Dedicated Host and launched an Instance, the underlying system connects the EBS with a root file system to the Mac Mini.</p>
<p>The Mac metal Instances can boot from the EBS-backed macOS AMIs only.</p>
<p>If you specified the EBS size to be more than AMI’s default, you need to resize the disk inside the system manually after the boot <sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>.</p>
<p>The time from the Instance launch until you can SSH into it varies between 5 and 20 minutes.</p>
<p>You have the option to access it over SSH with your private key. For example, if you need to set up Screen Sharing, you must allow it through the “kickstart” command-line utility and set the user password <sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup>.</p>
<h2 id="customizing-the-instance">Customizing the Instance</h2>
<p><figure>
<img loading="lazy"
src="customize.png"width="400"
height="200"
/>
</figure>
I wrote a separate post about mac1.metal AMI customization and creation, so check it out!</p>
<p><a href="https://devdosvid.blog/2021/02/01/customizing-mac1.metal-ec2-ami-new-guts-more-glory/"><strong>Customizing mac1.metal EC2 AMI β new guts, more glory</strong></a></p>
<p>Though, I would like to mention two things here:</p>
<ol>
<li>System updates are disabled by default in the macOS AMIs provided by AWS.</li>
</ol>
<p>But you can use them with no issues. For example:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>sudo softwareupdate --install --all
</span></span></code></pre></div><ol start="2">
<li>It is possible to set a custom screen resolution when connected to the instance using native ScreenSharing or any other VNC-compatible software.</li>
</ol>
<p>There are many tools, but AWS suggests the <a href="https://github.com/jakehilborn/displayplacer">displayplacer</a>.</p>
<h2 id="destroying-the-instance">Destroying the Instance</h2>
<p><figure>
<img loading="lazy"
src="cleanup.png"width="400"
height="200"
/>
</figure>
Such an easy thing to do, right? Well, it depends.</p>
<p>The complex Instance scrubbing process begins when you click on the “Terminate” item in the Instance actions menu.</p>
<p>AWS wants to ensure that anyone who uses the Host (Mac mini) after you will get your data stored neither on disks (including the physical SSD mentioned earlier) nor inside memory or NVRAM, nor anywhere else.</p>
<p>AWS does not share many details of this scrubbing process, but it takes more than an hour to complete.</p>
<p>When scrubbing is started, the Dedicated Host transitions to the Pending state.</p>
<p>Dedicated Host transitions to Available state once scrubbing is finished. But you must wait for another 10-15 minutes to be able to release it finally.</p>
<p>I don’t know why they set the Available state value earlier than the Host is available for operations, but this is how it works now (Jan'21).</p>
<p>Therefore, you can launch the next Instance on the same Host no earlier than ~1,5 hours after you terminated the previous one. That doesn’t seem very pleasant in the first couple of weeks, but you will get used to it. π</p>
<p>And again: you can release the “mac1.metal” Dedicated Host no earlier than 24 hours after it was allocated. So plan your tests wisely.</p>
<div class="attention">
If the lease time of a host is more than 24 hours, you donβt need to wait for the scrubbing process to finish to release that host.
</div>
<h2 id="legal-things">Legal things</h2>
<p>It is a bit tricky thing, but in short words:</p>
<ul>
<li>you are allowed to use the Instances solely for developer purposes</li>
<li>you must agree to all software EULAs on the system</li>
</ul>
<p>Here is the license agreement of the macOS Monterey if you want to deal with it like a pro β <a href="https://www.apple.com/legal/sla/docs/macOSMonterey.pdf">link</a>.</p>
<h2 id="some-more-cool-stuff-to-check">Some more cool stuff to check:</h2>
<p><a href="https://github.com/aws/ec2-macos-init">EC2 macOS Init</a> launch daemon, which initializes Mac instances.
<a href="https://github.com/aws/homebrew-aws">EC2 macOS Homebrew Tap</a> (Third-Party Repository) with several management tools which come pre-installed into macOS AMI from AWS.</p>
<p>Indeed it is powerful, and it has its trade-offs, such as price and some technical constraints. But it is an actual macOS device natively integrated into the AWS environment. So I guess it is worth to be tried!</p>
<p>Thanks for reading this! Stay tuned for more user experience feedback about baking custom AMIs, automated software provisioning with Ansible, and other adventures with mac1.metal!</p>
<div class="footnotes" role="doc-endnotes">
<hr>
<ol>
<li id="fn:1">
<p><strong>How to resize the EBS at mac1.metal in Terminal</strong></p>
<p>Get the identifier of EBS (look for the first one with GUID_partition_scheme):
<code>diskutil list physical external</code></p>
<p>Or here is a more advanced version to be used in a script:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#79c0ff">DISK_ID</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#ff7b72">$(</span>diskutil list physical external | grep <span style="color:#a5d6ff">'GUID_partition_scheme'</span>| tr -s <span style="color:#a5d6ff">' '</span> | cut -d<span style="color:#a5d6ff">' '</span> -f6<span style="color:#ff7b72">)</span>
</span></span></code></pre></div><p>It would probably be <code>disk0</code> if you did not attach additional EBS.</p>
<p>Then run the repair job for the disk, using its identifier:
<code>diskutil repairDisk disk0</code></p>
<p>Advanced version:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>yes | diskutil repairDisk <span style="color:#79c0ff">$DISK_ID</span>
</span></span></code></pre></div><p>Now get the APFS container identifier (look for Apple_APFS):
<code>diskutil list physical external</code></p>
<p>Advanced version:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#79c0ff">APFS_ID</span><span style="color:#ff7b72;font-weight:bold">=</span><span style="color:#ff7b72">$(</span>diskutil list physical external | grep <span style="color:#a5d6ff">'Apple_APFS'</span> | tr -s <span style="color:#a5d6ff">' '</span> | cut -d<span style="color:#a5d6ff">' '</span> -f8<span style="color:#ff7b72">)</span>
</span></span></code></pre></div><p>It would probably be <code>disk0s2</code> if you did not attach additional EBS.</p>
<p>Finally, resize the APFS container:
<code>diskutil apfs resizeContainer disk0s2</code></p>
<p>Advanced version</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>diskutil apfs resizeContainer <span style="color:#79c0ff">$APFS_ID</span>
</span></span></code></pre></div> <a href="#fnref:1" class="footnote-backref" role="doc-backlink">↩︎</a></li>
<li id="fn:2">
<p><strong>How to setup Screen Sharing at mac1.metal in Terminal</strong></p>
<p>The <code>kickstart</code> command-line tool resides in <code>/System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/</code> so you’ll better to cd into that directory for convenience:</p>
<div class="highlight"><pre tabindex="0" style="color:#e6edf3;background-color:#0d1117;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"># Turn On Remote Management for a user to be specified later</span>
</span></span><span style="display:flex;"><span>sudo ./kickstart -activate -configure -allowAccessFor -specifiedUsers
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"># Enable Remote Management for ec2-user user</span>
</span></span><span style="display:flex;"><span>sudo ./kickstart -configure -users ec2-user -access -on -privs -all
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#8b949e;font-style:italic"># Set the user password </span>
</span></span><span style="display:flex;"><span>sudo passwd ec2-user
</span></span></code></pre></div> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">↩︎</a></li>
</ol>
</div>
AWS CloudShell
2020-12-16T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2020/12/16/aws-cloudshell/
Serhii Vasylenko
https://devdosvid.blog/2020/12/16/aws-cloudshell/cover-image.png
<p>A simple but cool announcement from AWS β <a href="https://aws.amazon.com/cloudshell/">AWS CloudShell</a>.
A tool for ad-hoc AWS management via CLI directly in your browser.</p>
<p>I like when AWS releases something simple to understand and yet powerful.<br>
So it is not another <a href="https://aws.amazon.com/devops-guru/">DevOps Guru</a>, believe me :)</p>
<ul>
<li>Yes, this is similar to the shells that GCE and Azure have.</li>
<li>No, you canβt access your instances from it, so itβs not a jump server (bastion host).</li>
<li>Yes, it has AWS CLI and other tools pre-installed. Even Python and Node.js.</li>
<li>No, you canβt (well, you can, but should not) use it as an alternative to the day-to-day console on your laptop.</li>
<li>Yes, you can manage all resources from that shell as much as your IAM permissions allow you (even with SSO, which is pretty cool).</li>
<li>No, it does not support Docker.</li>
<li>Yes, you have 1 GB of permanent storage and the ability to transfer files in and out.</li>
</ul>
<h5 id="more-yes-and-nos-here">More Yes and Noβs here:</h5>
<p><a href="https://docs.aws.amazon.com/cloudshell/latest/userguide/faq-list.html">https://docs.aws.amazon.com/cloudshell/latest/userguide/faq-list.html</a></p>
<p><a href="https://aws.amazon.com/cloudshell/faqs/">https://aws.amazon.com/cloudshell/faqs/</a></p>
Terraform Workflow β Working Individually and in a Team
2020-09-16T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2020/09/16/terraform-workflow-working-individually-and-in-a-team/
Serhii Vasylenko
https://devdosvid.blog/2020/09/16/terraform-workflow-working-individually-and-in-a-team/cover-image.jpeg
<p>The work with Terraform code may become tangled sometimes. Here are some guides on how to streamline it and make it transparent for you and your team.</p>
<p>It is extremely helpful in a team, and can benefit you even if you work individually. A good workflow enables you to streamline a process, organize it, and make it less error-prone.</p>
<p>This article summaries several approaches when working with Terraform, both individually and in a team. I tried to gather the most common ones, but you might also want to develop your own.</p>
<p>The common requirement for all of them is a version control system (such as Git). This is how you ensure nothing is lost and all your code changes are properly versioned tracked.</p>
<h2 id="basic-concepts">Basic Concepts</h2>
<p>Letβs define the basic actions first.</p>
<p>All described workflows are built on top of three key steps: Write, Plan, and Apply. Nevertheless, their details and actions vary between workflows.</p>
<figure>
<img loading="lazy"
src="tf-workflow.png"
alt="Basic 3-steps Terraform workflow"width="800"
height="89.88"
/> <figcaption>
<p>Basic 3-steps Terraform workflow</p>
</figcaption>
</figure>
<p><strong>Write</strong> β this is where you make changes to the code.</p>
<p><strong>Plan</strong> β this is where you review changes and decide whether to accept them.</p>
<p><strong>Apply</strong> β this is where you accept changes and apply them against real infrastructure.</p>
<p>It’s a simple idea with a variety of possible implementations.</p>
<h2 id="core-individual-workflow">Core individual workflow</h2>
<p>This is the most simple workflow if you work alone on a relatively small TF project. This workflow suits both local and remote backends well.</p>
<figure>
<img loading="lazy"
src="tf-workflow-individual.png"
alt="Git-based Terraform workflow"width="800"
height="267.41"
/> <figcaption>
<p>Git-based Terraform workflow</p>
</figcaption>
</figure>
<h3 id="write">Write</h3>
<p>You clone the remote code repo or pull the latest changes, edit the configuration code, then run the <code>terraform validate</code> and <code>terraform fmt</code> commands to make sure your code works well.</p>
<h3 id="plan">Plan</h3>
<p>This is where you run the <code>terraform plan</code> command to make sure that your changes do what you need. This is a good time to commit your code changes changes (or you can do it in the next step).</p>
<h3 id="apply">Apply</h3>
<p>This is when you run <code>terraform apply</code> and introduce the changes to real infrastructure objects. Also, this is when you push committed changes to the remote repository.</p>
<h2 id="core-team-workflow">Core team workflow</h2>
<p>This workflow is good for when you work with configuration code in a team and want to use feature branches to manage the changes accurately.</p>
<figure>
<img loading="lazy"
src="tf-core-workflow-team.png"
alt="Git-based Terraform workflow in a team"width="800"
height="298.41"
/> <figcaption>
<p>Git-based Terraform workflow in a team</p>
</figcaption>
</figure>
<h3 id="write-1">Write</h3>
<p>Start by checking out a new branch, make your changes, and run the <code>terraform validate</code> and <code>terraform fmt</code> commands to make sure your code works well.</p>
<p>Running <code>terraform plan</code> at this step will help ensure that you’ll get what you expect.</p>
<h3 id="plan-1">Plan</h3>
<p>This is where code and plan reviews happen.</p>
<p>Add the output of the <code>terraform plan</code> command to the Pull Request with your changes. It would be a good idea to add only the changed parts of the common output, which is the part that starts with “Terraform will perform the following actions” string.</p>
<h3 id="apply-1">Apply</h3>
<p>Once the PR is reviewed and merged to the upstream branch, it is safe to finally pull the upstream branch locally and apply the configuration with <code>terraform apply</code>.</p>
<h2 id="team-workflow-with-automation">Team workflow with automation</h2>
<p>In a nutshell, this workflow allows you to introduce a kind of smoke test for your infrastructure code (using <code>plan</code>) and also to automate the feedback in the CI process.</p>
<p>The automated part of this workflow consists of a speculative plan on commit and/or Pull Request (PR ), along with adding the output of <code>plan</code> to the comment of the PR. A speculative plan mean just to show the changes, and not apply them afterward.</p>
<figure>
<img loading="lazy"
src="tf-workflow-team-automation-1.png"
alt="Git-based Terraform workflow with automation"width="800"
height="272.90"
/> <figcaption>
<p>Git-based Terraform workflow with automation</p>
</figcaption>
</figure>
<h3 id="write-2">Write</h3>
<p>This step is the same as in the previous workflow.</p>
<h3 id="plan-2">Plan</h3>
<p>This is where your CI tool does its job.</p>
<p>Letβs review this step by step:</p>
<p>1οΈβ£ You create a PR with the code changes you wish to implement.</p>
<p>2οΈβ£ The CI pipeline is triggered by an event from your code repository (such as webhook push) and it runs a speculative plan against your code.</p>
<p>3οΈβ£ The list of changes (a so-called “plan diff”) is added to PR for review by the CI.</p>
<p>4οΈβ£ Once merged, the CI pipeline runs again and you get the final plan that’s ready to be applied to the infrastructure.</p>
<h3 id="apply-2">Apply</h3>
<p>Now that you have a branch (i.e. main) with the fresh code to apply, you need to pull it locally and run <code>terraform apply</code>.</p>
<p>You can also add the automated apply here β step 5 in the picture below. This may be very useful for disposable environments such as testing, staging, development, and so on.</p>
<p>The exact CI tool to be used here is up to you: Jenkins, GitHub Actions, and Travis CI all work well.</p>
<p>An important thing to note is that the CI pipeline must be configured in a bi-directional way with your repository to get the code from it and report back with comments to PR.</p>
<p>As an option, you may consider using Terraform Cloud which has a lot of functionality, including the above mentioned repo integration, even with the free subscription.</p>
<p>If you have never worked with Terraform Cloud before and want to advice to get started, I’ll provide the links at the end of this article.</p>
<h2 id="import-workflow">Import workflow</h2>
<p>This workflow refers to a situation when you have some objects already created (i.e., up and running), and you need to manage them with Terraform.</p>
<p>Suppose we already have an S3 bucket in AWS called “someassetsbucket” and we want to include it into our configuration code.ββ</p>
<figure>
<img loading="lazy"
src="tf-workflow-import.png"
alt="Terraform resource import workflow"width="800"
height="238.20"
/> <figcaption>
<p>Terraform resource import workflow</p>
</figcaption>
</figure>
<h3 id="prepare">Prepare</h3>
<p>You should create a resource block to be used later for the real object youβre going to import.</p>
<p>You donβt need to fill the arguments in it at the start, so it may be just a blank resource block, for example:</p>
<pre tabindex="0"><code>resource "aws_s3_bucket" "someassetsbucket" {
}
</code></pre><h3 id="import">Import</h3>
<p>Now you need to import the information about the real object into your existing Terraform state file.</p>
<p>This can be done with the <code>terraform import</code> command, for example:</p>
<pre tabindex="0"><code>terraform import aws_s3_bucket.assets "someassetsbucket"
</code></pre><p>Be sure to also check the list of possible options import accepts with <code>terraform import -h</code></p>
<h3 id="write-3">Write</h3>
<p>Now you need to write the corresponding Terraform code for this bucket.</p>
<p>To avoid modifying your real object on the <code>terraform apply</code> action, you should specify all needed arguments with the exact values from the import phase.</p>
<p>You can see the details by running the <code>terraform state show</code> command, for example:</p>
<pre tabindex="0"><code>terraform state show aws_s3_bucket.assets
</code></pre><p>The output of this command will be very similar to the configuration code. But it contains both arguments and attributes of the resource, so you need to clean it up before applying it.</p>
<p>You can use one of the following tactics:</p>
<ul>
<li>either copy/paste it, and then run <code>terraform validate</code> and <code>terraform plan</code> several times to make sure there are no errors like “argument is not expected here” or “this field cannot be set”</li>
<li>or you can pick and write only the necessary arguments</li>
</ul>
<p>In any case, be sure to refer to the documentation of the resource during this process.</p>
<h3 id="plan-3">Plan</h3>
<p>The goal is to have a <code>terraform plan</code> output showing “~ update in-place” changes only.</p>
<p>However, it is not always clear whether the real object will be modified or only the state file will be updated. This is why you should understand how a real object works and know its life cycle to make sure it is safe to apply the plan.</p>
<h3 id="apply-3">Apply</h3>
<p>This is usual the <code>terraform apply</code> action.</p>
<p>Once applied, your configuration and state file will correspond to the real object configuration.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Here is an overview of Terraform Cloud for those who never worked with it before: <a href="https://www.terraform.io/docs/cloud/overview.html">ββOverview of Terraform Cloud Features</a></p>
<p>And here is a nice tutorial to start with: <a href="https://learn.hashicorp.com/collections/terraform/cloud-get-started">Get Started - Terraform Cloud</a></p>
<p>Also, here is an overview of workflows at scale from the HashiCorp CTO which might be useful for more experienced Terraform users: <a href="https://www.hashicorp.com/resources/terraform-workflow-best-practices-at-scale">Terraform Workflow Best Practices at Scale</a></p>
<p>Thank you for reading. I hope you will try one of these workflows, or develop your own!</p>
Terraform Certification Tips
2020-09-15T00:00:00Z
2024-03-25T17:29:29Z
https://devdosvid.blog/2020/09/15/terraform-certification-tips/
Serhii Vasylenko
https://devdosvid.blog/2020/09/15/terraform-certification-tips/cover-image.png
<p>I successfully passed the “HashiCorp Certified β Terraform Associate” exam last Friday and decided to share some advice for exam preparation.</p>
<h2 id="make-yourself-a-plan">Make yourself a plan</h2>
<p>Make a list of things you are going to go through: links to the study materials, practice tasks, some labs, some articles on relative blogs (Medium, Dev.to, etc.).
It should look at a “todo” or “check”-list. It may seem silly at first glance, but the list with checkboxes does its “cognitive magic”. When you go point by point, marking items as “done”, you feel the progress and this motivates you to keep going further.
For example, you can make a plan from the resources I outlined below in this article.</p>
<p>I encourage you to explore the Internet for something by yourself as well. Who knows, perhaps you will find some learning course that fits you better. And that is great! However, when you find it, take extra 5-10 minutes to go through its curriculum and create a list with lessons.</p>
<p>It feels so nice to cross out items off the todo list, believe me π
<figure>
<img loading="lazy"
src="todo-list.jpg"width="585"
height="540"
/>
</figure>
</p>
<h2 id="go-through-the-official-study-guide">Go through the official Study Guide</h2>
<p>Despite your findings on the Internet, I strongly suggest going through the official study guide</p>
<p><a href="https://learn.hashicorp.com/tutorials/terraform/associate-study">Study Guide - Terraform Associate Certification</a></p>
<p>It took me about 20 hours to complete it (including practice tasks based on topics in the guide), and it was the core of my studying. I did not buy or search for some third-party course intentionally because I did have some Terraform experience before starting the preparation. But give the official guide a chance even if you found some course. It is well-made and matches real exam questions very precisely.</p>
<p>Also, there is an official <a href="https://learn.hashicorp.com/tutorials/terraform/associate-review">Exam Review</a>. Someone might find this even better because it is a direct mapping of each exam objective to HashiCorp’s documentation and training.</p>
<h2 id="take-additional-tutorials">Take additional tutorials</h2>
<p>Here is a list of additional tutorials and materials I suggest adding into your learning program:</p>
<h4 id="official-guides--documentation">Official guides / documentation:</h4>
<ul>
<li><a href="https://learn.hashicorp.com/collections/terraform/automation">Automate Terraform</a></li>
<li><a href="https://learn.hashicorp.com/collections/terraform/cloud">Collaborate using Terraform Cloud</a></li>
<li><a href="https://learn.hashicorp.com/collections/terraform/0-13">Terraform tutorials</a></li>
<li><a href="https://learn.hashicorp.com/collections/terraform/modules">Reuse Configuration with Modules</a></li>
<li><a href="https://www.hashicorp.com/resources/a-practitioner-s-guide-to-using-hashicorp-terraform-cloud-with-github">A Practitionerβs Guide to Using HashiCorp Terraform Cloud with GitHub</a></li>
<li><a href="https://learn.hashicorp.com/collections/terraform/policy">Enforce Policy with Sentinel</a></li>
</ul>
<h4 id="third-party-articles-and-guides">Third-party articles and guides:</h4>
<ul>
<li><a href="https://prefetch.net/blog/2020/04/27/using-the-terraform-console-to-debug-interpolation-syntax/">Using the terraform console to debug interpolation syntax</a></li>
<li><a href="https://www.youtube.com/playlist?list=PL5VXZTK6spA2HF5Kf0rI9RDRHF9Hopffr">YouTube playlist with exam-like questions review</a></li>
</ul>
<h2 id="find-yourself-some-practice">Find yourself some practice</h2>
<h4 id="mockup-a-project">Mockup a project</h4>
<p>You can greatly improve your practice by mocking some real business cases.</p>
<p>If you already work in some company you can set up the project you’re working with using Terraform. If you donβt have a real project or afraid to accidentally violate NDA, try this open-source demo project: <a href="https://github.com/gothinkster/realworld">Real World Example Apps</a>.</p>
<p>It is a collection of different codebases for front-end and back-end used to build the same project. Just find the combination that suits your experience better and try to build the infrastructure for it using Terraform.</p>
<figure>
<img loading="lazy"
src="real-world-demo.jpg"width="585"
height="405"
/>
</figure>
<h4 id="answer-forum-topics">Answer forum topics</h4>
<p>Last but not least advice β try to answer some questions on the official <a href="https://discuss.hashicorp.com/c/terraform-core/">Terraform forum</a>.</p>
<p>This is a nice way to test your knowledge, help others, and develop the community around Terraform. Just register there, look for the latest topics, and have fun!</p>
<figure>
<img loading="lazy"
src="tf-forum.jpg"width="585"
height="650"
/>
</figure>
<p>π I sincerely wish you exciting preparation and a successful exam! π</p>