Infra As Code: CloudFormation

About CloudFormation

CloudFormation (CF) is IAC, or Infrastructure as Code, and you use it to convert your TEMPLATE to STACK. Even though while building Unicorn Pursuit we’ll be exclusivly using AWS CDK, it’s very important to understand CloudFormation Templates and different Sections, because AWS CDK is only there to synthetize the Template… we sometimes need to manualy check if Template includes everything we want to provision. Writting Python with AWS CDK Library instead of manually writting CloudFormation Templates is just a huge step forward, but at least keep this post at hand, for when you need to “translate” what your Template is trying to tell you.

We can use two different languages to create CF Templates:

• JSON
• YAML

Since IAC is CODE, we can treat it as Code, meaning - store it in repositories, version it, etc.

JSON

JSON is a Java Script Object Notation, so - based on Java Script notation. YAML is great, but JSON is still around, as many people have JS background, so it’s still good to know both.

There are NO IN LINE COMMENTS in JSON, so if you convert from YAML - you lose these.

Kay JSON components are:

• Key/Value pair, that contains data. Example: “Description”: “WebServer EC2 instance type”
• Commas - to separate Data.
• Objects, that are enclosed with {}. “KeyName”: { … }
• Square brackets: hold Arrays, for example:

"AllowedValues": [
 "t1.micro",
 …
]

YAML

YAML is designed to be readable by humans, to the Templates seem to be a bit smaller and “cleaner”, which is why we will be using YAML.

Key YAML components are:

• Key/Value pair, separated with “:", so it would be like year : 2001 or name : example.
• Arrays can have two formats, “- Item”, or [item, item, item]
• Objects are just a separation you choose to use, there is no syntax for it.

A YAML example for an employee object could look like this:

# Employee records
-  martin:
    name: Martin D'vloper
    job: Developer
    skills:
      - python
      - perl
      - pascal

YAML can be validated using YAML Validator

IMPORTANT: Don’t use TABS, use Spaces, cause spaces are important in YAML.

IAM

IAM is the most complex part of CloudFormation, as most people get confused, and either give too much permissions, or don’t give enough for CF to be able to deploy all the required resources. How do we “solve” the IAM in CF? The answer is - Service Role.

The Stacks are creating multiple Resources. What you’re granting CF permissions, you need to consider both:

• Permission to Create/Modify the CF Stack: by attaching a Policy to a Role or a Group, applying to a specific user
• Permission to create individual Resources: our CF will be creating Resources, which is why you need a Service Role which allows CloudFormation to manage your AWS Resources for you. If you go to IAM -> Roles, under the “Trusted entities” column, you will see a bunch of “AWS service:". These are Service Roles, and the AWS Services listed, are the ones the roles are configured to manage.

For example, in IAM, the AdministratorAccess Policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

CloudFormation read only would look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudformation:Describe*",
                "cloudformation:EstimateTemplateCost",
                "cloudformation:Get*",
                "cloudformation:List*",
                "cloudformation:ValidateTemplate",
                "cloudformation:DetectStackDrift",
                "cloudformation:DetectStackResourceDrift"
            ],
            "Resource": "*"
        }
    ]
}

The idea here is to always follow the rule of least privilege, and therefore create a specific Policy we need to assign to our users, combining:

• Actions
• Effects (Allow or Deny)
• Resources (ARN), which allow us to give a specific permission for a specific, already provisioned, Resource.

Templates

The first step is to have IAC in a CloudFormation Template. The Template describes all the RESOURCES that we want to create. This template is typically stored in a S3 Bucket (sure, you can store it locally… but it’s just easier to have it in a S3).

The first MOST important part about learning CloudFormation is understanding the Template Sections, and how each is section is used.

Template Sections

Description

In YAML you can use in-line comments, so this section isn’t as important as in JSON.

Metadata Provides inclusion of arbitrary objects that provide details about the template. For example, we can tell CloudFormation to install some software, like mysql, python etc. in the EC2 instance.

Parameters

Parameters allow you to pass desired values into your template when you create a Stack. Example: VPC, Subnets, Instance Type etc.

Parameters :
 Environment :
  Description : "environment"
  Default : "dev"
  AllowedValues : "dev" , "qa" , "pro"

You can use Parameters to create Conditions (if Dev - Instance type = XXX)

Transforms

There are 2 types of transforms:

• AWS::Serverless, referring to Lambda functions
• AWS::Include, for Code Reuse from other templates (stuff you use often)

Resources

Resources section is the ONLY required section and it’s where you define all the AWS Resources you want to create or modify. For example, to create a S3 bucker, you would define S3::Bucket as the resource Type, with properties Public Read for example:

 S3Bucket:
  Type: AWS::S3::Bucket
  Properties:
   AccessControl: PublicRead
   BucketName: public-bucket

Outputs

Stuff you need to Output, and maybe Import in another Stack.

Intrinsic Functions

These are built-in functions that let us dynamically assign value. For example, before assigning the public IP, you don’t know what the IP will be, but you need to refer to it somehow.

Functions that are used a lot are:

• Ref: returns the value of a specified Parameter or Resource (name or ID), example (YAML): !Red: VPC
• GetAtt
• Fn::Join [ delimiter , string ]
• Fn::Select [ order , array ]
• Fn::Split [ delimiter , source string]
• Fn::Transform

Pseudo Parameters are Global Values in AWS, and you can refer to them (Ref statement) anywhere in the template, just using the AWS::Region, or AWS::AccountId.

Condition Functions

Condition functions are kind of Intrinsic Functions, and can be very useful when you’re, for example, using the same CF to create Stacks in different environments, with different types of resources.

Condition Function evaluates the Input Parameter (in the Parameters section of the Stack), and based on the input - it defines the Variables. This is done using the “if” function, which would basically work like this:

• Fn::If [Condition, Option if YES, Option if ELSE]

Another use of Conditions is without “if”, meaning - in the Resources section, you would define a Resource, and if the Condition is met - Resource is created… otherwise - it´s not!

Templates and Designer

Designer is a GUI tool that can help you in the beginning, as it visualizes and validates your template.

Before you actually start designing Templates, it’s highly recommended that you Bookmark, and get to know the structure of the Resource and Property page in AWS:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html

When you’re in the Designer, and you click on any object, the “?” icon will actually take you to the corresponding part of this page.

Cloud Former

Beta tool for creating CloudFormation Templates from AWS Resources. You would go to the new stack, and choose the CloudFormer from the Sample templates, as shown below, and launch it:

This creates a EC2 instance of a Cloud Former Web Server (don’t worry about the cost, its actually a T2small by default). You can then https into it, and choose the resources from your account, and turn into CFT-s.

Template Best Practices

• Reuse templates to replicate Stacks. We can use “Includes”, Cross-Stack references, and Stack Nesting.
• Do not embed credentials! Or any kind of sensitive information. Use PARAMETERS. Also, include “NoEcho” to TRUE, to protect the credentials.
• Validate Templates before using them.
• Templates are CODE so use Code Reviews and Revision Control.

CloudFormation Stacks

When you create resources in a Stack, you should always use a Stack to manage those resources (update, delete etc.). When you launch a Stack, have in mind that if there´s a ROLLBACK and you get the “ROLLBACK_FAILED”, you might have resources that are still running and cannot be deleted.

Deleted Stacks are retained for 90 days.

To PROTECT the Stack, from accidental DELETION (or UPDATE) we can apply:

• Termination Protection, within Stack Creation Options (disabled by default). For Nested Stacks, Termination Protection propagates down. We need to make sure to not allow anyone to remove the Protection using the IAM policy. Termination Protection protects the STACK, not the resources, we need to also protect the Resources.
• Stack Policies are used for protecting of some or all Resources within the Stack. We can define only one policy per stack, but we can protect multiple Resources.
• Resource Level Policies, that go per resource, and you can define Deletion Policy to be Retain on an individual Resource level.
• IAM Policy: Within the Template you can attach Policies (IAM roles) to Groups or Resources.

Rollback Triggers

We can tie our Stack to the Alarms, so that CF rolls back the entire Stack if our trigger goes to the ALARM state. You can have max 5 rollback triggers.

Monitoring the Stack

We can use AWS Config to monitor the Stacks, and even use SNS to get notifications about the change. This way we can make sure that out Stack is in compliance. We need to turn Recording ON within the Config service, and this way the Resources in the Region will be recorded (for 7 years!!!).

Bootstrapping

• Repeatable process of getting app running on an EC2 instance (not everything can and should be done in AMI)
• Examples: Update files, Get latest software version, register with ELB etc.
• In EC2 you can do all this under the “Advanced Data” within the Instance Details, but in CloudFormation - we would use “cloud-init” and METADATA. This way, CloudFormation would wait for helper scripts “success”, to continue with the Stack.

To correctly configure the Bootstrapping, you need to be familiar with the following terms.

Wait Condition

Often used with Hybrid Environments, when you’re waiting for something on-prem to continue creating the stack.

Wait Condition stays in “create in progress”, waiting for success during the timeout you define. You need to configure:

• Wait condition, which defines the Timeout and Resource you’re waiting on.
• Wait handler is required, as a signaling mechanism for Condition Success/Failure using JSON.

Creation Policies

Creation policy is an ATTRIBUTE of a RESOURCE, unlike a Wait Condition, which is a Resource.

Similar to Wait Condition, they pause the creation of a resource until condition is met. The condition can be a timeout, like in the wait condition, or it can be something else, like for example NGINX installed in EC2 or similar. The idea is to wait for a number of Success signals to be received, before the Stack continues.

Update Policies

Update policy is an ATTRIBUTE of a RESOURCE, like a Condition Policy, and unlike a Wait Condition, which is a Resource.

Helper Scripts

It’s a Python code to install stuff on your EC2 basically. They are called directly from the Template, and they use the METADATA defined in the Template.

There are 4 helper scripts, and you need to include calls to execute them:

• Cfn-init: to install packages, create files and start services.
• Cfn-signal: designed to retrieve error codes (analyzes the feedback from cfn-init to CloudFormation).
• Cfn-get-metadata: a wrapper for out METADATA within the Template. It’s mostly used for Troubleshooting.
• Cfn-hup, which is actually a Daemon, that checks for updates to metadata and executes stuff when changes are detected.

Systems Manager (SSM) Parameter Store

Systems Manager Parameter Store can be used to store confidential information. You can store it in plain text, or encrypt using KMS. The idea is for CF to go to Parameter Store to retrieve the Template Parameters in order to create a Stack.

Available SSM parameters are:

• AWS::SSM::Parameter::Name
• AWS::SSM::Parameter::Value<String>
• AWS::SSM::Parameter::Value<List<String>
• AWS::SSM::Parameter::Value<Any AWS Type>

Aside from SSM, CF supports two more dynamic reference patterns: SSM-SECURE and SECRETSMANAGER for “secrets” stored in AWS Secrets Manager.

For example, within the EC2 “Instance Type” parameter, we would use SSM to define that devInstance equals whichever instance type we want, and then in the YAML parameter section:

InstanceType:
 Type: 'AWS::SSM::Parameter::Value<String>'
 Default: devInstances

You can also use referencing from other section, for example Resources, and you would also reference SSM parameters… But it would be something like:

Resources:
 S3bucket:
 Type: 'AWS::S3::Bucket'
 Properties:
  AccessControl: '{resolve:ssm:S3AccessControl:1}'

In this example above, SSM parameter was configured with the name S3AccessControl, and the value is “PublicRead”, meaning - we’re creating an S3 bucket with public read access.

Secrets Manager

We don’t want to be hard-coding Passwords, we want to create reusable code, as much as possible. Have in mind that the price is 40 cents a month per secret. Not a lot, but… there’s a cost, not like SSM where parameters are for free.

The big advantage is that from Secrets Manager we can automatically rotate our keys, and generate random passwords, AND you can share secrets across multiple accounts.

First, from CloudFormation you would create a secret in the Secrets Manager:

Resources:
 MyRotationSecret:
  Type: 'AWS::SecretsManager::Secret'
  Properties:
   GeneratedSecretString:
    SecretStringTemplate: {"username":"password"}
    # this next line generates random password and saves it to a variable
    GenerateStringKey: 'password'
    PasswordLenght: 16

And now, to “resolve” the secret when, for example, creating an RDS database using CF, the resolution of the username and password for your database, you would something like this:

- 'resolve:secretsmanager:RDSSecret:SecretString:username'
- 'resolve:secretsmanager:RDSSecret:SecretString:password'

CloudFormation Macros

Macros are powered by Lambda Functions.

There are 2 types of Macros:

• Template Level Macros, where “Transform” is on the same level as Resources. This Macro has access to an entire Template.
• Snippet Level Macros, where “Transform” is a function, as in “Fn::Transform”, as one of the Properties of a Resource, so it applies ONLY to that particular Resource.

Custom Resources

CloudFormation does NOT support every Resource in AWS. The NEW Resources are sometimes NOT SUPPORTED. You can use (and it is highly recommended to use) Custom Resources, but it requires quite a bit of manual work.

You will find some practical examples here.

To use Lambda function, our CF will require 3 elements:

• Custom Resource (Custom::OurString), with a must-have property - Service Token.
• Lambda Execution Role.
• Lambda Function (inline, meaning - in our code, or a zip in s3 if it’s a huge lambda):
  • Handler
  • Runtime (language)
  • Role
  • Timeout (how long it can run)

Stack Sets

Large companies have multiple AWS Accounts, and multiple Regions, and the Ops team needs to ease the operations within this multi-account environment.

Stack Sets are used to standardize deployments across various Regions and Accounts. Stack Sets gives us a possibility to deploy templates from Admin Account, into Child Accounts. There are 2 types of AWS accounts:

• Admin Account, where you use one template to deploy in multiple accounts
• Target Accounts, or Child Accounts, where you get everything deployed from Admin Account.

IAM is the complex part here, and as you probably guessed - it all works with Service roles, in both Admin and Target accounts. Target accounts need to have permissions to create the resources defined in the Template in the Admin account.

When defining the Service Role in the Target accounts, for Admin Account Template to be able to assume the role, you need to make sure the “trusted account” is correctly configured as your Admin account.

CloudFormation training and resources

I highly recommend you to do a Linux Academy CloudFormation Deep Dive. Linux Academy has a “1 week for free offer”, you can register and do this course in 1 week. I’ve been a annual-plan Linux Academy user for years, and it’s probably my favorite learning platform.

Bookmark CloudFormation Resource Type Reference

YAML can be validated using YAML Validator

Feedback

Was this page helpful?

Awesome! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.