Automating AWS with CloudFormation

INTRODUCTION

The Basics

By scmGalaxy.com

About Me

DevOps@RajeshKumar.XYZ

“Amazon Web Servicesis a platform of web services offering solutions for computing, storing, and networking, at different layers of abstraction.”

Amazon Web Services in Action, Manning 2015

• Manage Infrastructure Manually
  • Good way to start
  • Perfect for Proof-of-Concept and Prototyping
• Automate Infrastructure
  • Improve quality and efficiency
  • Major benefit of using AWS

Background Knowledge

AWS CloudFormation

• Automate almost every part of AWS with CloudFormation
• Learn how to automate:
  • EC2 Instance
  • VPC
  • Security Group
  • RDS

• Use CloudFormation to create stacks
  • Managment Console
  • Command Line Interface
• Create CloudFormation templates

Let’s start!

Exploring AWS CloudFormation

---

Overview

• Automating infrastructure
  • Lower costs
  • Improve quality
  • Improve flexibility
• Describing infrastructure in code
  • No scripting or programming needed
  • Perfect tool for the job
• AWS CloudFormation
  • Describing AWS infrastructure in code
  • Highly integrated with AWS

Automation vs. Manual Work

---

Managing AWS Manually

• Adam, DevOps engineer at Globomantics
• Frustrated with current situation
• Goals
  • Lower costs
  • Improve quality
  • Improve flexibility

Advantages of Automating Infrastructure

What is the biggest advantage of Amazon Web Services?

Every part of AWS is controllable via an API

AWS API Enables Automation

---

Controlling AWS with an API

API Usage Examples

• Launch EC2 instance
• Create Security Group
• Update RDS database configuration
• Create networking infrastructure
• Add new IAM user
• ....

Tools to Access AWS API

• Resolve dependencies
  • RDS database needs to be created before EC2 instances are created.
  • Security Groups needs to be created before EC2 instance is created.

• Update infrastructure
  • Which steps are necessary to update existing infrastructure?
  • Hard coded transformation not very flexible.

Describing Infrastructure in Code

---

Describing Infrastructure in Code

• Describing infrastructure in code
  • Resolves dependencies automatically
  • Creates, updates or deletes infrastructure
  • Performs idempotent and predictable changes to infrastructure

AWS CloudFormation

AWS CloudFormation

---

Features of CloudFormation

	"Type": "AWS::EC2::Instance",
	"Properties": {
	   "ImageId": "ami-bff32ccc",
	   "InstanceType": "t2.nano”
	}              

Template

Written in JSON describing the target state of the infrastructure.

Stack, Instance of a Template

1 Template

=

Multiple Stacks

• Simple WordPress environment
  • EC2 instance running Web Server
  • RDS instance with MySQL engine
  • Security Groups
• Have a look at template defining target state
• Create stack based on template

• Text Editor
• IDE Integration
  • Eclipse
  • Visual Studio
• AWS CloudFormation Designer
  • Graphical tool for creating templates
  • Part of AWS Management Console

Summary

• Automating infrastructure
  • Lower costs
  • Improve quality
  • Improve flexibility
• Describing infrastructure in code
  • No scripting or programming needed
  • Perfect tool for the job
• AWS CloudFormation
  • Describing AWS infrastructure in code
  • Highly integrated with AWS

Creating a CloudFormation Template

---

Overview

• Creating a template from scratch
• Anatomy of a template
• Define target state of your infrastructure
• Explore resource types and their properties
• Describe dependencies between resources

• Adam, DevOps engineer at Globomantics
• Starting with a manageable scope
• Creating his first CloudFormation template

SSH Bastion Host

Needed Parts for Bastion Host

Anatomy of a CloudFormation Template

---

Minimal Template

	{
	 "AWSTemplateFormatVersion": "2010-09-09",
	 "Description": "SSH Bastion Host",
	 "Resources": {
	    "EC2Instance": {...},
	    "ElasticIp": {...},
	    "SecurityGroup": {...} 
	 }
	}
            

	{
	  "AWSTemplateFormatVersion": "2010-09-09",
	  "Description": "SSH Bastion Host",
	}
            

Template Format Version and Description

Version of the template anatomy: only valid value: 2010-09-09

A description helps you to organize and document your template

	"Resources": {
		"EC2Instance": {...},
		"ElasticIp": {...},
		"SecurityGroup": {...}
	}
            

Resources Section

Includes the description of your infrastructure

The target state of all involved resources is described here

Working with Resources

---

Resource Types

• AWS::EC2::Instance
• AWS::EC2::SecurityGroup
• AWS::EC2::EIP
• AWS::RDS::DBInstance
• AWS::IAM::Role
• AWS::EC2::VPC
• AWS::ElastiCache::CacheCluster
• AWS::DynamoDB::Table
• ...

Resource Definition

 {
	...
	"Resources": {
		"EC2Instance": {
		   "Type": "AWS::EC2::Instance",
		   "Properties": {...}
		}
	}
 }
            

	"Type": "AWS::EC2::Instance",
	"Properties": {
		"ImageId": "ami-bff32ccc",
		"InstanceType": "t2.nano"
	}
            

EC2 Instance

Depending on the Resource Type, there are different properties available

It is possible to specify AMI and Instance Type for an EC2 instance

EC2 Instance

 "EC2Instance": {
	"Type": "AWS::EC2::Instance",
	"Properties": {
		"ImageId": "ami-bff32ccc",
		"InstanceType": "t2.nano",
		"NetworkInterfaces": [{"SubnetId": “..."}]
    }
 }

               

	"Type": "AWS::EC2::EIP",
	"Properties": {
	    "InstanceId": "i-d0f2e055",
		"Domain": "vpc"
	}}
            

Elastic IP

Properties of resource of type AWS::EC2::EIP

An Elastic IP address needs to be linked to an EC2 instance

Domain vpc as we are operating in a VPC

	"Type": "AWS::EC2::SecurityGroup",
	"Properties": {
		"GroupDescription": "ssh-bastion-host",
		"VpcId": "vpc-6b53320e",
		"SecurityGroupIngress": [{...}]
	}
            

Security Group

Resource describing a Security Group

Includes link to VPC and nested ingress rules

	"SecurityGroupIngress": [{
		"CidrIp": "0.0.0.0/0",
		"FromPort": 22,
		"IpProtocol": "tcp",
		"ToPort": 22
	}]
            

Security Group Ingress

Nested ingress rules of Security Group

Allowing incoming TCP traffic from everywhere on port 22

Dependencies Between Resources

---

• CloudFormation is able to resolve dependencies between resources automatically
• But how do you define dependencies within the template

Dependencies Between Resources

Referencing Security Group

 "EC2Instance": {
	"Properties": {
		"NetworkInterfaces": [{
		  "GroupSet": [{"Ref": "SecurityGroup"}]
		}]
	}
 },
 "SecurityGroup": {...}	
            

Referencing EC2 Instance

 "EC2Instance": {...},
 "ElasticIP": {
    "Type": "AWS::EC2::EIP",
    "Properties": {
	   "InstanceId": {"Ref": "EC2Instance"},
	   "Domain": "vpc"
   }
 }
            

• Walk-through template
  • EC2 Instance
  • Security Group
  • Elastic IP
  • References between Resources
• Create stack out of template by using Command Line Interface (CLI)

Install and Configure AWS Command Line Interface (CLI)

https://aws.amazon.com/cli/

Summary

• Minimal CloudFormation template
  • Format Version
  • Description
  • Resources Section
• Target state is defined within resources section
• Describing resources
  • Name
  • Resource Type
  • Parameters

Using Parameters and Outputs to Customizea Stack

---

Overview

• Reuse templates for multiple slightly different stacks
• Add parameters to a template
• Integrate stacks into existing environments
• Use outputs to access results from a running stack

• Adam, DevOps engineer at Globomantics
• Already done
  • CloudFormationtemplate including a SSH bastion host environment
• To do
  • Customize stack for slightly different use cases

Input and Output

Parameters to Specify Input

---

• Adam wants to reuse his template for multiple SSH bastion host setups
• Small changes are needed depending for different setups
  • Key Pair
  • Instance Type

Reuse Template

Parameters Section

  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	"Parameters": {...},
	"Resources": {...}
  }
				  
			      

Defining a Parameter

  "Parameters": {
     "KeyPair": {
	  "Description": "A SSH key pair.",
	  "Type": "AWS::EC2::KeyPair::KeyName"
     }
  }
				  
			      

  "InstanceType": {
	"Description": “The instance type.",
	"Type": "String"
  }
				  
			      

Description

Describe your parameters to simplify filling them out later

Defining a Parameter

  "Parameters": {
     "KeyPair": {
	  "Description": "A SSH key pair.",
	  "Type": "AWS::EC2::KeyPair::KeyName"
	 }
  }
				  
			      

   String: "string"
   Number: 1
   List<Number>: [1,2,3]
   CommaDelimitedList: "1,2,3"
				  
			      

Parameter Type

General parameter types

	AWS::EC2::Image::Id: "ami-b43503a4"
	AWS::EC2::Instance::Id: "i-47cad982"
	AWS::EC2::SecurityGroup::Id: "sg-6246b70c"
	AWS::Route53::HostedZone::Id: "Z2VHZWU704GYI1"
				  
			      

Parameter Type

AWS parameter types

Referencing Parameters

 "Parameters": {
   "KeyPair": {...}
 },
 "Resources": {
   "EC2Instance": {
      "Properties": {
         "KeyName": {"Ref": "KeyPair"}
       }
   }}
				  
			      

	AllowedValues: ["a", "b", "c"]
	AllowedPattern: "[a-z0-9]*"
	MaxLength/ MinLength: 1
	MaxValue/ MinValue: 2
				  
			      

Input Validation

Software development 101 : validate your inputs

Validating Instance Type

 "InstanceType": {
	"Description": "The instance type.",
	"Type": "String",
	"AllowedValues": ["t2.nano", "t2.micro", "t2.small"]
  }
				  
			      

Default Value

 "InstanceType": {
	"Description": "The instance type.",
	"Type": "String",
	"Default": "t2.nano"
	}
				  
			      

• Walk-through template
  • Input Parameters
  • Validation
• Create stack out of template by using Command Line Interface (CLI)
• Specify input parameters for stack

Outputs to Access Results from Stack

---

• Adam wants to integrate SSH bastion server stacks into existing environment
• He needs to access the results from a stack to be able to
  • Update DNS record with a Public IP
  • Add a Private IP to firewall configuration

Access Results of Stack

Outputs Section

  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	"Parameters": {...},
	"Resources": {...},
	"Outputs": {...}
  }
				  
			      

Defining an Output

  "Outputs": {
	"SSHBastionHostPublicIp": {
	  "Description": "Public IP address...",
	  "Value": {"Ref": "ElasticIP"}
	}
  }
				  
			      

	"SSHBastionHostPublicIp": {
	  "Description": "Public IP address...",
	  "Value": {"Ref": "ElasticIP"}
	}
				  
			      

Default Resource Return Value

Accessible via built -in Ref function

Return Value depends on the Resource Type

  "SSHBastionHostPrivateIp": {
	"Description": "Private IP address...",
	"Value": {"Fn::GetAtt": ["EC2Instance", "PrivateIp"]}
  }
				  
			      

Specific Resource Return Value

Accessible via built-in Fn::GetAtt function

Different return values depending on the Resource Type

Returning Public IP

    "Resources": {
	  "ElasticIP": {...}
	}
	"Outputs": {
	  "SSHBastionHostPublicIp": {
	    "Description": "Public IP address...",
	    "Value": {"Ref": "ElasticIP"}
	  }
	}
				  
			      

Returning Private IP

    "Resources": {
      “EC2": {...}
   }
   "Outputs": {
     "SSHBastionHostPrivateIp": {
	    "Description": "Private IP address...",
	    "Value": {"Fn::GetAtt": ["EC2", "PrivateIp"]}
     }
    }
				  
			      

• Walk-through template
  • Defining Outputs
  • Accessing return values from resources
• Get outputs from a running stack via CLI

Summary

• Inputs
  • Allows you to reuse a template in slightly differnt use cases
  • Template defines possible Parameters
  • Paramters have to be specified when creating/updating a stack

• Outputs
  • Allows you to access results from a stack
  • Template defines outputs
  • Outputs can be accessed after stack was created/updated successfully

Building More Powerful Templates with Built in Helpers

---

Overview

• Creating multi-region template
  • Mappings
  • Fn::FindInMap
• Using Pseudo Parameters
  • AWS::AccountId, AWS::Region, …
  • Fn::GetAZs
• Provisioning an EC2 Instance
  • User Data
  • Fn::Join and Fn::Base64

• Common problems when creating a CloudFormationtemplate …
  • Multi-region support
  • Install and configure software on EC2 instances automatically
  • Reduce number of parameters
• But how to do that in JSON?
  • CloudFormation offers built in helpers for common problems

• Adam, DevOps engineer at Globomantics
• Create CloudFormation template
  • Networking configuration
  • EC2 Instance running web server
  • Latest Amazon Linux AMI
• Create stacks in the following regions
  • us-east-1
  • eu-west-1
  • ap-northeast-1

Independent AMI per Region

Introducing Mappings

---

“A map is an object that maps keys to values.”

What is a map?

Mappings Section

  {
	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "SSH Bastion Host",
	“Mappings": {...},
	"Resources": {...}
  }
				  
			      

Defining a Mapping

  "Mappings" : {
	"Mapping" : {
	   "Key" : {
		"Name" : "Value"
	   }
	}
   }	  
			      

    "Mapping" : {
	  "KeyA" : {},
	  "KeyB" : {}
	}	  
			      

Unique Key

A key has to be unique within a mapping

    "Key" : {
	   "NameA" : "ValueA",
	   "NameB" : "ValueB",
	}	  
			      

Named Values

A key can contain multiple named values

A name has to be unique within a key

Map AMIs per Region

 "RegionAMI": {
	"us-east-1": {
	   "AmazonLinux": "ami-8fcee4e5",
	   "Ubuntu": "ami-fce3c696"
	},
	"eu-west-1": {
	   "AmazonLinux": "ami-e1398992",
	   "Ubuntu": "ami-f95ef58a"
 }}	  
			      

   "Fn::FindInMap": ["RegionAMI", "eu-west-1", "AmazonLinux"]  
			      

Fn:: FindInMap

Allows to access values from a Mapping

Accepts Mapping name, Key, and Name as parameters

• CloudFormation template
  • Networking configuration
  • EC2 Instance
  • Security Group
• Mapping contains AMI IDs for multiple regions
• Create stack in multiple regions with the help of AWS CLI

Using Pseudo Parameters

---

• Adam, DevOps engineer at Globomantics
• CloudFormation template reusable for multiple regions
• How to define the Availability Zone for a Subnet?

Define AZ for Multi-Region Template

 "Subnet": {
	"Type": "AWS::EC2::Subnet",
	"Properties": {
	"AvailabilityZone": "???",
	"CidrBlock": "10.0.0.0/24",
	"VpcId": {"Ref": "VPC"},
	"MapPublicIpOnLaunch": "true"
 }}
			      

Pseudo Parameters

	AWS::AccountId
	AWS::NotificationARNs
	AWS::NoValue
	AWS::Region
	AWS::StackId
	AWS::StackName
			      

   {"Ref": "AWS::Region"} 
			      

Accessing a Pseudo Parameter

Use built in Ref function to access pseudo parameters

   {"Fn::GetAZs": {"Ref": "AWS::Region"}}
			      

Fn::GetAZs

Returns all Availability Zones for a region

Uses current region of stack in this example

   {"Fn::Select": ["0", ["a", "b", "c"]}
			      

Fn::Select

Selects a value from a list

Selects first value of list in this example

Define AZ for Multi-region Template

	"Type": "AWS::EC2::Subnet",
	"Properties": {
	  "AvailabilityZone": {
		"Fn::Select": [
		  "0",
		  {"Fn::GetAZs": {"Ref": "AWS::Region"}}
		]
	  }, ...
	}
			      

Template Reference

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-reference.html

• CloudFormationTemplate
  • Pseudo Parameters
  • Fn::GetAZs
  • Fn::Select

Provision EC2 Instance Automatically

---

• Adam, DevOps engineer at Globomantics
• CloudFormation template includes EC2 Instance
• Launch Web Server on EC2 Instance automatically

Necessary Steps During Bootstrap

User Data of EC2 Instance

   "Type": "AWS::EC2::Instance",
	  "Properties": {
	  "UserData": "..."
	}
			      

User Data Property

User Data can be described as property of an EC2 Instance

But User Data needs to be encoded in base64

   {"Fn::Base64": "A String"}
			      

Fn::Base64

There’s a built in function for that

Encodes a String in base64

   {"Fn::Join": [";", ["a", "b", "c"]]}
			      

Fn::Join

Allows you to concatenate Strings

Accepts a delimiter and a list of Strings as input

User Data Containing Shell Script

   "UserData": {"Fn::Base64": {"Fn::Join": ["\n", [
	  "#!/bin/bash-ex",
	  "yum install-y httpd",
	  "cd /var/www/html",
	  "echo '<html><body>...</body></html>' > index.html",
	  "service httpd start"
   ]]}}
			      

• CloudFormation Template
  • EC2 Instance
  • User Data containing Shell Script
• Create stack with the help of AWS CLI and access the created Web Server

Summary

• Mappings section
  • Map between region and AMI
• Pseudo Parameters
• Built in functions
  • Fn::GetAZs
  • Fn::Select
  • Fn::Join
  • Fn::Base64
• User Data
  • Provision EC2 Instance automatically

Questions?

Thanks for You!

---