About me

DevOps@RajeshKumar.XYZ

“All service interfaces, without exception, must be designed from the ground up to be externalizable.” –Attributed to Jeff Bezos

Why does this matter?

Everything in AWS Uses the API

The Boiling Frog Analogy

• Console is easy for short tasks
• Very little learning curve
• Learning the console increases speed
• Until......
• Too many tasks
• No guaranteed quality
• = Boiled frog!
• How can we automate tasks with quality?

Course Content

• Install and configure CLI
• Secure and monitor CLI
• Simple operations
• Complex operations
• Scripting tasks: alternatives

What Is the AWS CLI?

Install CLI on Windows

• Install CLI on MacOS
  • Python Six prerequisite workaround

• Install CLI on Linux
  • PIP
  • Standalone
• Configure CLI
  • Basics
  • Extras
• AWS Completer
• AWS Shell

Module Summary

• Console is easy, until you require automation
• Single, unified CLI for many OS
• Many options after install
• Up Next: Securing and Monitoring the CLI

Securing and Monitoring the CLI

2

Module Intro

• Securing the CLI
  • Credentials
  • EC2 instance profile credentials
• Demo: Credential Order of Preference
• Demo: Integrate CloudTrail and CloudWatch Logs to Monitor CLI

Order of Preference for Credentials

• Command line options
• Environment variables
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
• Credentials file
  • ~/.aws/credentials
  • C:\Users\USERNAME\.aws\credentials
• CLI configfile
  • ~/.aws/config
  • C:\Users\USERNAME\.aws\config
• Instance profile credentials

Instance Profile Credentials Explained

• Credential Order of Preference
  • Show ways to supply credentials to CLI

• Monitoring CLI Usage
  • Enable CloudTrail
  • Integrate with CloudWatch Logs
  • Create Filter
  • Create Alarm
  • Test

Module Summary

• Many ways to supply credentials to CLI
• Adopting security early is a best practice
• Up Next: Simple Operations – CloudWatch metrics

Simple Operations: Exporting Data from CloudWatch

3

Module Intro

• Strategic goal: cost optimization
• Scenario: Use EC2 tags to locate underutilized instances
• Demo: Analyze CloudWatch Data and Set EC2 Tags

“You can achieve cost optimization using techniques to select the appropriate architecture, reduce unused resources, and select the most economical approach.”

http://d0.awsstatic.com/whitepapers/architecture/AWS_Well-Architected_Framework.pdf

Storing CloudWatch Data in EC2 Tags

How can we use this data?

Generate instance inventory monthly and use EC2 tag values to identify candidates for resizing

Example: Filter EC2 instances for application X and downsize any that have max CPU percentage less than 30

• Analyze CloudWatch Data and Set EC2 Tags
  • Part 1
    • Script to initialize tag value
  • Part 2
    • Command to export cpu metrics
    • Comparison to existing tag value
    • Update tag if appropriate

Module Summary

• The CLI can be used to accomplish strategic goals
• Tags can be set and updated frequently
• Up Next: Simple Operations – Copying and Syncing Objects Between S3 Buckets

Simple Operations: Copying and Syncing Objects Between S3 Buckets

4

Module Intro

• Strategic goal: Simplicity of data migration
• Scenario: Frequent data transfer between S3 buckets
• Demo: S3 copy and sync between buckets

AWS S3 Dashboard

Why Use S3?

• Durability
• Ease of access
• Sticky glue that binds data-driven workloads together

Data Transfer Between S3 Buckets

• S3 copy and sync between buckets
  • Part 1
    • Upload and/or sync data to S3
  • Part 2
    • Copy between buckets in same account
    • Copy between buckets in different accounts

Module Summary

• AWS console is not easy for certain workflows
• Data transfer involves more than a source and destination
• Up Next: Complex Operations: Turn Off Environment After Hours

Complex Operations: Turn Off Environment After Hours

5

Module Intro

• Strategic goal: Cost optimization and automation
• Scenario: Shut down non-production environment after hours/weekends
• Demo: Build and test environment stop and start scripts

CLI vs OS Operations

Best Practices and Challenges

• Launch from scratch and bootstrap
  • What about time constraints?
• Stop Unused Resources
  • What if some resources must stay up?
• Use CLI for all Operations
  • How do you combine CLI and OS?

Environment Stop/Start Prerequisites

Organic Growth of Environment

Scripts Required

• Environment Stop
  • Part 1
    • Generate List of Instances
  • Part 2
    • Stop Services In Order
  • Part 3
    • Stop Instances, Ignore Exceptions

Module Summary

• Organic environment growth happens
• Real world ops can be complicated
• Use tags for simplifying operations
• Up Next: Rotate and Expire AMI Backups

Complex Operations: Rotate and Expire AMI Backups

6

Module Intro

• Strategic goal: cost optimization and devopsautomation
• Scenario: create daily AMI and rotate 7 versions
• Demo: schedule daily AMI creation, rotation scripts

AMI Delete Flow (All OS)

Challenge:Daily Backup

• Recovery point objective: <1 day
• Recovery time objective: <1 hour
• Reboot OK at night: <5 minutes
• Singleton server
• Infrequent updates
• No continuous deployment requirement

Implementation Process

• AMI backup and rotation
  • Part 1
    • Create AMI
  • Part 2
    • Rotate AMI

Module Summary

• AMI creation: 1 Step, AMI delete: 2 Steps
• AMI is not an identical backup
• CLI has no mechanism for AMI rotation
• Up next: Scripting Tasks: Alternatives to Shell Scripts

Scripting Tasks: Alternatives to Shell Scripts

7

Module Intro

• Linux shell scripts
  • Important
  • Not always appropriate or necessary
• What about ServerlessArchitectures?
• Demo: Using Lambda for SSH
• What if I like PowerShell for scripting?
• Demo: Using PowerShell for CLI

AWS Lambda

• Define a function
• Configure a trigger
• Specify permissions and resources
• Choose location (such as VPC)
• That’s it!
• Unless you want to perform OS ops

• Using Lambda for SSH
  • Function 1
    • Inventory and trigger
  • Function 2
    • SSH and perform OS operations

MS Power Shell

• Using PowerShell for CLI
  • Part 1
    • Examples of CLI
  • Part 2
    • Examples of using AWS Tools

Module Summary

• CLI is not the only tool for automation
• Evaluate serverless technology first
• Some operations are easier using SDK

Course Summary

• Evolve beyond manual operations
• CLI usage should include security and auditability
• CLI is simple by design – both good and bad
• Operations cover a wide spectrum of services and OS
• Use the most appropriate tool for the job
• Stay current via AWS Blog!

Questions?

Thanks for You!

---