Update Image Using Packer

Background

Recently I published a post to show how to remove old EC2 instances using a Lambda function. This in itself does not do much for security, but it plays and important part in keeping infrastructure safe.

I asked my self what more could I do to keep the EC2 instances secure. This post extends my earlier work, and what I am going to demonstrate here is the procedure I implemented to keeping my EC2 servers up to date by creating new AMI with updated software.

Packer is a well know open source tool that can be used for creating identical machine images for multiple platforms from a single source configuration. Packer Intro Link.

I am going to use packer to demonstrate how I keep my Ubuntu images up to date with the latest software packages.

Requirements

There are a few requirements that you will have to take care off, to follow my post.

  • My demonstration is done using AWS CLI and a bash script, which means you will need a server with the CLI installed. See below for useful links.
  • You will have to install the packer software.
  • To build a new AWS AMI, you need to have the correct role or permissions. This means that the server from where you are going to run the script shown here must have an AWS profile or access keys set. Build Image access.
  • My sample code does not have access keys embedded, but you can easily modify the code to do so. I run my scripts for an EC2 server which already has a profile set on it.
  • See more on access keys here.
  • jq is a big help in parsing output from AWS CLI.

Sample Code

Let us take a look at same code.

  • ubuntu-intall.sh – this file is used by packer to update the software on my Ubuntu based EC2 instance.
  • ubuntu-packer.json – this is where we define how packer is going to build the new image.
  • update-asg-lt.sh – is the script that does all the work.

The ubuntu-install script is fairly easy to understand. you can add other commands here if needed. You may want to add some software or create some other files, or make configuration changes.


  "variables": {
      "demo-source-ami": "ami-xxxxxxxxxx",
      "demo-subnet": "subnet-xxxxxxxxxx",
      "demo-costcenter": "infra",
      "demo-createdby": "infra",
      "demo-name": "packer_AWS-{{timestamp}}"
  },

The ubuntu-packer.json file starts of by defining some variables that I use with my AMI’s. I use tags to keep tack of my infrastructure. If a variable is not passed from command line, the defaults specified here are used.

Note

The defaults shown here are not valid. As I do not want to show real AMI or Subnet id’s.


  "builders": [
    {
      "type": "amazon-ebs",
      "region": "us-east-1",
      "subnet_id": "{{user `demo-subnet`}}",
      "source_ami": "{{user `demo-source-ami`}}",
      "instance_type": "t2.small",
      "ssh_username": "ubuntu",
      "ami_name": "{{user `demo-ami-name`}}",
      "tags": {
         "CostCenter": "{{user `demo-costcenter`}}",
         "CreatedBy": "{{user `demo-createdby`}}",
         "Name": "{{user `demo-name`}}"
      }
    }
  ],

The ‘builders’ is where you may want to review the region being used, the EC2 instance type and tags if you use them. You can change instance type, username used by packer to ssh.

Packer documentation is very helpful and I have provided links in this post for you to read more.

‘Provisioners’ is where our shell script is defined that does the software update on the EC2 instance when it gets created.

The ‘post-processors’ is needed to output information about the AMI that packer created. We need this AMI id to update our launch template.

I have a post where I show how to setup a Launch Template. You can refer to it if you need more information on AWS Launch Templates.

The shell script brings everything together.


# Query ASG for Launch Template Id, Version and a usable subnet
out=$(aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name ${ASG})
srcSubnet=$(echo $out | jq -r '.AutoScalingGroups[].VPCZoneIdentifier' | cut -f1 -d ',')
srcLTId=$(echo $out | jq -r '.AutoScalingGroups[].LaunchTemplate.LaunchTemplateId')
srcLTVersion=$(echo $out | jq -r '.AutoScalingGroups[].LaunchTemplate.Version')

This section looks up the Auto Scaling Group and finds out the Launch Template attached to it, the version of the template being used and the subnets being used by the ASG.


# Obtain the AMI id
srcAMI=$(aws ec2 describe-launch-template-versions --launch-template-id ${srcLTId} --versions ${srcLTVersion} \
             --query 'LaunchTemplateVersions[].LaunchTemplateData.ImageId' --output text)

Query the AMI id from the Launch Template.


# Get Tags from AMI
out=$(aws ec2 describe-images --image-id  ${srcAMI})
CreatedBy=$(echo $out | jq -r '.Images[].Tags[] | select(.Key=="CreatedBy") | .Value')
Name=$(echo $out | jq -r '.Images[].Tags[] | select(.Key=="Name") | .Value')
CostCenter=$(echo $out | jq -r '.Images[].Tags[] | select(.Key=="CostCenter") | .Value')

This section looks up tags of the existing image, so they can be set for the new image.


packer build -machine-readable\
  -var "demo-source-ami=${srcAMI}" \
  -var "demo-subnet=${srcSubnet}" \
  -var "demo-name=${Name}" \
  -var "demo-ami-name=${Name}-${Now}" \
  -var "demo-createdby=admin" \
  -var "demo-costcenter=${CostCenter}" \
  ubuntu-packer.json

This is the build command that starts off the process of creating an EC2 instance in your account and then make an AMI from it. The most important variables here are the source AMI and the subnet you want the EC2 to be created in.

If you remove variable or add more, you will have to make sure that they all match up with the packer template being used.

You can run a packer validate command to ensure your command and json template match up.


# Create new version of launch template
aws ec2 create-launch-template-version --launch-template-id ${srcLTId} \
    --version-description OSUpdate \
    --source-version ${srcLTVersion} \
    --launch-template-data "{\"ImageId\":\"${outAMI}\"}"

This final step creates a new Launch Template version and updates it with the new AMI id.

Note:

This example is for ubuntu based servers but can easily be adapted for other operating servers.

Testing

Let us go ahead and test our image build process. I already have an Auto Scaling Group called demo which uses a launch template to create instances. I am going to run my test script against this ASG.


./update-asg-lt.sh demo

1583673463,,ui,say,==> amazon-ebs: Prevalidating any provided VPC information
1583673463,,ui,say,==> amazon-ebs: Prevalidating AMI Name: demo-20200308-131741
1583673464,,ui,message,    amazon-ebs: Found Image ID: ami-01a0a0xxxxxx
1583673464,,ui,say,==> amazon-ebs: Creating temporary keypair: packer_5e64f077-2098-0271-69b6-c6624a379571
1583673464,,ui,say,==> amazon-ebs: Creating temporary security group for this instance: packer_5e64f078-b0d3-ed44-f66f-xxxxxxxxxxx
1583673465,,ui,say,==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
1583673465,,ui,say,==> amazon-ebs: Launching a source AWS instance...
1583673465,,ui,say,==> amazon-ebs: Adding tags to source instance
1583673465,,ui,message,    amazon-ebs: Adding tag: "Name": "Packer Builder"
1583673466,,ui,message,    amazon-ebs: Instance ID: i-0de452xxxxxx
1583673466,,ui,say,==> amazon-ebs: Waiting for instance (i-0de452xxxxxxx) to become ready...

As you can see above, the script was able to validate my packer information and start the process of creating an EC2 instance.


..... more output .....
1583673592,,ui,error,==> amazon-ebs: + sudo sh -c apt-get update -y
1583673593,,ui,message,    amazon-ebs: Hit:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial InRelease
1583673593,,ui,message,    amazon-ebs: Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
1583673595,,ui,message,    amazon-ebs: Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB]
..... more output .....
1583673681,,ui,say,==> amazon-ebs: Stopping the source instance...
1583673681,,ui,message,    amazon-ebs: Stopping instance
1583673682,,ui,say,==> amazon-ebs: Waiting for the instance to stop...
1583673727,,ui,say,==> amazon-ebs: Creating AMI demo-20200308-131741 from instance i-0de452xxxxxx
1583673727,,ui,message,    amazon-ebs: AMI: ami-05b44c5cxxxxxxx
1583673727,,ui,say,==> amazon-ebs: Waiting for AMI to become ready...
..... more output .....
1583673803,,ui,say,==> amazon-ebs: Creating snapshot tags
1583673803,,ui,say,==> amazon-ebs: Terminating the source AWS instance...
1583673819,,ui,say,==> amazon-ebs: Cleaning up any extra volumes...
1583673819,,ui,say,==> amazon-ebs: No volumes to clean up%!(PACKER_COMMA) skipping
1583673819,,ui,say,==> amazon-ebs: Deleting temporary security group...
1583673819,,ui,say,==> amazon-ebs: Deleting temporary keypair...
1583673819,,ui,say,==> amazon-ebs: Running post-processor: manifest
1583673819,,ui,say,Build 'amazon-ebs' finished.

The output is quite long, so I have abbreviated it, but you can see that packer was able to create the EC2 instance and started applying the ubuntu updates.

After the AMI is ready, packer cleans up all the temporary resources that were created during the build process.


{
    "LaunchTemplateVersion": {
        "CreatedBy": "arn:aws:iam::xxxxxxxxxxxxxx:user/sbali",
        "LaunchTemplateData": {
            "Monitoring": {
                "Enabled": false
        ..... more output .....       
        "VersionDescription": "OSUpdate",
        "LaunchTemplateName": "demo-20200216-001",
        "DefaultVersion": false,
        "VersionNumber": 5,
        "LaunchTemplateId": "lt-07fa0d7xxxxxxx",
        "CreateTime": "2020-03-08T13:23:40.000Z"
    }
}

This is followed by output from the command used to create a new version of Launch Template. Once the new Launch Template is ready, you can create an instance from this template for testing before you update your auto scaling group.

Improvements

My sample code is enough for you to be able to write your own script to create an updated AMI for you to use in your account. This code was for demonstration purpose only and should be enhanced before you use it in a production environment.

  • To keep the sample code to the point, I do not have error handling or notification. In a production environment, you should handle exception from every command and implement a notification process.
  • In my script I start the process by querying an Auto Scaling Group, however you can easily adapt the script to start the process by querying a launch template. Not all launch templates are used by ASG’s.
  • You can also modify the script to have a for loop to process all Auto Scaling Groups at once.
  • If your script is mature enough and you are confident that it is what you need, you could go one step further and update the Auto Scaling Group to use the new version of Launch Template. Any new EC2 instance created after this update would create instances from the new AMI.
  • Check AMI is for Ubuntu – use tags to filter.
  • Extend procedure for other operating systems.

Summary

This procedure demonstrates, that it is fairly easy to automate the process of creating new server images which have the latest software packages installed on them.

Use along with a lambda function to remove older instances, you can be assured that your infrastructure is secure and always upto date.

Note:

  • You should always test any image you create in a non production environment to ensure, there is no software incompatibility or configuration issues as a result of the update procedure.
  • AMI cost money, please remember to cleanup resources that you do not need.

Further Reading

Photo Credit:

unsplash-logohenrique setim

Leave a Reply