Create S3 Bucket using Terraform

Background

In an earlier post, I provided some information on using AWS Encryption SDK and in that post, I created a KMS key using the AWS CLI. In this post I am going to create the KMS key and S3 bucket using Terraform, which you can then use to store objects which are encrypted using Server Side Encryption.

The level of encryption you need depends on your individual case. You may want to just rely on Server Side Encryption, or you may want to encrypt using a Data Encryption Key.

Requirements

  • Install Terraform on your server or laptop.
  • Install AWS CLI on your server or laptop.
  • In my demonstration I have already configured an AWS profile called ‘automation’. You can either create a profile or assume a role or simply modify the provider.tf file and add your AWS credentials. See Access.

Sample Code

The Terraform code is split into multiple files, I will discuss the code in detail later in this post.

Here are some facts about what this Terraform example does.

Note: Running this demo will incur some cost. You can lower your cost by cleaning up and removing all resources after you are done. You may have to wait a bit when you create the resource. I have noticed that if you run an ‘apply’ and immediately follow it by a ‘destroy’, it fails. This is probably due to S3 propagation. Wait a while and then retry the ‘delete’.

  • The KMS Customer Managed Key I am creating has a smaller delete window and it will have an alias called ‘mycmk’.
  • An S3 bucket is created ‘skbali-demo-area’. You should use a unique name as S3 names are global. Two buckets cannot have the same name.
  • The S3 bucket has Server Side Encryption enabled and uses the Key we created ‘mycmk’.
  • S3 bucket also has a policy attached to it which allows the current user access to it.
  • There is a life cycle policy attached, which deletes objects under demo/ if they are older than 7 days.
  • In addition to the above, the bucket is set to disable public access.

Let us review the code a bit more in detail.

The data.tf file allows Terraform to pull the current authenticated user information and account id. I use this information to build the policy that is attached to the S3 bucket being created.


kms_key_alias = "mycmk"
kms_deletion_window_in_days = 7

tags = {
    "Purpose" = "Demo",
    "CostCenter" = "infra"
}

s3_bucket = "skbali-demo-area"

Here, I am defining an alias for my KMS key and reducing the delete window for the key. I am setting the name for my S3 bucket and some Tags that can be applied to it.

The code that puts all of it together is in main.tf. All the AWS resources are being created here. Each resource that will be created starts with the ‘resource’ line.

The outputs.tf file specifies what information will be displayed by Terraform when it completes.

  • Resource block for the S3 bucket, which creates bucket with a lifecycle rule and encryption enabled.
  • A resource block to make the bucket private.
  • Another resource block for the policy attached to it.

If you need any additional clarification on it, please feel free to comment below and ask.

Testing

To run our demo and create the resources, we have to first initialize our Terraform environment.

terraform --version
Terraform v0.12.23

terraform init

Initializing the backend...

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.54.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.54"

Terraform has been successfully initialized!

The ‘terraform init’ is an important step where Terraform scans your files and determines which is the cloud provider and then it installs the plugins needed by Terraform to run.

I also ran the version check to show what version I used in my demo.


terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.aws_caller_identity.current: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_kms_alias.kms_key_alias will be created
  + resource "aws_kms_alias" "kms_key_alias" {
      + arn            = (known after apply)
      + id             = (known after apply)
      + name           = "alias/mycmk"
      + target_key_arn = (known after apply)
      + target_key_id  = (known after apply)
    }

............... More Output ..........

  # aws_s3_bucket_public_access_block.bucket will be created
  + resource "aws_s3_bucket_public_access_block" "bucket" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

Plan: 5 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

I always make it a habit to run ‘terraform plan’. This will give you a clear idea of what would happen when you run an ‘apply’. You can see in the abbreviated output, what Terraform thinks will happen.

Five new resources will be created for us, this includes the KMS key, S3 bucket and associated policies.


terraform apply

aws_kms_key.key: Creating...
aws_kms_key.key: Creation complete after 1s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]
aws_kms_alias.kms_key_alias: Creating...
aws_s3_bucket.bucket: Creating...
aws_kms_alias.kms_key_alias: Creation complete after 1s [id=alias/mycmk]
aws_s3_bucket.bucket: Creation complete after 3s [id=skbali-demo-area]
aws_s3_bucket_public_access_block.bucket: Creating...
aws_s3_bucket_policy.demo-policy: Creating...
aws_s3_bucket_public_access_block.bucket: Creation complete after 0s [id=skbali-demo-area]
aws_s3_bucket_policy.demo-policy: Creation complete after 1s [id=skbali-demo-area]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

account_id = xxxxxxxxxxxx
caller_arn = arn:aws:iam::xxxxxxxxxxxx:user/tf
caller_user = XXXXXXXXXXXXXXXXXXXXX
kms_key_alais_arn = arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/mycmk
kms_key_alias_name = alias/mycmk
kms_key_arn = arn:aws:kms:us-east-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
kms_key_id = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

If you are ok with the plan, go ahead and run ‘terraform apply’. Depending on the resources being created, it can take from few seconds to hours. In our case, it should not take more than a few seconds.

My plan output prints the information I wanted to display from my outputs.tf file. The information is obscured here to protect my account number, key id and user. You should not share this information.

dd if=/dev/urandom of=./xyz bs=1024 count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB, 9.8 MiB) copied, 0.0688974 s, 149 MB/s

ls -la xyz
-rw-r--r-- 1 sbali sbali 10240000 Mar 20 08:46 xyz

aws s3 cp xyz s3://skbali-demo-area/demo/xyz --profile automation
upload: ./xyz to s3://skbali-demo-area/demo/xyz        

Next step is to create a random file and upload it to S3 and examine its meta data.

The image below clearly shows that the uploaded file is using AWS-KMS for encryption.

S3 Object with SSE

Summary

Using Terraform, it was quite easy to setup a KMS key, S3 bucket with Server Side Encryption enabled.

Let me know by commenting below, if you need any clarification with this demo. Also if no longer required, you can run ‘terraform destroy’ to remove all the resources that were created.

Further Reading

Leave a Reply