Create S3 Bucket using Terraform

Updated on 3rd Sep 2021. Tested build with Terraform version 1.0.6

Background

In an earlier post, I provided some information on using AWS Encryption SDK and in that post, I created a KMS key using the AWS CLI. In this post I am going to create the KMS key and S3 bucket using Terraform, which you can then use to store objects which are encrypted using Server Side Encryption.

The level of encryption you need depends on your individual case. You may want to just rely on Server Side Encryption, or you may want to encrypt using a Data Encryption Key.

Requirements

  • Install Terraform on your server or laptop.
  • Install AWS CLI on your server or laptop.
  • In my demonstration I have already configured an AWS profile called ‘automation’. You can either create a profile or assume a role or simply modify the provider.tf file and add your AWS credentials. See Access.

Note: Published a new post to Create Storage Bucket in Google Cloud using Terraform.

Sample Code

The Terraform code is split into multiple files, I will discuss the code in detail later in this post.

Here are some facts about what this Terraform example does.

Note: Running this demo will incur some cost. You can lower your cost by cleaning up and removing all resources after you are done. You may have to wait a bit when you create the resource. I have noticed that if you run an ‘apply’ and immediately follow it by a ‘destroy’, it fails. This is probably due to S3 propagation. Wait a while and then retry the ‘delete’.

  • The KMS Customer Managed Key I am creating has a smaller delete window and it will have an alias called ‘mycmk’.
  • An S3 bucket is created ‘skbali-demo-area’. You should use a unique name as S3 names are global. Two buckets cannot have the same name.
  • The S3 bucket has Server Side Encryption enabled and uses the Key we created ‘mycmk’.
  • S3 bucket also has a policy attached to it which allows the current user access to it.
  • There is a life cycle policy attached, which deletes objects under demo/ if they are older than 7 days.
  • In addition to the above, the bucket is set to disable public access.

Let us review the code a bit more in detail.

The data.tf file allows Terraform to pull the current authenticated user information and account id. I use this information to build the policy that is attached to the S3 bucket being created.


kms_key_alias = "mycmk"
kms_deletion_window_in_days = 7

tags = {
    "Purpose" = "Demo",
    "CostCenter" = "infra"
}

s3_bucket = "skbali-demo-area"

Here, I am defining an alias for my KMS key and reducing the delete window for the key. I am setting the name for my S3 bucket and some Tags that can be applied to it.

The code that puts all of it together is in main.tf. All the AWS resources are being created here. Each resource that will be created starts with the ‘resource’ line.

The outputs.tf file specifies what information will be displayed by Terraform when it completes.

  • Resource block for the S3 bucket, which creates bucket with a lifecycle rule and encryption enabled.
  • A resource block to make the bucket private.
  • Another resource block for the policy attached to it.

If you need any additional clarification on it, please feel free to comment below and ask.

Testing

To run our demo and create the resources, we have to first initialize our Terraform environment.

Note: Now that Terraform 0.14 is GA, I upgraded my binary and tested the plan, apply, destroy and it worked well. Do remember to run a terraform init.

Terraform v1.0.6
on linux_amd64

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.57.0...
- Installed hashicorp/aws v3.57.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The ‘terraform init’ is an important step where Terraform scans your files and determines which is the cloud provider and then it installs the plugins needed by Terraform to run.

I also ran the version check to show what version I used in my demo.


terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_kms_alias.kms_key_alias will be created
  + resource "aws_kms_alias" "kms_key_alias" {
      + arn            = (known after apply)
      + id             = (known after apply)
      + name           = "alias/mycmk"
      + name_prefix    = (known after apply)
      + target_key_arn = (known after apply)
      + target_key_id  = (known after apply)


............... More Output ..........

  # aws_s3_bucket_public_access_block.bucket will be created
  + resource "aws_s3_bucket_public_access_block" "bucket" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + account_id         = "123456789012"
  + caller_arn         = "arn:aws:iam::123456789012:user/tf"
  + caller_user        = "AIDXXXXXXXXXXX"
  + kms_key_alais_arn  = (known after apply)
  + kms_key_alias_name = "alias/mycmk"
  + kms_key_arn        = (known after apply)
  + kms_key_id         = (known after apply)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

I always make it a habit to run ‘terraform plan’. This will give you a clear idea of what would happen when you run an ‘apply’. You can see in the abbreviated output, what Terraform thinks will happen.

Five new resources will be created for us, this includes the KMS key, S3 bucket and associated policies.


terraform apply

aws_kms_key.key: Creating...
aws_kms_key.key: Creation complete after 1s [id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]
aws_kms_alias.kms_key_alias: Creating...
aws_s3_bucket.bucket: Creating...
aws_kms_alias.kms_key_alias: Creation complete after 1s [id=alias/mycmk]
aws_s3_bucket.bucket: Creation complete after 3s [id=skbali-demo-area]
aws_s3_bucket_public_access_block.bucket: Creating...
aws_s3_bucket_policy.demo-policy: Creating...
aws_s3_bucket_public_access_block.bucket: Creation complete after 0s [id=skbali-demo-area]
aws_s3_bucket_policy.demo-policy: Creation complete after 1s [id=skbali-demo-area]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

account_id = xxxxxxxxxxxx
caller_arn = arn:aws:iam::xxxxxxxxxxxx:user/tf
caller_user = XXXXXXXXXXXXXXXXXXXXX
kms_key_alais_arn = arn:aws:kms:us-east-1:xxxxxxxxxxxx:alias/mycmk
kms_key_alias_name = alias/mycmk
kms_key_arn = arn:aws:kms:us-east-1:xxxxxxxxxxxx:key/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
kms_key_id = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

If you are ok with the plan, go ahead and run ‘terraform apply’. Depending on the resources being created, it can take from few seconds to hours. In our case, it should not take more than a few seconds.

My plan output prints the information I wanted to display from my outputs.tf file. The information is obscured here to protect my account number, key id and user. You should not share this information.

dd if=/dev/urandom of=./xyz bs=1024 count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB, 9.8 MiB) copied, 0.0688974 s, 149 MB/s

ls -la xyz
-rw-rw-r-- 1 ubuntu ubuntu 10240000 Sep  3 18:13 xyz

aws s3 cp xyz s3://skbali-demo-area/demo/xyz --profile automation
upload: ./xyz to s3://skbali-demo-area/demo/xyz        

Next step is to create a random file and upload it to S3 and examine its meta data.

The image below clearly shows that the uploaded file is using AWS-KMS for encryption.

Summary

Using Terraform, it was quite easy to setup a KMS key, S3 bucket with Server Side Encryption enabled.

Let me know by commenting below, if you need any clarification with this demo. Also if no longer required, you can run ‘terraform destroy’ to remove all the resources that were created.

Note

To be able to run a terraform destroy, the bucket must be empty. In the demo above I copied an object to the S3 bucket, so it is not empty.

Terraform can destroy all objects in the bucket is you add the following

force_destroy = true

Be very careful with this option. There is no undo. If you run a terraform destroy and proceed with it, all objects in the S3 bucket will be deleted.

Further Reading

8 COMMENTS

  1. thanks for the post, I need to know few things:
    1. is it better to use tfvars file instead of variable.tf file and interpolate the variables ?
    2. I did not understand what is “aws_caller_identity”, how is it being used
    3. Can we define the bucket policy in another file and refer that json file, instead of putting policy in main.tf
    4. Can you provide a tutorial on hosting a static website on s3 using cloudfront, creating OAI, creating IAM role/user etc. ?
    Thanks again

    • 1. TBH – I am not sure what is better. Will need to research more on it.
      2. If you look at line 60 and 69 of main.tf where I refer “${data.aws_caller_identity.current.arn}”. I want to give S3 access to the current user. aws_caller_identity provides me the ARN of the IAM user running this demo.
      3. See https://learn.hashicorp.com/terraform/aws/iam-policy You could define policy in another file. But I wanted it to be dynamic as I want to provide bucket access to the user running the demo. If the policy was in another file, I might have hard coded the user.
      4. Hopefully I will get to try it out.

      Thanks for taking the time to read, point out my mistake and trying the demo code!

  2. I declared variables and it works fine in that case. I want to know if we need to create any cmk first and then refer in the variables as “mycmk” ? didn’t get that part as what is mycmk, do we need to create that in AWS console first and then refer that? any way to create it by terraform ?

    • The sample code creates the customer managed key ‘cmk’ and then uses it for the S3 bucket. You do not have to do it in the console.

      ‘mycmk’ is the alias for the key that is created. It is ‘my customer managed key’. I could have named it s3cmk to identify it is a customer managed key used for s3.

    • I just did a test and it works for me. A KMS key describe shows when it was created and the scheduled deletion. I set my window to 8 days.

      “CreationDate”: 1587121410.154,
      “Enabled”: false,
      “Description”: “mycmk”,
      “KeyUsage”: “ENCRYPT_DECRYPT”,
      “KeyState”: “PendingDeletion”,
      “DeletionDate”: 1587859200.0,

Leave a Reply