Create GCP Storage Bucket Using Terraform

Updated on 4th Dec 2020. Tested build with Terraform version 0.14

Background

In an earlier post, I demonstrated how one could setup an AWS S3 bucket using Terraform. The S3 bucket was setup to use a Customer Managed Key (CMK). In this post I am going to show how to create a GCP Storage Bucket using Terraform. The storage bucket will be encrypted using a Customer Managed Key just like I did for AWS S3.

The code will first create a Customer Managed Key and then a GCP Storage Bucket using that key.

Requirements

In my demonstration I am not setting the credentials in the Terraform code, but setting a variable which points to my credentials file.


export GOOGLE_APPLICATION_CREDENTIALS="/home/sbali/my-gke-003-xxxxxxxxxx.json"

I am using the latest version which is 0.13.5 at the time of writing this post.

Note: Terraform 0.14 is GA. I upgraded my binary and tested it with the latest release.


terraform --version
Terraform v0.14.0

Code

Note: Running this demo will incur some cost. You can lower your cost by cleaning up and removing all resources after you are done.

The Code listed here will be used to create:

  • A Google Customer Managed Key
  • A Google Storage Bucket encrypted with the CMK.
  • The Storage Bucket has versioning enabled and a lifecycle rule to delete objects older than 1 day.

This setup is for demonstration purpose only, you would use a rule suitable for your environment.

Here are the files I used to setup my key and storage bucket.

Let us review the code a bit more in detail.

The code that puts all of it together is in main.tf. All the GCP resources are being created here. Each resource that will be created starts with the ‘resource’ line.

The variables.tf is where I have specified some defaults.

The main.auto.tfvars file supplies the overrides for values.

The outputs.tf file specifies what information will be displayed by Terraform when it completes.

The first resource in main.tf, ‘random_id’ allows me to get a 4 character string that will remain the same on all Terraform apply until I change the key_name or bucket_name.

The next two resources create a Key ring and Key. This is our Customer Managed Key.

The resource ‘google_kms_crypto_key_iam_binding’ is used to give permissions to Encrypt and Decrypt to the Storage service account in our project.

The resource ‘google_storage_bucket’ is used to create the storage bucket. Pay attention to the depends_on clause here.


  depends_on = [
    google_kms_crypto_key_iam_binding.crypto_key
  ]

This ensure that the GCP Storage Bucket is created after the Service Account is given access to the Customer Managed Key.

Without the depends_on and older Terraform version, you would get an error on the first run of Terraform apply.

Testing

To run our demo and create the resources, we have to first initialize our Terraform environment.

Run the terraform init command as shown

The ‘terraform init’ is an important step where Terraform scans your files and determines which is the cloud provider and then it installs the plugins needed by Terraform to run.


terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Finding latest version of hashicorp/random...
- Installing hashicorp/google v3.49.0...
- Installed hashicorp/google v3.49.0 (signed by HashiCorp)
- Installing hashicorp/random v3.0.0...
- Installed hashicorp/random v3.0.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/google: version = "~> 3.49.0"
* hashicorp/random: version = "~> 3.0.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

After the init, I can go ahead and run the plan to see what infrastructure would be setup.


terraform plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.google_project.project: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_kms_crypto_key.key will be created
  + resource "google_kms_crypto_key" "key" {
      + id                            = (known after apply)
      + key_ring                      = (known after apply)
      + name                          = "key-sbali"
      + purpose                       = "ENCRYPT_DECRYPT"
      + rotation_period               = "2592000s"
      + self_link                     = (known after apply)
      + skip_initial_version_creation = false

      + version_template {
          + algorithm        = "GOOGLE_SYMMETRIC_ENCRYPTION"
          + protection_level = "SOFTWARE"
        }
    }

  # google_kms_crypto_key_iam_binding.crypto_key will be created
  + resource "google_kms_crypto_key_iam_binding" "crypto_key" {
      + crypto_key_id = (known after apply)
      + etag          = (known after apply)
      + id            = (known after apply)
      + members       = [
          + "serviceAccount:[email protected]",
        ]
      + role          = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
    }

  # google_kms_key_ring.keyring will be created
  + resource "google_kms_key_ring" "keyring" {
      + id        = (known after apply)
      + location  = "us-central1"
      + name      = (known after apply)
      + project   = (known after apply)
      + self_link = (known after apply)
    }

  # google_storage_bucket.bucket will be created
  + resource "google_storage_bucket" "bucket" {
      + bucket_policy_only          = (known after apply)
      + force_destroy               = true
      + id                          = (known after apply)
      + location                    = "US-CENTRAL1"
      + name                        = (known after apply)
      + project                     = (known after apply)
      + self_link                   = (known after apply)
      + storage_class               = "STANDARD"
      + uniform_bucket_level_access = true
      + url                         = (known after apply)

      + encryption {
          + default_kms_key_name = (known after apply)
        }

      + lifecycle_rule {
          + action {
              + type = "Delete"
            }

          + condition {
              + age                   = 1
              + matches_storage_class = []
              + with_state            = (known after apply)
            }
        }

      + versioning {
          + enabled = true
        }
    }

  # random_id.rand will be created
  + resource "random_id" "rand" {
      + b64_std     = (known after apply)
      + b64_url     = (known after apply)
      + byte_length = 4
      + dec         = (known after apply)
      + hex         = (known after apply)
      + id          = (known after apply)
      + keepers     = {
          + "bucket_name" = "sbali-storage"
          + "key_name"    = "key-sbali"
        }
    }

Plan: 5 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

I always make it a habit to run ‘terraform plan’. This will give you a clear idea of what would happen when you run an ‘apply’. You can see in the output, what Terraform thinks will happen.

Five new resources will be created for us, this includes the keyring, key, a GCP Storage bucket, random id and associated policies.

Let us go ahead and apply now.


terraform apply --auto-approve
data.google_project.project: Refreshing state...
random_id.rand: Creating...
random_id.rand: Creation complete after 0s [id=xfSJ1w]
----------------------------
----------------------------
google_storage_bucket.bucket: Creating...
google_storage_bucket.bucket: Creation complete after 1s [id=sbali-storage-c5xxxxxx]

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

gcs_account = "[email protected]"
kms_key_id = projects/my-gke-003/locations/us-central1/keyRings/keyring-sbali-c5xxxxxx/cryptoKeys/key-sbali
storage_bucket_id = sbali-storage-c5xxxxxx

My plan output prints the information I wanted to display from my outputs.tf file. The information is obscured and shortened here to protect some of the data.

I am going to create a small file and upload it to the storage bucket we just created and confirm that it is protected by our Customer Managed Key.


dd if=/dev/urandom of=./xyz bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB, 100 KiB) copied, 0.0179364 s, 5.7 MB/s

gsutil cp xyz gs://sbali-storage-c5xxxxx/
Copying file://xyz [Content-Type=application/octet-stream]...
/ [1 files][100.0 KiB/100.0 KiB]
Operation completed over 1 objects/100.0 KiB

Let us check the metadata associated with the file that we just uploaded to the storage bucket.


gsutil stat  gs://sbali-storage-c5xxxxxx/xyz
gs://sbali-storage-c5xxxxxx/xyz:
    Creation time:          Mon, 23 Nov 2020 21:47:28 GMT
    Update time:            Mon, 23 Nov 2020 21:47:28 GMT
    Storage class:          STANDARD
    KMS key:                projects/my-gke-003/locations/us-central1/keyRings/keyring-sbali-c5xxxxx/cryptoKeys/key-sbali/cryptoKeyVersions/1
    Content-Language:       en
    Content-Length:         102400
    Content-Type:           application/octet-stream

The output above shows us that the object is protected by the key that we created with Terraform earlier.

Summary

Using Terraform, it was quite easy to setup a GCP Storage bucket encrypted with a Customer Managed Key.

Let me know by commenting below, if you need any clarification with this demo. Also if no longer required, you can run ‘terraform destroy‘ to remove all the resources that were created.

Further Reading

Photo Credit

Photo by Steve Johnson on Unsplash

2 COMMENTS

  1. Hi, Will it be possible to remove the bucket resource and keep the bucket.

    I came across the situation that previously bucket was created by count loop which wont allowing me to add retention policy. Now I’m trying to create same bucket with for_each loop. So to achieve this I’m trying to drop that bucket resource without deleting the bucket.

    Once that resource will be destroyed i would like to create it with for_each loop with retention policy.

    Thanks in Advance for your solution!

    • You want to remove the resource declaration from the tf file but not delete the bucket that was created by it. If you already have data in the bucket and are worried about loosing it, I suggest you try your experiments with a different set of buckets. I have not tried this out but it should be a good experiment. Whatever you do, make sure you run a plan to understand what you are about to do.

Leave a Reply