S3

Amazon Simple Storage Service, and how to deploy using AWS CDK

Share this page:

If you’ve just landed here, we’re doing a “Become a Cloud Architect” Unicorn Workshop by building a Unicorn Pursuit Web App step by step, and you’re more then welcome to join!

About S3

S3 is an Object Storage. This means it allows you to upload only files, or “objects”.

Files can be 0 - 5 TB
There’s an unlimited storage (you don’t need to allocate)
Files are stored in Buckets, organized in Folders.

There are different storage classes:

S3
S3 IA (Infrequent Access)
S3 One Zone IA
Reduced Redundancy, for files that can be easily replaced, and this is no longer recommended by AWS.
Glacier (acceptable to wait 3-5 hours for access)

S3 objects have:

Key (name)
Value (data)
Version ID
Metadata

Securing our Buckets

We can secure our S3 using:

Bucket Policies, attached to a Bucket
Access Control Lists, applied on a PER OBJECT level.
Access LOGs (log all requests to access the bucket)

By default, all buckets are configured not to be publicly available. To change this, you need to change the Bucket Policy first. Once this is configured, you need to make sure the files you need to be publicly available have the right Access Control List.

Code

Before we start, let’s understand what we’re building. Check out the diagram below:

We want to store our publically accessable images to the S3 bucket, meaning - we need a Public Read access policy on the bucket level. This means, in our CloudFormation, we need something like this:

Resources:
  www.unicornpursuit.com:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead
      BucketName: www.unicornpursuit.com

If we check out our iac_stack.py within /unicorn/iac/iac, which is basically our Python that will create our resources, we first need to import “aws_s3”:

from aws_cdk import (
    aws_s3 as s3,
    core
)

Ok, now we need to see how to instantiate a class Bucket, and create the S3 Bucket exactly with the parameters we need. If we go to the official Python API Reference Documentation, the “Bucket” class gives us everything we can use:

classaws_cdk.aws_s3.Bucket(scope, id, *, access_control=None, block_public_access=None, bucket_name=None,
cors=None, encryption=None, encryption_key=None, lifecycle_rules=None, metrics=None, public_read_access=None, 
removal_policy=None, server_access_logs_bucket=None, server_access_logs_prefix=None, versioned=None,
website_error_document=None, website_index_document=None, website_redirect=None, website_routing_rules=None)

Based on this, we can create a S3 bucket, called www.unicornpursuit.com. At the moment we’ll leave it simple, and later we’ll add the encryption using KMS key with the encryption=BucketEncryption.KMS:

# Create an S3 bucket for Unicorn Pursuit web page, and grant public read:
bucket = s3.Bucket(self, "www.unicornpursuit.com",
    bucket_name="www.unicornpursuit.com",
    access_control=s3.BucketAccessControl.PUBLIC_READ,
    )

bucket.grant_public_access()

If you want to change the DeletionPolicy from Retain (default) to Delete, to the s3 class, so your “bucket” will look like this:

bucket = s3.Bucket(self, "www.unicornpursuit.com",
  bucket_name="www.unicornpursuit.com",
  access_control=s3.BucketAccessControl.PUBLIC_READ,
  removal_policy=core.RemovalPolicy.DESTROY,
  )

Ok, now lets Synchetize the CloudFormation Template:

(.env) ➜  iac git:(dev) ✗ cdk synth
Resources:
  wwwunicornpursuitcom6F23076E:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: PublicRead
      BucketName: www.unicornpursuit.com
      Tags:
        - Key: project
          Value: unicorn
        - Key: bu
          Value: cloud
        - Key: environment
          Value: prod
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: UnicornIaC/www.unicornpursuit.com/Resource
  wwwunicornpursuitcomPolicy46AB873C:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket:
        Ref: wwwunicornpursuitcom6F23076E
      PolicyDocument:
        Statement:
          - Action: s3:GetObject
            Effect: Allow
            Principal: "*"
            Resource:
              Fn::Join:
                - ""
                - - Fn::GetAtt:
                      - wwwunicornpursuitcom6F23076E
                      - Arn
                  - /*
        Version: "2012-10-17"
    Metadata:
      aws:cdk:path: UnicornIaC/www.unicornpursuit.com/Policy/Resource
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Modules: aws-cdk=1.36.1,@aws-cdk/aws-events=1.36.1,@aws-cdk/aws-iam=1.36.1,@aws-cdk/aws-kms=1.36.1,@aws-cdk/aws-s3=1.36.1,@aws-cdk/cloud-assembly-schema=1.36.1,@aws-cdk/core=1.36.1,@aws-cdk/cx-api=1.36.1,@aws-cdk/region-info=1.36.1,jsii-runtime=Python/3.7.3

And, let’s DEPLOY:

(.env) ➜  iac git:(dev) ✗ cdk deploy
This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening).
Please confirm you intend to make the following modifications:

IAM Statement Changes
┌───┬─────────────────────────────────┬────────┬──────────────┬───────────┬───────────┐
│   │ Resource                        │ Effect │ Action       │ Principal │ Condition │
├───┼─────────────────────────────────┼────────┼──────────────┼───────────┼───────────┤
│ + │ ${www.unicornpursuit.com.Arn}/* │ Allow  │ s3:GetObject │ *         │           │
└───┴─────────────────────────────────┴────────┴──────────────┴───────────┴───────────┘
(NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299)

Do you wish to deploy these changes (y/n)? y
UnicornIaC: deploying...
UnicornIaC: creating CloudFormation changeset...
 0/4 | 11:12:10 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack | UnicornIaC User Initiated
 0/4 | 11:12:13 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata
 0/4 | 11:12:13 PM | CREATE_IN_PROGRESS   | AWS::S3::Bucket       | www.unicornpursuit.com (wwwunicornpursuitcom6F23076E)
 0/4 | 11:12:14 PM | CREATE_IN_PROGRESS   | AWS::S3::Bucket       | www.unicornpursuit.com (wwwunicornpursuitcom6F23076E) Resource creation Initiated
 0/4 | 11:12:14 PM | CREATE_IN_PROGRESS   | AWS::CDK::Metadata    | CDKMetadata Resource creation Initiated
 1/4 | 11:12:14 PM | CREATE_COMPLETE      | AWS::CDK::Metadata    | CDKMetadata
 2/4 | 11:12:35 PM | CREATE_COMPLETE      | AWS::S3::Bucket       | www.unicornpursuit.com (wwwunicornpursuitcom6F23076E)
 2/4 | 11:12:36 PM | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy | www.unicornpursuit.com/Policy (wwwunicornpursuitcomPolicy46AB873C)
 2/4 | 11:12:38 PM | CREATE_IN_PROGRESS   | AWS::S3::BucketPolicy | www.unicornpursuit.com/Policy (wwwunicornpursuitcomPolicy46AB873C) Resource creation Initiated
 3/4 | 11:12:38 PM | CREATE_COMPLETE      | AWS::S3::BucketPolicy | www.unicornpursuit.com/Policy (wwwunicornpursuitcomPolicy46AB873C)
 4/4 | 11:12:39 PM | CREATE_COMPLETE      | AWS::CloudFormation::Stack | UnicornIaC

 ✅  UnicornIaC

Stack ARN:
arn:aws:cloudformation:eu-west-1:[HIDDEN]]:stack/UnicornIaC/3d9a9df0-8b27-11ea-9900-022b3a528618
(.env) ➜  iac git:(dev) ✗

Ok, let’s check out if the bucket is created:

(.env) ➜  iac git:(dev) ✗
(.env) ➜  iac git:(dev) ✗ aws s3 ls | grep www.
2020-04-30 23:12:38 www.unicornpursuit.com
(.env) ➜  iac git:(dev) ✗

Cool, it’s there!

Let’s now imagine we remove the removal_policy=core.RemovalPolicy.DESTROY from the Bucket, cause we changed our mind and we don’t want to risk deleting our Bucket if someone deletes the CD Stack. Here is how that would look:

Retention

For more S3 examples, check out the S3 Construct Library, as it’s sometimes easier to copy from the example, then “decrypt” the official API documentation.

Deep Dive

S3 Encryption

There are 2 types of Encryption:

In Transit (SSL/TLS)
At REST, which can use one of 3 kinds of Server Side Encryption:
S3 Managed Keys, SSE-S3, which uses AWS256
SSE-KMS (Key Management Service)
SSE-C, where you manage your own keys (you create, rotate…)
Or Client Side Encryption, where you would encrypt files before taking them to S3.

IMPORTANT: To ENFORCE encryption, you need to configure the Bucket Policy to deny any S3 PUT request which does not include “x-ams-server-side-encryption” parameter in the request header.

Bucket Policies are created using the Policy Generator (available under S3 Bucket - Permissions), and you would basically generate this policy in JSON yourself.

CORS: Cross Origin Resource Sharing Use case: Code in one S3 Bucket using resources from another S3 bucket. For example, index.html on a web page using a picture from another bucket.

It’s configured within the Bucket Permissions, and CORS configuration.

Storage Gateway

Storage Gateway is a service that connects an on-premises software appliance with cloud-based storage to provide seamless and secure integration between your on-premises IT environment and the AWS storage infrastructure in the cloud.

AWS Storage gateway can be used to connect on premises hardware to AWS, used for DR. There are:

Gateway CACHED, when using S3 as STORAGE solution
Gateway STORED, for Backup and DR

Where to find more info

Relevant links:

Next:DynamoDB as NoSQL Database

Feedback

Was this page helpful?

Awesome! Please tell us how we can improve.

Sorry to hear that. Please tell us how we can improve.

Last modified June 8, 2020: AWS Diagrams Added (c98aff0)