Hosting this Blog with AWS and Pulumi

Hello, dear reader. Welcome to the first post on my blog. In the tradition of self-reference that we find ourselves,¹ I'm going to write my first blog post about the infrastructure behind this blog. My blog is hosted on AWS and is composed of two S3 buckets, a route53 zone certified via acm and all appropriate records, and Cloudfront Distributions to cache and encrypt. If you are familiar with AWS, you can probably already picture the resource diagram. If you can't imagine the diagram, I have provided a less imagery version:

I'm going to walk through each component, why it exists and the code that describes it. I will assume that you are familiar with programming, and comfortable with TypeScript syntax. I will not assume specialized knowledge of AWS, networking, Pulumi or any other technology used.

So how are you creating this stuff anyway

Even a simple static website requires multiple pieces of infrastructure wired together to appear on your screen. Like most software projects, errors are easy to make and poorly tolerated. I don't want to manually click through the AWS console. I don't trust myself to get it right. I certainly don't trust myself to get it right each time. Because of this, I'm using an Infrastructure as Code (IaC) tool called Pulumi² to provision everything. I declare³ my infrastructure in typescript, and then Pulumi makes the changes to AWS on my behalf.

The rest of this post will walk through the TypeScript code I wrote to get this site stood up. The great thing about IaC tools, is that they fully and unambiguously describe their desired state. That means that this will be a walkthrough not only of my TypeScript code, but also the infrastructure it describes.

You are welcome to follow along at home.

A new project and its secrets

I started by setting up a new project and teaching it how to talk to AWS. In Pulumi, a project is a description of some infrastructure, while a stack is a deployment of that infrastructure. If you are from a Object Oriented background, you can think of this as a class and an instance.⁴ The file that defines a project in Pulumi is called Pulumi.yaml, and it looks like this:

name: iwahbe
runtime: nodejs
description: |
  The infra to host iwahbe.com & www.iwahbe.com.

This file should be self explanatory: There is a project called iwahbe, and it is written in Typescript (which runs on NodeJS). To deploy our infrastructure, we also need a stack. The easiest way to do this is with the Pulumi CLI. To deploy and configure my main stack, I ran

pulumi stack select --create dev                     # Create a new stack, and set it as the current stack
pulumi config set domain iwahbe.com                  # The domain of my site
pulumi config set root-dir ../www/public             # This is where my HTML goes
pulumi config set --path \                           # Disable ambient configuration
  'pulumi:disable-default-providers[0]' '*'          # '*' means for all providers

I want Pulumi to be able to access my AWS account, and I don't want it to use ambient credentials. I added my AWS credentials to the stack config as well, but marked them as secret.

echo $AWS_ACCESS_KEY | pulumi config set --secret access-key
echo $AWS_SECRET_KEY | pulumi config set --secret secret-key

This is it for configuring the project and stack, but we still want a way to consume this definition in the code itself. In a new file: utils.ts, I wrote a simple class to abstract creating providers with the appropriate config:

class Project {
  private config: pulumi.Config;

  // The root of the static content to be uploaded.
  public uploadRoot: string;
  // The domain of the site being deployed.
  public domain: string;

  constructor() {
    this.config = new pulumi.Config();
    this.uploadRoot = this.config.require("root-dir");
    this.domain = this.config.require("domain");
  }

  // Create a new AWS provider configured with project appropriate credentials and tags.
  awsProvider(name: string, args: aws.ProviderArgs): aws.Provider {
    return new aws.Provider(name, {
      accessKey: this.config.requireSecret("access-key"),
      secretKey: this.config.requireSecret("secret-key"),
      defaultTags: {
        tags: {
          project: this.domain,
        },
      },
      ...args,
    });
  }
}

This class serves two purposes:

1. It abstracts configuration handling and credentials out of my infrastructure code. This allows me to separate the shape of my infrastructure from details of my AWS setup.
2. It projects my configuration into the Typescript type system.

Because there will only ever be one Project in my project, I export a project variable instead of exporting the Project class. This gives us a singleton Project.

export const project = new Project();

I can then import project into the Typescript project root (index.ts).

import * as aws from "@pulumi/aws";
import { bucketPolicy, project, uploadFiles } from "./utils";

const domain = project.domain;
const wwwDomain = `www.${domain}`;

const provider = project.awsProvider("aws", { region: "us-west-2" });

I'll explain what bucketPolicy and uploadFiles are later when they become relevant.

That's it for configuration, I promise. We can now start actually deploying things.

Starting simple (with S3)

The most basic and most important part of a static website is the static content. The bucket that holds my content is called www.iwahbe.com (as required by AWS), and makes use of AWS's S3 static site support. We start by defining a utility function in utils.ts to describe the commonality between our buckets:

export function publicBucketArgs(bucket: string): aws.s3.BucketArgs {
  return {
    bucket,
    acl: "public-read",
    policy: JSON.stringify({
      "Version": "2012-10-17",
      "Statement": [{
        "Sid": "PublicReadGetObject",
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": `arn:aws:s3:::${bucket}/*`,
      }],
    }),
  };
}

bucket defines the AWS name of the bucket, and must correspond to the website the bucket is serving. acl: "public-read" tells AWS that anyone can request read from the bucket.⁵ We will pass this to both our public buckets. First the main content bucket:

const wwwBucket = new aws.s3.Bucket("www-bucket", {
    ...publicBucketArgs(wwwDomain),
    corsRules: [{
        allowedHeaders: ["Authorization", "Content-Length"],
        allowedMethods: ["GET", "POST"],
        allowedOrigins: [`https://${wwwDomain}`],
        maxAgeSeconds: 3000,
    }],
    website: {
      indexDocument: "index.html",
      errorDocument: "404.html",
    },
  }, { provider });

website tells AWS that this bucket is serving a static site, and which document to serve on root access (index.html) or when requesting a document that doesn't exist (404.html).

The second bucket provides a redirect from iwahbe.com to www.iwahbe.com:

const rootBucket = new aws.s3.Bucket("root-bucket", {
    ...publicBucketArgs(domain),
    website: { redirectAllRequestsTo: `https://${wwwDomain}` },
  }, { provider });

redirectAllRequestsTo tells AWS that this bucket is serving as a proxy for a static website, but not serving any content directly.

AWS requires that S3 buckets that serve as static websites have their bucket name match the domain they are serving. This is why we have two different buckets: one bucket to server www.iwahbe.com and another to serve iwahbe.com (even as a simple redirect).

Technically, this is all you need to serve a static website on AWS.⁶ There are some major caveats:

• You don't control the url, so it will start with a prefix like https://my-bucket.s3-us-west-2.amazonaws.com.
• Content served from a raw S3 bucket doesn't carry a SSL certificate.
• Only HTTP is supported. HTTPS is not supported.

The rest of the blog post will add the infrastructure necessary to serve a SSL secured website from a custom URL over HTTPS. That's our goal.

A custom domain in AWS

The most obvious problem with serving directly from S3 buckets is the URL. Credible sites are served from a memorable and custom domain, such as iwahbe.com. It's also a prerequisite to serving a validated site, so we will start there. The abstraction that AWS uses for domains is a Route53 Zone, so we will start by defining a zone:

const rootZone = new aws.route53.Zone("root-zone", {
  name: domain,
}, { provider });

Remember that domain is "iwahbe.com". The zone is for iwahbe.com, our root domain. We can use our root domain to effect subdomains, such as www.iwahbe.com.

The Route53 zone defines a set of name servers, which I pointed my domain registrar towards. To serve authorized traffic from a domain, the domain needs to be validated. AWS provides a validation service: ACM⁷. Getting a ACM certificate declaring that we own a domain has 3 steps in a Pulumi program:

1. Declaring that we want a certificate, and how we want to prove ownership of the domain.
2. Following the instructions to establish ownership.
3. Waiting for AWS to validate that we have followed the steps successfully.

We want the entire process to be done in our pulumi program, so we validate with DNS records on the domain itself. That way, we can use Pulumi to add the DNS records requested by ACM, all within the same pulumi up.

According to AWS, all certificates need to be hosted in the us-east-1 region. In Pulumi's AWS provider, we do this by declaring a new provider with a different region. Just like before, we use our project singleton to help us define the provider:

const awsUsEast = project.awsProvider("awsEast", { region: "us-east-1" });

This is how we declare the certificate:

const sslCert = new aws.acm.Certificate("ssl-certificate", {
  domainName: domain,
  subjectAlternativeNames: [`*.${domain}`],
  validationMethod: "DNS",
}, { provider: awsUsEast, deleteBeforeReplace: true, parent: rootZone });

subjectAlternativeNames allows us to declare that in addition to iwahbe.com, the certificate should apply to any domain that ends with .iwahbe.com. This allows us to apply the certificate to www.iwahbe.com.

validationMethod: "DNS" declares that we will confirm ownership of the domain by adding a DNS record to iwahbe.com. Because we are managing iwahbe.com with a Route53 Zone, this is easy to do:

const certValidationRecord = new aws.route53.Record("ssl-validation-record", {
  name: sslCert.domainValidationOptions[0].resourceRecordName,
  zoneId: rootZone.id,
  type: sslCert.domainValidationOptions[0].resourceRecordType,
  records: [sslCert.domainValidationOptions[0].resourceRecordValue],
  ttl: 10 * 60, // 10 minutes
}, { provider, dependsOn: sslCert, parent: sslCert });

Here we are taking the domain validation options provided by sslCert and feeding them into the record. Since validation needs to wait on AWS seeing the DNS updating and validating the certificate, we also declare a phony resource that waits on the validation to be completed:

const certValidation = new aws.acm.CertificateValidation("ssl-validation", {
  certificateArn: sslCert.arn,
  validationRecordFqdns: [certValidationRecord.fqdn],
}, { provider: awsUsEast, parent: sslCert });

That's all we need to set up our domain in AWS. We can now use setup AWS resources that serve content from iwahbe.com.

Serving HTTPS from our domain

Amazon's S3 can't serve HTTPS requests. Cloudfront Distributions can, and they can serve directly from our existing S3 Buckets. Because we serve content from two buckets, we will need two distributions, one for each bucket. We will then attach A and AAAA records to our domain to direct incoming requests to our distributions.

We encapsulate the idea of a Cloudfront distribution directing to a bucket in a function:

type PartialBy<T, K extends keyof T> = Omit<T, K> & Partial<Pick<T, K>>
type DistributionArgs = PartialBy<aws.cloudfront.DistributionArgs,
  "viewerCertificate" | "origins" | "enabled" | "restrictions">

// Create a Cloudfront Distribution that routes to an s3 Bucket.
function bucketDistribution(
  name: string, source: aws.s3.Bucket, args: DistributionArgs,
): aws.cloudfront.Distribution { ... }

Inside bucketDistribution, we define a aws.cloudfront.Distribution. We define the origin as the passed in source aws.s3.Bucket, configure the Distribution to alias the bucket and set the cert to the cert we defined above: sslCert. Finally, we mix in any arguments that we passed in at the call site.

The Distribution looks like this:

  new aws.cloudfront.Distribution(
    `${name}-s3-distribution`,
    {
      ...args,
      origins: [{
        domainName: source.websiteEndpoint,
        originId: pulumi.interpolate`S3-${source.bucket}`,

        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originProtocolPolicy: "http-only",
          originSslProtocols: ["TLSv1", "TLSv1.1", "TLSv1.2"],
        },
      }],

      enabled: true,
      isIpv6Enabled: true,

      aliases: [source.bucket],

      restrictions: {
        geoRestriction: {
          restrictionType: "none",
        },
      },

      viewerCertificate: {
        acmCertificateArn: sslCert.arn,
        sslSupportMethod: "sni-only",
        minimumProtocolVersion: "TLSv1.1_2016",
      },
    },
    // We depend on `certValidation` to make sure that we don't deploy a Distribution with
    // a cert that hasn't been validated yet.
    { provider, dependsOn: certValidation, parent: source },
  );

To point the bucket's domain to the Distribution, we need to attach Address Records (often abbreviated A Records) to our domain. This tells DNS to resolve [www.]iwahbe.com to the address of our Distribution, instead of targeting the bucket directly. Since IPv6 is enabled, we also need to attach AAAA Records. AAAA Records are the IPv6 equivalent of A Records.

  for (const type of ["A", "AAAA"]) {
    new aws.route53.Record(`${name}-${type.toLowerCase()}`, {
      zoneId: rootZone.zoneId,
      name: source.bucket,
      type,
      aliases: [{
        name: dist.domainName,
        zoneId: dist.hostedZoneId,
        evaluateTargetHealth: false,
      }],
    }, { provider, parent: rootZone });
  }

The entire function looks like this:

// Create a Cloudfront Distribution that routes to an s3 Bucket.  The Distribution will
// front the bucket's domain, aliasing the bucket with HTTPS and IPv6 support.
function bucketDistribution(
  name: string, source: aws.s3.Bucket, args: DistributionArgs,
): aws.cloudfront.Distribution {
  const dist = new aws.cloudfront.Distribution(
    `${name}-s3-distribution`,
    {
      ...args,
      origins: [{
        domainName: source.websiteEndpoint,
        originId: pulumi.interpolate`S3-${source.bucket}`,

        customOriginConfig: {
          httpPort: 80,
          httpsPort: 443,
          originProtocolPolicy: "http-only",
          originSslProtocols: ["TLSv1", "TLSv1.1", "TLSv1.2"],
        },
      }],

      enabled: true,
      isIpv6Enabled: true,

      aliases: [source.bucket],

      restrictions: {
        geoRestriction: {
          restrictionType: "none",
        },
      },

      viewerCertificate: {
        acmCertificateArn: sslCert.arn,
        sslSupportMethod: "sni-only",
        minimumProtocolVersion: "TLSv1.1_2016",
      },
    },
    // We depend on `certValidation` to make sure that we don't deploy a Distribution with
    // a cert that hasn't been validated yet.
    { provider, dependsOn: certValidation, parent: source },
  );

  for (const type of ["A", "AAAA"]) {
    new aws.route53.Record(`${name}-${type.toLowerCase()}`, {
      zoneId: rootZone.zoneId,
      name: source.bucket,
      type,
      aliases: [{
        name: dist.domainName,
        zoneId: dist.hostedZoneId,
        evaluateTargetHealth: false,
      }],
    }, { provider, parent: rootZone });
  }

  return dist;
}

Using our newly defined bucketDistribution, we can easily define our Distributions:

First, we define the bucket for www.iwahbe.com. The only parts we customize are error handling and caching behavior. Most of the configuration is taken care of by bucketDistribution.

bucketDistribution("www", wwwBucket, {
  defaultRootObject: "index.html",

  customErrorResponses: [{
    errorCachingMinTtl: 0,
    errorCode: 404,
    responseCode: 200,
    responsePagePath: "/404.html",
  }],

  defaultCacheBehavior: {
    allowedMethods: ["GET", "HEAD"],
    cachedMethods: ["GET", "HEAD"],
    targetOriginId: `S3-${wwwDomain}`,

    forwardedValues: {
      queryString: false,
      cookies: { forward: "none" },
    },

    viewerProtocolPolicy: "redirect-to-https",
    compress: true,
  },
});

Second, we define the root bucket, customizing the caching behavior and nothing else.

bucketDistribution("root", rootBucket, {
  defaultCacheBehavior: {
    allowedMethods: ["GET", "HEAD"],
    cachedMethods: ["GET", "HEAD"],
    targetOriginId: `S3-.${domain}`,

    forwardedValues: {
      queryString: false,
      cookies: { forward: "none" },
      headers: ["Origin"],
    },

    viewerProtocolPolicy: "allow-all",
  },
});

This is all we need to serve HTTPS content from our buckets. The program is complete.

Conclusion

That's everything I used to host this blog. By combining AWS with Pulumi, I'm able to declare my infrastructure in a clear and expressive language. I am able to deploy all my infrastructure with pulumi up.

If you made it to the end, congratulations! I'm always happy to receive feedback at @iwahbe@hachyderm.io. If you think I did something wrong (or right), please let me know.