AWS Account creation, fully automated - ascode style

An introduction into creating accounts in AWS, in code, fully automated with Terraform.

We like AWS. In fact, it's our preferred public cloud provider. Why? Because AWS understands the need to automate things. We use Terraform for automating our cloud infrastructures, and Hashicorp usually has first day support for new services coming out of Amazon. If Terraform doesn't have support for something, you can always fall back to the AWS CLI (which can be called with Terraform using a null_resource). I was automating the AWS cost & usage report the other day, and found out there is no Athena support in the "aws_cur_report_definition" resource, so I had to fall back to the CLI to create the report definition, like so:

resource "null_resource" "create_cur_report" {
  provisioner "local-exec" {
    command = <<EOF
      AWS_DEFAULT_REGION="us-east-1" aws cur put-report-definition --report-definition \
      AdditionalArtifacts=[\"ATHENA\"],AdditionalSchemaElements=[\"RESOURCES\"],\
      Compression=\"Parquet\",Format=\"Parquet\",RefreshClosedReports=true,\
      ReportName=${var.report_name},ReportVersioning=\"OVERWRITE_REPORT\",\
      S3Bucket=${var.cf_bucket_name},S3Prefix=${var.s3_prefix},S3Region=${var.s3_region},\
      TimeUnit=\"${var.time_unit}\"
      EOF
  }
}

The nice thing about using the null_resource is that you can use the variables that you have configured for your other resources, and use the output from other resources in your Terraform template.

Anyway, I digress. Creating the cost & usage report is a good subject for another blog post. This post is about account creation automation.

AWS Organizations

We use AWS Organizations to structure our AWS accounts. For every new client, we roll out a basic setup that looks like this:

"AWS Organizations"

By default, we create the following accounts:

 

This setup is sufficient to get a customer started in AWS. Because of AWS Organizations, consolidated billing is in place. Tagging is enforced so that resources (and therefore, costs) are identifiable by customer and/or environment. The Terraform code for this setup:

data "aws_organizations_organization" "root" {}

locals {
  dtap_accounts = {
    dev = "${var.email_group}+-${var.client_short_name}-dev@${var.email_domain}"
    tst = "${var.email_group}+-${var.client_short_name}-tst@${var.email_domain}"
    acc = "${var.email_group}+-${var.client_short_name}-acc@${var.email_domain}"
    prd = "${var.email_group}+-${var.client_short_name}-prd@${var.email_domain}"
  }
  core_accounts = {
    log = "${var.email_group}+-${var.client_short_name}-log@${var.email_domain}"
    svc = "${var.email_group}+-${var.client_short_name}-svc@${var.email_domain}"
  }
  sbx_accounts = {
    sbx = "${var.email_group}+-${var.client_short_name}-sbx@${var.email_domain}"
  }
}

resource "aws_organizations_organizational_unit" "customer_root" {
  count = var.has_organizations ? 1 : 0

  name      = var.client_name
  parent_id = data.aws_organizations_organization.root.roots.0.id
}

resource "aws_organizations_organizational_unit" "customer_dtap" {
  count = var.has_organizations ? 1 : 0

  name      = "${var.client_name}-environments"
  parent_id = aws_organizations_organizational_unit.customer_root.0.id
}

resource "aws_organizations_organizational_unit" "customer_sandbox" {
  count = var.has_organizations ? 1 : 0

  name      = "${var.client_name}-sandbox"
  parent_id = aws_organizations_organizational_unit.customer_root.0.id
}


resource "aws_organizations_organizational_unit" "customer_core" {
  count = var.has_organizations ? 1 : 0

  name      = "${var.client_name}-core"
  parent_id = aws_organizations_organizational_unit.customer_root.0.id
}

resource "aws_organizations_account" "dtap" {
  for_each = var.has_organizations ? local.dtap_accounts : {}

  name      = "${var.client_short_name}-${each.key}"
  email     = each.value
  parent_id = aws_organizations_organizational_unit.customer_dtap.0.id
}

resource "aws_organizations_account" "core" {
  for_each = var.has_organizations ? local.core_accounts : {}

  name      = "${var.client_short_name}-${each.key}"
  email     = each.value
  parent_id = aws_organizations_organizational_unit.customer_core.0.id
}

resource "aws_organizations_account" "sandbox" {
  for_each = var.has_organizations ? local.sbx_accounts : {}

  name      = "${var.client_short_name}-${each.key}"
  email     = each.value
  parent_id = aws_organizations_organizational_unit.customer_sandbox.0.id
}

This code is part of a module that we created for anything client-related. Creation is done when the variable has_organizations is set to true. We did this so we can exclude this from testing the module, as creating AWS accounts is easy but removing them is not. When you remove the AWS account from Terraform, Terraform removes it from the state but does not actually delete the account - this must be done manually and can only be done after at least 7 days. When we test a module, we create and destroy the resources in a sandbox environment to make sure it works - doing this for AWS accounts would mean it would become a mess real soon with leftover accounts. Using the has_organizations variable gives the added benefit that we can set this to false when a customer already has AWS Organizations in place, for example.

The outputs that are defined look like this:

output "customer_dtap_accounts_map" {
  value = var.has_organizations ? [for environment, accountid in zipmap(keys(local.dtap_accounts), values(aws_organizations_account.dtap)[*]["id"]) : { "environment" = environment, "account_id" = accountid }] : null
}

output "customer_core_accounts_map" {
  value = var.has_organizations ? [for environment, accountid in zipmap(keys(local.core_accounts), values(aws_organizations_account.core)[*]["id"]) : { "environment" = environment, "account_id" = accountid }] : null
}

output "customer_sandbox_accounts_map" {
  value = var.has_organizations ? [for environment, accountid in zipmap(keys(local.sbx_accounts), values(aws_organizations_account.sandbox)[*]["id"]) : { "environment" = environment, "account_id" = accountid }] : null
}

output "customer_root_ou_id" {
  value = var.has_organizations ? aws_organizations_organizational_unit.customer_root.0.id : ""
}

output "customer_dtap_ou_id" {
  value = var.has_organizations ? aws_organizations_organizational_unit.customer_dtap.0.id : ""
}

output "customer_sandbox_ou_id" {
  value = var.has_organizations ? aws_organizations_organizational_unit.customer_sandbox.0.id : ""
}

output "customer_core_ou_id" {
  value = var.has_organizations ? aws_organizations_organizational_unit.customer_core.0.id : ""
}

output "all_account_ids_map" {
  value = var.has_organizations ? [for environment, accountid in merge(zipmap(keys(local.dtap_accounts), values(aws_organizations_account.dtap)[*]["id"]), zipmap(keys(local.core_accounts), values(aws_organizations_account.core)[*]["id"]), zipmap(keys(local.sbx_accounts), values(aws_organizations_account.sandbox)[*]["id"])) : { "environment" = environment, "account_id" = accountid }] : null
}

What you see here is the Organizational Unit id's are configured as output so they can be used as parent_id when creating additional accounts. The account id's are configured as output so they can be used to automatically create the AWS CLI config, but I will get into that later on. For now, I will break up the export of the account id's into smaller pieces to explain what is done.

value = var.has_organizations ? - the value is only exported if the variable has_organizations is set to true (which is the default).
[for environment, accountid in zipmap(keys(local.dtap_accounts), values(aws_organizations_account.dtap)[*]["id"]) : { "environment" = environment, "account_id" = accountid }] - this links the environment name that is defined in locals to the account id that is created.
zipmap(keys(local.dtap_accounts), values(aws_organizations_account.dtap)[*]["id"]) - this creates a map of environment with account_id. The result for the core accounts looks like this:

{
    "svc" = "123456789",
    "log" = "123456788",
}

Then, using the for loop [for environment, accountid in zipmap(...) : { "environment" = environment, "account_id" = accountid }] a map is created that looks like this:

{
    "account_id" = "123456788"
    "environment" = "log"
  },
  {
    "account_id" = "123456789"
    "environment" = "svc"
  }

All the outputs are merged together to get a nice overview of all the provisioned accounts.

Additional AWS accounts

So, we have the default setup in place. It could be possible that a client wants an additional account, for example an additional sandbox account or an isolated account for a high-secure application. We want the creation of this account to be automated, incorporated into AWS Organizations and set up with some default roles and resources that are required in each account.

First, the account is created. It is possible to add the account to an existing Organizational Unit for AWS Organizations, like an additional sandbox account that goes into the sandbox OU. It is also possible to add an additional OU in which we then create the account. The code for the module looks like this:

resource "aws_organizations_account" "additional" {
  count = var.create_account ? 1 : 0

  name      = "${var.customer_tla}-${var.aws_account_name}"
  email     = var.aws_account_email_address
  parent_id = var.aws_account_parent_id
}


resource "aws_organizations_organizational_unit" "ou" {
  count = var.create_ou ? 1 : 0

  name      = var.ou_name
  parent_id = var.aws_ou_parent_id
}

Not too difficult, right? When you want to create an OU, create_account is set to false and create_ou is set to true, and vice versa.

Using this in a module, with the example of creating an additional OU, looks like this:

module "new_aws_ou" {
    source                    = "modules/aws/account"
    create_account            = false
    create_ou                 = true
    ou_name                   = "custom"
    aws_ou_parent_id          = module.client_customer_name.customer_root_ou_id
}

module "new_aws_account" {
    source                    = "modules/aws/account"
    customer_tla              = var.customer_tla
    aws_account_name          = var.additional_account_name
    aws_account_email_address = "${var.email_group}+${var.customer_tla}-{var.environment}@${var.email_domain}"
    aws_account_parent_id     = module.new_aws_ou.organizational_unit_id
}

This creates a new OU named custom with a new account created in this OU. Now we can create the AWS CLI config file with the output of all the resources that we created:

# can use specific config file with export AWS_CONFIG_FILE=/var/tmp/config-${var.customer_name} in CI/CD pipeline

resource "null_resource" "aws_config" {
  depends_on = [module.client_customer_name, module.new_aws_account]
  provisioner local-exec {
    working_dir = "/keybase/team/customer/customerbot/home/.aws"
    command     = <<EOT
      echo "##########################################################################" >| config-${var.customer_name}
      echo "# AWS config file for customer ${var.customer_name}" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "[profile ${var.customer_name}-root-user]\nregion=eu-west-1\noutput=json\n" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "# DTAP Accounts" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      %{for account in module.client_customer_name.customer_dtap_accounts_map}
        echo "\n[profile ${var.customer_name}-${account.environment}]\nrole_arn=arn:aws:iam::${account.account_id}:role/AccountAssumeRole\nsource_profile=${var.customer_name}-root-user\nregion=eu-west-1\noutput=json\n" >> config-${var.customer_name}
      %{endfor~}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "# Core Accounts" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      %{for account in module.client_${var.customer_name}.customer_core_accounts_map}
        echo "\n[profile ${var.customer_name}-${account.environment}]\nrole_arn=arn:aws:iam::${account.account_id}:role/AccountAssumeRole\nsource_profile=${var.customer_name}-root-user\nregion=eu-west-1\noutput=json\n" >> config-${var.customer_name}
      %{endfor~}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "# Sandbox Accounts" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      %{for account in module.client_${var.customer_name}.customer_sandbox_accounts_map}
        echo "\n[profile ${var.customer_name}-${account.environment}]\nrole_arn=arn:aws:iam::${account.account_id}:role/AccountAssumeRole\nsource_profile=${var.customer_name}-root-user\nregion=eu-west-1\noutput=json\n" >> config-${var.customer_name}
      %{endfor~}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "# Other Accounts" >> config-${var.customer_name}
      echo "##########################################################################" >> config-${var.customer_name}
      echo "\n[profile ${var.customer_name}-${module.new_aws_account.aws_account_id["environment"]}]\nrole_arn=arn:aws:iam::${module.new_aws_account.aws_account_id["account_id"]}:role/AccountAssumeRole\nsource_profile=${var.customer_name}-root-user\nregion=eu-west-1\noutput=json\n" >> config-${var.customer_name}
     EOT
  }
}

As you can see, we use Keybase with a bot account in the CI/CD pipeline that uses an AWS config file. I hate having to update this file every time a new account is created, so by using this resource the file is updated every time a new account is created! The (partial) result looks like this:

##########################################################################
# AWS config file for customer CUSTOMER
##########################################################################
[profile customer-root-user]
region=eu-west-1
output=json

##########################################################################
# DTAP Accounts
##########################################################################

[profile customer-acc]
role_arn=arn:aws:iam::123456787:role/AccountAssumeRole
source_profile=customer-root-user
region=eu-west-1
output=json


[profile customer-dev]
role_arn=arn:aws:iam::123456786:role/AccountAssumeRole
source_profile=customer-root-user
region=eu-west-1
output=json
...

Once the file is updated, it can be used to provision resources in the new account. This is done by configuring a module that contains basic resources that we want deployed in all accounts. For example, Cloudtrail must be enabled, some IAM roles must be deployed, centralized logging must be set up and anything else that you can think of that you want to be part of your default AWS account rollout.

First, we configure an AWS provider with an alias:

provider "aws" {
  region  = var.aws_region
  profile = "customer-custom"
  alias   = "custom"

  skip_metadata_api_check = true

  assume_role {
    role_arn = "arn:aws:iam::${module.new_aws_account.aws_account_id["account_id"]}:role/AccountAssumeRole"
  }
}

Then we can provision the resources in the new account using this provider:

module "new_aws_account_baseline" {
  source                 = "../modules/aws-account-baseline"
  aws_account_id         = var.aws_account_id
  customer_tla           = var.customer_tla
  environment            = var.environment
  tags                   = local.tags
  customer_name          = var.customer_name
  cloudtrail_bucket_name = module.centralized_logging.cloud_trail_bucket_id
  cloudtrail_kms_key_id  = module.centralized_logging.log.outputs.cloud_trail_kms_arn
  cloudtrail_s3_prefix   = data.aws_caller_identity.current.account_id
  providers = {
    aws = aws.custom
  }
}

Each client gets it's own Terraform template using a set of shared modules, that defines the base accounts and any possible additional accounts. The client module can be extended with whatever services or resources you want to be part of a new AWS account. That way, all accounts are provisioned in exactly the same way. You could also create different contexts with IAM permissions boundaries, allowing only certain services to be provisioned in the account. You can then apply those contexts to an account based on a variable. The possibilities are nearly endless...