visit
In this blog, the Terraform modules we develop will generate the following necessary resources for a functional EKS Cluster. Additionally, we will demonstrate the functionality of the EKS Cluster by installing an nginx
Helm chart.
my-eks-tf
on your machine.my-eks-tf
directory, organize your modules as follows:
modules
directory, which contains the eks
and vpc_and_subnets
modules. These modules should have opinionated defaults set by your Core Platform team, allowing developers to only modify certain values.cluster
directory, which includes the scaffold module. This module invokes the eks
and vpc_and_subnets
modules and provides additional abstraction. It can have hardcoded values, simplifying the parameters that developers need to specify.cluster
module is invoked by the code in my-eks-tf/main.tf
, which is written by your team’s Developer team member.main.tf
contains the actual terraform code, including provider settings and invocations of external APIs to create resources in the cloud provider.variables.tf
declares input variables that can be overridden when invoking the module.outputs.tf
declares variables that can be used by other modules. For example, the vpc
module might output subnet IDs that are used by the eks
module..tfvars
files as input files for your terraform modules. Consider having separate files for each environment, such as dev.tfvars
, stage.tfvars
, and prod.tfvars
.
Although it’s possible to structure the discussed modules into multiple repositories, for simplicity, we kept everything within the same module. However, you can consider separating the modules
directory, the scaffolding modules (e.g., cluster
module) built by your team’s Platform team member, and a separate repository for .tfvars
files and the module that invokes the scaffolding modules.
my-eks-tf/
.
├── README.md
├── cluster
│ ├── README.md
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── modules
│ ├── README.md
│ ├── eks
│ │ ├── README.md
│ │ ├── main.tf
│ │ ├── outputs.tf
│ │ └── variables.tf
│ └── vpc_and_subnets
│ ├── README.md
│ ├── main.tf
│ ├── outputs.tf
│ └── variables.tf
├── main.tf
├── outputs.tf
├── sample.tfvars
└── variables.tf
In our opinionated vpc_and_subnets
module, we prompt the user for parameters like VPC name, CIDR block, and whether they want to create an internet gateway or NAT gateways. We place the module invocation in main.tf
and define variables in variables.tf
. The outputs.tf
file contains the outputs required for other modules.
It is worth noting that we specify the module source as version 5.0.0
of the open-source module. This is a good practice as it locks down the module version, preventing unexpected behavior due to compatibility issues in newer versions. We create the VPC in the first three availability zones returned by the AWS Terraform API’s aws_availability_zones
data resource. Additionally, we reference local variables using local.private_subnets
and local.public_subnets
within the module.
In the locals
block, we leverage the Terraform function to generate 3 public and 3 private subnets.
Please note that the provided code snippet may have been abbreviated for brevity. You can find the complete working code for main.tf
.
data "aws_availability_zones" "available" {}
locals {
newbits = 8
netcount = 6
all_subnets = [for i in range(local.netcount) : cidrsubnet(var.vpc_cidr, local.newbits, i)]
public_subnets = slice(local.all_subnets, 0, 3)
private_subnets = slice(local.all_subnets, 3, 6)
}
# vpc module to create vpc, subnets, NATs, IGW etc..
module "vpc_and_subnets" {
# invoke public vpc module
source = "terraform-aws-modules/vpc/aws"
version = "5.0.0"
# vpc name
name = var.name
# availability zones
azs = slice(data.aws_availability_zones.available.names, 0, 3)
# vpc cidr
cidr = var.vpc_cidr
# public and private subnets
private_subnets = local.private_subnets
public_subnets = local.public_subnets
# create nat gateways
enable_nat_gateway = var.enable_nat_gateway
single_nat_gateway = var.single_nat_gateway
one_nat_gateway_per_az = var.one_nat_gateway_per_az
# enable dns hostnames and support
enable_dns_hostnames = var.enable_dns_hostnames
enable_dns_support = var.enable_dns_support
# tags for public, private subnets and vpc
tags = var.tags
public_subnet_tags = var.additional_public_subnet_tags
private_subnet_tags = var.additional_private_subnet_tags
# create internet gateway
create_igw = var.create_igw
instance_tenancy = var.instance_tenancy
}
Sample variables.tf
file looks as below, for working code please refer .
variable "name" {
type = string
description = "name of the vpc"
}
variable "vpc_cidr" {
type = string
description = <<EOT
vpc cidr
e.g. 10.0.0.0/16
EOT
}
variable "enable_nat_gateway" {
description = "Should be true if you want to provision NAT Gateways for each of your private networks"
type = bool
default = true
}
variable "single_nat_gateway" {
description = "Should be true if you want to provision a single shared NAT Gateway across all of your private networks"
type = bool
default = false
}
variable "one_nat_gateway_per_az" {
description = "Should be true if you want only one NAT Gateway per availability zone."
type = bool
default = true
}
variable "enable_dns_hostnames" {
description = "Should be true to enable DNS hostnames in the VPC"
type = bool
default = true
}
variable "enable_dns_support" {
description = "Should be true to enable DNS support in the VPC"
type = bool
default = true
}
variable "tags" {
description = "A mapping of tags to assign to all resources"
type = map(string)
default = {}
}
variable "additional_public_subnet_tags" {
description = "Additional tags for the public subnets"
type = map(string)
default = {}
}
variable "additional_private_subnet_tags" {
description = "Additional tags for the private subnets"
type = map(string)
default = {}
}
variable "create_igw" {
description = "Controls if an Internet Gateway is created for public subnets and the related routes that connect them."
type = bool
default = true
}
variable "instance_tenancy" {
description = "A tenancy option for instances launched into the VPC"
type = string
default = "default"
}
Sample outputs.tf
file looks as below, for working code please refer . Below is an example on how to declare an output variable retrieving from output of the remote invoked module. Here it’s retrieving outputs from vpc_and_subnets
module which invokes module.
output "vpc_id" {
description = "The ID of the VPC"
value = module.vpc_and_subnets.vpc_id
}
output "private_subnets" {
description = "List of IDs of private subnets"
value = module.vpc_and_subnets.private_subnets
}
output "public_subnets" {
description = "List of IDs of public subnets"
value = module.vpc_and_subnets.public_subnets
}
output "public_route_table_ids" {
description = "List of IDs of public route tables"
value = module.vpc_and_subnets.public_route_table_ids
}
output "private_route_table_ids" {
description = "List of IDs of private route tables"
value = module.vpc_and_subnets.private_route_table_ids
}
output "nat_ids" {
description = "List of allocation ID of Elastic IPs created for AWS NAT Gateway"
value = module.vpc_and_subnets.nat_ids
}
output "nat_public_ips" {
description = "List of public Elastic IPs created for AWS NAT Gateway"
value = module.vpc_and_subnets.nat_public_ips
}
output "natgw_ids" {
description = "List of NAT Gateway IDs"
value = module.vpc_and_subnets.natgw_ids
}
output "igw_id" {
description = "The ID of the Internet Gateway"
value = module.vpc_and_subnets.igw_id
}
In our opinionated eks
module, we prompt the user for parameters like VPC id, subnet ids for EKS, subnet ids for eks node groups, and EKS cluster name. By default. the module creates EKS cluster with k8s version 1.27
but you can override that as well. By default, this module creates an EKS managed worker named worker
but this configuration can be overridden. We place the module invocation in main.tf
and define variables in variables.tf
. The outputs.tf
file contains the outputs required for other modules.
It is worth noting that we specify the module source as version 19.5.3
of the open-source module. This is a good practice as it locks down the module version, preventing unexpected behavior due to compatibility issues in newer versions. We create the EKS Cluster in the provided subnets and EKS Managed node groups in the provided subnet. And by default this module creates a public and private endpoint for EKS Cluster, enables by creating OIDC provider and installs coredns
, vpc-cni
and kube-proxy
addons.
Please note that the provided code snippet may have been abbreviated for brevity. You can find the working code for main.tf
.
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "19.15.3"
# eks cluster name and version
cluster_name = var.eks_cluster_name
cluster_version = var.k8s_version
# vpc id where the eks cluster security group needs to be created
vpc_id = var.vpc_id
# subnets where the eks cluster needs to be created
control_plane_subnet_ids = var.control_plane_subnet_ids
# to enable public and private access for eks cluster endpoint
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
# create an OpenID Connect Provider for EKS to enable IRSA
enable_irsa = true
# install eks managed addons
# more details are here - //docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html
cluster_addons = {
# extensible DNS server that can serve as the Kubernetes cluster DNS
coredns = {
preserve = true
most_recent = true
}
# maintains network rules on each Amazon EC2 node. It enables network communication to your Pods
kube-proxy = {
most_recent = true
}
# a Kubernetes container network interface (CNI) plugin that provides native VPC networking for your cluster
vpc-cni = {
most_recent = true
}
}
# subnets where the eks node groups needs to be created
subnet_ids = var.eks_node_groups_subnet_ids
# eks managed node group named worker
eks_managed_node_groups = var.workers_config
}
Sample variables.tf
file looks as below, for working code please refer .
variable "eks_cluster_name" {
type = string
description = "eks cluster name"
}
variable "k8s_version" {
type = string
description = "kubernetes version"
default = "1.27"
}
variable "control_plane_subnet_ids" {
type = list(string)
description = "subnet ids where the eks cluster should be created"
}
variable "eks_node_groups_subnet_ids" {
type = list(string)
description = "subnet ids where the eks node groups needs to be created"
}
variable "vpc_id" {
type = string
description = "vpc id where the cluster security group needs to be created"
}
variable "workers_config" {
type = map(any)
description = "workers config"
default = {
worker = {
min_size = 1
max_size = 2
desired_size = 1
instance_types = ["t3.large"]
capacity_type = "SPOT"
}
}
}
Sample outputs.tf
file looks as below, for working code please refer . Below is an example on how to declare an output variable retrieving from output of the remote invoked module. Here it’s retrieving outputs from eks
module which invokes module.
output "cluster_arn" {
description = "The Amazon Resource Name (ARN) of the cluster"
value = module.eks.cluster_arn
}
output "cluster_certificate_authority_data" {
description = "Base64 encoded certificate data required to communicate with the cluster"
value = module.eks.cluster_certificate_authority_data
}
output "cluster_endpoint" {
description = "Endpoint for your Kubernetes API server"
value = module.eks.cluster_endpoint
}
output "cluster_oidc_issuer_url" {
description = "The URL on the EKS cluster for the OpenID Connect identity provider"
value = module.eks.cluster_oidc_issuer_url
}
output "oidc_provider" {
description = "The OpenID Connect identity provider (issuer URL without leading `//`)"
value = module.eks.oidc_provider
}
output "oidc_provider_arn" {
description = "The ARN of the OIDC Provider"
value = module.eks.oidc_provider_arn
}
When working with Terraform modules, it is considered a best practice to specify the version of the AWS provider you are using. By explicitly defining the version, you ensure that the module is built and tested consistently, mitigating the risk of compatibility issues. If the version is not specified, Terraform will automatically retrieve the latest version, which may lead to unexpected behavior and potential compatibility problems. In this example I am setting it to current latest version 5.6.2
() at the time of writing this blog.
Copy paste the following code block to eks
and vpc_and_subnet
module’s main.tf
in modules/
directory.
# setup aws terraform provider version to be used
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "5.6.2"
}
}
}
Below is the main.tf
of the module, which is supposed to create an EKS Cluster in private VPC and a EKS Node group in private VPC.
You will observe that both vpc_with_subnets
and eks_with_node_group
modules refer to and modules we built in sections above. You will also observe that eks_with_node_group
module is using vpc_id
and private_subnets
output values from vpc_with_subnets
module. This creates a dependency order, such that, EKS module invocation will wait until VPC and Subnets are created.
# invoking vpc and subnets modules
module "vpc_with_subnets" {
# invoke vpc_and_subnets module under modules directory
source = "../modules/vpc_and_subnets"
# passing the required parameters
name = var.vpc_name
vpc_cidr = var.vpc_cidr
}
# invoking eks module to eks cluster and node group
module "eks_with_node_group" {
# invoke eks module under modules directory
source = "../modules/eks"
# passing the required parameters
eks_cluster_name = var.eks_cluster_name
k8s_version = var.k8s_version
# pass vpc and subnet details from vpc_with_subnets module
vpc_id = module.vpc_with_subnets.vpc_id
eks_node_groups_subnet_ids = module.vpc_with_subnets.private_subnets
control_plane_subnet_ids = module.vpc_with_subnets.private_subnets
}
Since this API is tailored for a specific team, the variables.tf
file will have fewer parameters compared to the actual module. It will either use default values from the modules or set its own defaults. In our example, users of the cluster
module only need to provide vpc_name
, vpc_cidr
, eks_cluster_name
, and k8s_version
. The vpc_id
and private_subnets
are obtained from the VPC module’s output. Even vpc_name
, vpc_cidr
, and k8s_version
have default values. In the next section, we will demonstrate how invoking this module is as simple as specifying the eks_cluster_name
to create our EKS module.
The variables.tf
file may have been abbreviated for brevity, please refer the working code .
variable "vpc_name" {
type = string
description = "name of the vpc to be created"
default = "platformwale"
}
variable "vpc_cidr" {
type = string
description = "vpc cidr block to be used"
default = "10.0.0.0/16"
}
variable "eks_cluster_name" {
type = string
description = "eks cluster name"
}
variable "k8s_version" {
type = string
description = "kubernetes version"
default = "1.27"
}
This module will simply output vpc detatils and eks cluster details which may be useful for the developer. The following outputs.tf
may have been abbreviated for brevity, please refer the working code .
output "vpc_id" {
description = "The ID of the VPC"
value = module.vpc_with_subnets.vpc_id
}
output "private_subnets" {
description = "List of IDs of private subnets"
value = module.vpc_with_subnets.private_subnets
}
output "public_subnets" {
description = "List of IDs of public subnets"
value = module.vpc_with_subnets.public_subnets
}
output "cluster_certificate_authority_data" {
description = "Base64 encoded certificate data required to communicate with the cluster"
value = module.eks_with_node_group.cluster_certificate_authority_data
}
output "cluster_endpoint" {
description = "Endpoint for your Kubernetes API server"
value = module.eks_with_node_group.cluster_endpoint
}
output "cluster_oidc_issuer_url" {
description = "The URL on the EKS cluster for the OpenID Connect identity provider"
value = module.eks_with_node_group.cluster_oidc_issuer_url
}
The following main.tf
may have been abbreviated for brevity, please refer the working code .
# to use s3 backend
# s3 bucket is configured at command line
terraform {
backend "s3" {}
}
# setup terraform aws provider to create resources
provider "aws" {
region = var.region
}
# invoke cluster module which creates vpc, subnets and eks cluter
module "cluster" {
source = "./cluster"
eks_cluster_name = var.cluster_name
}
In the variables.tf
file, only the EKS cluster name needs to be provided by the user. Other values can be overridden if desired. During invocation, you will observe that we only pass the EKS cluster name. This is because the cluster module only requires the cluster name as a parameter, while all other parameters have default values.
The variables.tf
below may have been abbreviated for brevity, please refer the working code .
variable "region" {
type = string
description = "aws region where the resources are being created"
}
variable "vpc_name" {
type = string
description = "name of the vpc to be created"
default = "platformwale"
}
variable "vpc_cidr" {
type = string
description = "vpc cidr block to be used"
default = "10.0.0.0/16"
}
variable "cluster_name" {
type = string
description = "eks cluster name"
default = "platformwale"
}
variable "k8s_version" {
type = string
description = "k8s version"
default = "1.27"
}
The outputs.tf will only output the eks details as that may only be the details the developer might be interested in. The outputs.tf
below may have been abbreviated for brevity, please refer the working code .
output "cluster_certificate_authority_data" {
description = "Base64 encoded certificate data required to communicate with the cluster"
value = module.cluster.cluster_certificate_authority_data
}
output "cluster_endpoint" {
description = "Endpoint for your Kubernetes API server"
value = module.cluster.cluster_endpoint
}
output "cluster_oidc_issuer_url" {
description = "The URL on the EKS cluster for the OpenID Connect identity provider"
value = module.cluster.cluster_oidc_issuer_url
}
Finally, we must create the .tfvars
file. This file can be tailored to your specific environment, such as dev.tfvars
or test.tfvars
, containing environment-specific configurations. In our example, we only create a sample.tfvars
file where we specify the region and EKS cluster name. This is all that’s required to create an EKS cluster within its own VPC, thanks to the modularization capabilities of Terraform and the creation of an API.
# aws region
region = "us-east-2"
# eks cluster name
cluster_name = "platformwale"
To initialize Terraform and download the necessary provider plugins, navigate to the my-eks-tf
directory containing main.tf
and .tfvars
file as created in the sections above. Make sure .tfvars
file is prepared, you can refer the file in the github repository.
Below we are creating an s3 bucket to store the tfstate file for our execution. And configuring terraform CLI -backend-config
to point to the s3 bucket created above and intialize the terraform module.
# tfstate s3 bucket name
tfstate_bucket_name="unique s3 bucket name"
# make sure to create the s3 bucket for tfstate file if it doesn't exist
aws s3api create-bucket --bucket "${tfstate_bucket_name}" --region "us-east-1"
# tfstate file name
tfstate_file_name="<some name e.g. eks-1111111111>"
# initialize the terraform module
terraform init -backend-config "key=${tfstate_file_name}" -backend-config "bucket=${tfstate_bucket_name}" -backend-config "region=us-east-1"
To validate the setup and see what resources Terraform will create, use:
terraform plan -var-file="path/to/your/terraform.tfvars"
# example
terraform plan -var-file="sample.tfvars"
To apply the changes and create the VPC, Subnets and EKS cluster, use:
terraform apply -var-file="path/to/your/terraform.tfvars"
# example
terraform apply -var-file="sample.tfvars"
Terraform will show you an execution plan, indicating what resources it will create. If everything looks good, type “yes” to proceed.
After the EKS cluster is created, you need to update your kubeconfig to point kubectl to the newly created EKS cluster and install nginx
helm chart.
# retrieve kubeconfig
aws eks update-kubeconfig --region "<aws region>" --name "<eks cluster name>"
This will update existing kubeconfig at ~/.kube/config
location on your local machine and set the current-context to point to this new EKS Cluster. Now, you can interact with your EKS cluster using kubectl.
$ kubectl config current-context
arn:aws:eks:us-east-2:xxxxxxxx:cluster/platformwale
Install nginx helm chart as below, this will create an external loadbalancer:
# add bitnami helm chart repo
helm repo add bitnami //charts.bitnami.com/bitnami
# install nginx helm chart
helm install -n default nginx bitnami/nginx
Make sure the nginx
pod is running and an external loadbalancer svc is created, example a below –
$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
nginx-7c8ff57685-77ln4 1/1 Running 0 6m45s
$ kubectl get svc -n default nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx LoadBalancer 172.20.45.97 xxxxxxxxxxxx.us-east-2.elb.amazonaws.com 80:31008/TCP 5s
This means you have functional EKS Cluster and you are able successfully deploy services to it.
You can click on the EXTERNAL-IP
and open in the browser, that’s the loadbalancer public record, you should see something like below if the nginx pod is running correctly.
Refer this section of the for complete instructions.
This is the most important step, make sure you destroy all the resources you created using Terraform earlier, otherwise you may see the unexpected costs in your AWS Account.
First uninstall nginx
helm chart to remove the loadbalancer created:
# uninstall nginx chart
helm uninstall -n default nginx
# make sure nginx svc is gone
$ kubectl get svc -n default nginx
Error from server (NotFound): services "nginx" not found
Now destroy infrastructure using following terraform command, in about ~10 mins all the infrastructure will be destroyed:
# destroy infrastructure
terraform destroy -var-file="sample.tfvars"
Refer this section of the for complete instructions.
# for recursively formatting all the files in current and sub-directories
terraform fmt -recursive
# for formatting files in a current directory only
terraform fmt
Below is an example on how we generated doc for modules/eks
–
cd my-eks-tf/modules/eks
terraform-docs markdown .
This command should generate the markdown using the main.tf
, variables.tf and outputs.tf files. This CLI supports other formats as well, feel free to explore.
Originally at