Background
When I first started using Terraform, my biggest complaint was the amount of boilerplate code when creating a complex deployment. Creating a module is essential for DRY principles when using Terraform. The official and community modules are capable, but include a lot of cruft and options. With this plethora of options comes the ability to misconfigure your services. Bringing up new services is an extremely common task for DevOps/Platform engineers, and copying/pasting code from the previous deployment is unmaintainable. To solve this problem, this guide will teach you how to create Terraform modules.
Creating a module is not unlike creating any other computer program. You have inputs, code that operates on inputs, and the results, or outputs. The goal is to create a module that will work for any new service or set of services you want to bring up in your environment. In the module you set all the necessary security and governance settings required. This is what’s known as a “paved road.” When you build a paved road you both make it easier to get where you want to go, and prevent straying off to the land of insecure configurations.
One important note, is that modules can also be composed of other modules. This allows you to logically separate resources for more complex modules. By separating out related resources, it also allows you to unit test each part of your module that logically goes together.
Anatomy of a module
A module consists of 4 files. The important ones are main.tf, variables.tf, and output.tf. The final one, versions.tf, is less important and just sets minimum compatible versions of Terraform and the required provider(s).
variables.tf
This file declares your input variables. These variables hold the data that you want to pass into your module, such as name, tags, and other attributes you need to set to configure your resources. It also allows you to set default values that will be secure and won’t need to be set when using the module, but can be easily overridden.
outputs.tf
This file declares and sets your output variables. This allows you to pass attributes of created resources outside of the module.
main.tf
This is where you make Terraform do its thing by creating resources and using variables from variables.tf in the attributes. As discussed earlier, security and governance settings can be hardcoded or at the very least defaulted to.
modules/
This is where you break up your complex module into small, simple components. Each directory within modules/ is its own module and will have the 4 files mentioned above.
examples/
This directory holds examples on how to use your module. If it handles multiple deployment scenarios, you can create an example for each scenario.
tests/
This holds the Golang code for testing, as well as some Terraform code for unit testing the submodules. The Terraform code uses main.tf to create the module with knowm inputs and the outputs.tf file passes outputs to verify in the Golang code.
The challenge
A developer creates 2 services with a REST-like API. The first converts SVG files to PDF. The second converts web pages to PDF. The services should be ran using an ECR repository and will serve an existing application.
The solution
I broke the module that I created for this into 4 sections: cluster, ecr, service, and loadbalancer. The cluster module just creates an ECS cluster and can be easily unit tested. The ecr module creates a repository and an IAM role with push permissions that developers or service accounts can be attached with. This also can be easily unit tested.
The service module requires a cluster and repository to be created and some values passed in.
You can see this in the root main.tf file, where the values of some attributes start with module..
These values are from the output.tf file of the associated module.
This module uses these values to create ECS tasks to run the services.
An ALB is created to serve the ECS tasks with HTTPS.
An API gateway could be used but would be overkill with these being served for an internal application.
For testing I used Terratest.
With a small amount of Golang, you an easily test your Terraform code.
For unit testing, I created a folder for each module within tests/.
The main.tf file within that folder defines an instance of that module, and the output.tf file contains the output values that you want to check.
Within the Golang test code, use assertions to check that the module outputs are what you expect them to be.
You can also use the AWS SDK to write checks more complex than checking output variables.
When you run go test ./... from the root directory Terratest tests your code by creating resources, checking assertions and any custom checks you have written, then destroys the resources created to clean itself up.
Make sure you have your AWS environment variables set properly before testing, as this uses your account for these tests.
To do some actual functional testing on these, I wrote two services in Sinatra, one for converting SVG files and one for converting web pages. I used ChatGPT to get a starting point, then made some edits so that they would actually work. From there I created some simple Dockerfiles for them, and it was off to the races!
Further Improvements
- Create more examples
- More intermediate testing, this jumps from unit testing to a full end to end test.
- On further testing, using the AWS SDK to test against AWS instead of the Terraform state file.
Repository
The code for this is hosted at: