Terraform Configuration

The terraform project extension allows for the configuration of toolchains and sourcesets.

Toolchains

Terraform toolchains are configured in the toolchains block. On xref:platform-installation-support.adoc it offers the option of configuring a Terraform version, which Gradle will then download, cache and use.

By default, you do not need to configure anything as there will be a toolchain called standard which will have a version of Terraform defined. You can override this version.

terraform {
  toolchains {
    standard {
      executableByVersion('1.8.8') (1)
      executableByPath('/here/is/my/installed/terraform') (2)
      executableBySearchPath('terraform') (3)
    }
  }
}
1 Use a specific version. If it is not available locally, then Gradle will bootstrap and cache it.
2 Use a fixed location of the executable.
3 Look in the system search path for an executable called terraform (or terraform.exe on Windows).

Backends

Backends is where remote state is stored. A configuration needs to have at least one backend defined which are then referenced by name from the source sets.

import org.ysb33r.gradle.terraform.backends.S3Backend (1)

terraform {
  backends {
    s3(S3Backend) { (2)
      bucket = 'my-bucket' (3)
    }
  }

  sourceSets {
    main {
      useBackend('s3') (4)
    }
  }
}
1 Import the backend that you need.
2 Name the backend and use the appropriate type that was imported. In this case the backend is named s3
3 Configure the backend appropriately depending on the backend type.
4 Link a source set to a specific backend. The following backend types are supported out of the box:

  • GenericBackend - For supporting any other backend that is not directly supported.

  • LocalBackend - Not really meant for real usage, but still has some benefit.

  • S3Backend - Store state in an S3 bucket.

Source Sets

Source sets are managed from terraform.sourceSets. When the org.ysb33r.terraform plugin is applied, it adds a main source set. Associated with this source set is the default folder of src/tf/main and a number of tasks including:

  • tfApply

  • tfCleanupWorkspaces

  • tfDestroy

  • tfDestroyPlan

  • tfFmtCheck

  • tfFmtApply

  • tfImport

  • tfInit

  • tfOutput

  • tfPlan

  • tfShowState

  • tfStateMv (terraform state mv)

  • tfStatePush (terraform state push)

  • tfStateRm (terraform state rm)

  • tfValidate

If additional source sets are needed, they can be added by convention i.e.

terraform {
    sourceSets {
        s3Buckets (1)
        cloudFront {
            srcDir = 'src/elsewhere/cf' (2)
        }
    }
}
1 Creates an Terraform source set named 's3Buckets' with default directory src/tf/s3Buckets
2 Creates an Terraform source set named 'cloudFront` and set the directory to src/elsewhere/cf.

Tasks for additional source sets follow the tofo<SourceSetName><TerraformCommand> format. For instance in the above example the initialisation task for the s3Buckets source set will be called tfS3BucketsInit.

By convention all tasks that map Terraform commands start with tf. Other non-commands tasks might start with terraform or contain Terraform within the task name.

Configuring source sets

terraform {
    sourceSets {
      main {
         srcDir = 'src/tf/main' (1)
         variables { (2)
           var 'aws_region', 'us-east-1'
         }
         secondarySources 'src/tf/modules' (3)

         workspaces 'alpha', 'beta' (4)

         useBackend 's3' (5)
      }
    }
}
1 Source directory, which will also be the working directory for terraform.
2 Configure any variables that are specific to the source set. See the Variables block for more details.
3 Additional sources that should affect re-running of tasks, but which are not directly part of the existing source set.
4 Adds additional workspaces to the source set.
5 Link this source set to a named backend.

Variables

A number of Terraform task types support variables and files containing variables. Any variables block support the following functionality

variables {
    var 'foo', 'bar' (1)
    map 'fooMap', foo: 'bar' (2)
    list 'fooList', 'foo', 'bar' (3)
    list 'fooList', [ 'foo', 'bar'] (4)
    files 'filename1.tfvars' , 'filename2.tfvars'(5)
    provider { VariablesSpec v -> v.var('foo2','bar2')} (6)
    remoteStateMap {
      injectVar = true (7)
      varName = 'my_alt_remote_state' (8)
    }
}
1 Adds one variable called foo or value bar. The provided value can be anything that can be lazy-evaluated to a string.
2 Adds a map called fooMap. The map keys have to be strings, but the map values can be anything that will lazy-evaluate to a string.
3 Adds a list called fooList with values foo and bar. The list entries can be anything that will lazy-evaluate to a string.
4 Alternative list format
5 Adds one or more files containing a list of terraform variables. The names can be anything that will lazy evaluate to a string and may be a relative path. The files will be resolved relative to the source set directory.
6 Adds an action that will be called on a VariablesSpec instance to provide more variables. These actions will be evaluated before any of the above methods are evaluated.
7 Inject the tokens from the backend into a map called remote_state.
8 Use an alternative variable name than remote_state.

Workspaces

This plugin suite adds naming conventions to easily deal with workspaces. If you add a workspace called alpha then the apply task for this workspace will be tfApplyAlpha. If you add a workspace called beta to a source set called release, then the apply task will be tfReleaseApplyBeta. These conventions only apply to tasks which are workstate/state-aware in terraform. For instance there will be no task named tfInitAlpha or tfFmtCheckAlpha. There is also no need to switch workspaces, as the plugin will do that under the hood automatically. If you run ./gradlew tfApplyAlpha tfApplyBeta tfApplyGamma tfOutput, the plugin will automatically perform a a terraform select before executing terraform apply or terraform output.

If you decide to remove workspaces, simply cleanup the state by running the appropriate tfDestroy task(s) and then remove the workspaces from the source set DSL. Finally run tfCleanupWorkspaces.

Override Terraform version for a source set

Sometimes you might need one of your source sets to run with a different version of Terraform. For instance, you might want to upgrade, but one of the source sets will take more work and leaving it at an older version for a period might be a good solution.

Assuming that you have sourceSet called monkey, this can be achieved via task actions

build.gradle
terraform {
  toolchains {
    legacyTofu { (1)
      executableByVersion('1.8.3')
    }
  }

  sourceSets {
    monkey {
      useToolchain('legacyTofu') (2)
    }
  }
}
1 Create a toolchain for the legacy version
2 Tell your source set to use that toolchain.

Configure multiple source sets

build.gradle
terraform {
  sourceSets.all { (1)
    useBackend('s3')
  }
}
1 Apply this configuration to all source sets.

Have relationships between source sets

Sometimes you need to ensure that the infrastructure of one source set is applied, before applying that of another source set. There is a simple DSL method to eliminate the need for you to add all the inter-tasks dependencies.

terraform {
  sourceSets {
    database {
    }
    etl {
        mustRunAfter('database') (1)
    }
  }
}
1 Ensures that tfEtlApply and tfEtlPlan will run after tfDatabaseApply* and tfDatabasePlan*. The relationship is a simple mustRunAfter.

Configure the plugin cache directory

Be default the plugin cache directory from terraformrc in the root project is used.

It is possible to also use a custom plugin cache directory per source set.

build.gradle
terraform {
  sourceSets {
    main {
      useConfiguredPluginCache() (1)
      useCustomPluginCache() (2)
      useCustomPluginCache('path/to/my/cache', 100000) (3)
    }
  }
}
1 Use the plugin cache directory that is configured in terraformrc (Default behaviour).
2 Use a custom plugin cache directory for this source set. This cache dir is stored under the project cache directory
3 Provide a custom directory for the plugin cache. If the directory is shared with other source sets or projects, provide a lock timeout in milliseconds, otherwise pass 0.

Secrets

You can create arbitrary secrets and share them with backends and source sets

build.gradle
import org.ysb33r.gradle.iac.base.secrets.AwsSecrets

terraform {
  secrets {
    awsAcct1(AwsSecrets) { (1)
      useAccessKeyId('1234567890')
      useSecretAccessKey('abcdefghijklmn')
    }
  }

  backends {
    s3(S3Backend) {
      fromSecretsProvider(terraform.secrets.awsAcct1) (2)
    }
  }

  sourceSets {
    main {
       fromSecretsProvider(terraform.secrets.awsAcct1) (3)
    }
  }
}
1 Declare a collection of secrets. In this example we use org.ysb33r.gradle.iac.base.secrets.AwsSecrets for declaring AWS secrets like profile, credentials file, access keys etc.
2 Tell the S3 backend to use the same secrets that was defined in our secrets definition.
3 You can also tell the source set to use the values from a secrets definition.

You can also use org.ysb33r.gradle.iac.base.secrets.GenericSecrets as a way of rolling your own secrets.