We are now ready to write ourselves a Jenkins build pipeline that will be triggered whenever a new Git branch is created or deleted, or a branch has had code changes applied to it.
Let’s begin by filling in our example project - we can use the same project we used when building our iac-env Docker container and grow it later.
Start off by checking out a new Git branch to work in within the example-project folder - remember that we will need to create pull requests from now on if we want to merge any code into the master branch so we can’t directly edit and push code on the master branch:
Note: Make sure you are in the
example-projectfolder.
git checkout -b feature/add-project-files
Now copy all the files from the setup/iac-env/resources/docker/kitchen-setup into our example-project folder. The last thing we will do before pushing this new feature branch into Bitbucket is to add a basic Jenkinsfile which defines our pipeline.
Note: We will use the Jenkins declarative pipeline approach which is a simplified way to run Jenkins pipeline builds. It is a bit restrictive in some ways and if you intend to do more complex build pipelines you might instead consider to use the scripted pipeline approach. The declarative approach will be good enough for our experiment though.
Before we add a Jenkinsfile definition, take a moment to review this site which has information about working with Terraform in automation: https://learn.hashicorp.com/terraform/development/running-terraform-in-automation. We will observe a couple of things from that site and apply some more later when we wire up some Amazon resources.
Create a new text file named Jenkinsfile in the example-project folder with the following content:
final def IS_PULL_REQUEST_BUILD = env.BRANCH_NAME.startsWith('PR-')
final def IS_MASTER_BUILD = env.BRANCH_NAME == 'master'
final def DOCKER_IMAGE_NAME = 'iac-env:latest'
final def DOCKER_IMAGE_ARGS = '-it'
pipeline {
agent none
stages {
stage("Start: Verification") {
agent { docker image: DOCKER_IMAGE_NAME, args: DOCKER_IMAGE_ARGS }
environment {
TF_IN_AUTOMATION = 'TF_IN_AUTOMATION'
}
stages {
stage("Basic Validation") { steps {
sh 'terraform fmt -check -recursive -list=true -diff'
sh 'tflint'
sh 'terraform init -input=false'
sh 'terraform plan -out=tfplan -input=false'
sh 'terraform validate'
sh 'terraform graph | dot -Tpng > terraform-graph.png'
archiveArtifacts artifacts: 'terraform-graph.png'
}}
stage("Integration Tests") {
when { expression { IS_PULL_REQUEST_BUILD || IS_MASTER_BUILD } }
steps {
sh 'kitchen verify'
}
post {
always {
sh 'kitchen destroy'
}
}
}
stage("End: Verification") { steps {
milestone(label: 'End: Verification', ordinal: 1)
}}
}
post {
cleanup {
cleanWs()
}
}
}
stage("Publish Approval") {
when { expression { IS_MASTER_BUILD } }
steps {
input message: "Deploy these changes?"
}
}
stage("Start: Publish") {
when { expression { IS_MASTER_BUILD } }
agent { docker image: DOCKER_IMAGE_NAME, args: DOCKER_IMAGE_ARGS }
environment {
TF_IN_AUTOMATION = 'TF_IN_AUTOMATION'
}
stages {
stage("Deploy") {
steps {
milestone(label: 'Deployment started', ordinal: 2)
sh 'terraform init -input=false'
sh 'terraform plan -out=tfplan -input=false'
sh 'terraform apply -input=false tfplan'
}
}
stage ("End: Publish") {
steps {
milestone(label: 'Deployment complete', ordinal: 3)
}
}
}
post {
cleanup {
cleanWs()
}
}
}
}
}
Let’s walk through the Jenkins declarative pipeline definition so I can explain each part.
final def IS_PULL_REQUEST_BUILD = env.BRANCH_NAME.startsWith('PR-')
final def IS_MASTER_BUILD = env.BRANCH_NAME == 'master'
final def DOCKER_IMAGE_NAME = 'iac-env:latest'
final def DOCKER_IMAGE_ARGS = '-it'
We are defining a few properties here to help us conditionally run parts of our pipeline:
IS_PULL_REQUEST_BUILD: If the Git branch starts with PR-, it means this is a pull request build. We will only run integration tests on pull request build or a master build.IS_MASTER_BUILD: If the Git branch is master, it means this is a master branch build and therefore all of our build pipeline steps should be run.DOCKER_IMAGE_NAME / DOCKER_IMAGE_ARGS: These are for convenience and simply identify which Docker image we will be running as the Jenkins agent node to run our pipeline on. Notice that we are using our home grown iac-env!! This is how we will be recycling our iac-env for both developer and continuous integration use.pipeline {
agent none
We are setting the default agent to none to force us to declare what agent to use to build any given stage in our pipeline.
stages {
stage("Start: Verification") {
agent { docker image: DOCKER_IMAGE_NAME, args: DOCKER_IMAGE_ARGS }
environment {
TF_IN_AUTOMATION = 'TF_IN_AUTOMATION'
}
stages {
We will be declaring two groups of stages to allow our build pipeline to pause between each group and wait for a human to agree to continue the pipeline.
The first group of stages is named Start: Verification and you can see it is declaring which agent to use for all the build operations within it - this is what causes Jenkins to start up an instance of our iac-env Docker image to execute the next sequence of build operations:
agent { docker image: DOCKER_IMAGE_NAME, args: DOCKER_IMAGE_ARGS }
We are also setting a TF_IN_AUTOMATION environment variable which hints to Terraform that we are running in an automated way. See the web link I referenced earlier to learn about this.
stage("Basic Validation") { steps {
sh 'terraform fmt -check -recursive -list=true -diff'
sh 'tflint'
sh 'terraform init -input=false'
sh 'terraform plan -out=tfplan -input=false'
sh 'terraform validate'
sh 'terraform graph | dot -Tpng > terraform-graph.png'
archiveArtifacts artifacts: 'terraform-graph.png'
}}
This is the first of our build stages, comprised of the following series of steps:
terraform fmt -check -recursive -list=true -diff: This performs a code formatting check against all the Terraform source files in the working directory. If any of the Terraform code isn’t formatted properly, this command will fail and the build will stop.tflint: This runs the TFLint tool across the Terraform code, looking for problems that aren’t necessarily related to syntax errors or code structure.terraform init -input=false: We ask Terraform to initialise itself here - Terraform will look for its required plugins within the iac-env container as we had configured it to do so when creating the iac-env Docker image.terraform plan -out=tfplan -input=false: Terraform will calculate what the plan of operations are that will be performed if we apply the code change. The plan tells us about the diff between the state of the world and the state after the changes. We should always review the plan to see what will happen if we apply the changes.terraform validate: Terraform has a built in validate command that will check the syntax and usage of our Terraform code.terraform graph | dot -Tpng > terraform-graph.png: We can produce a graphic image that shows a graph view of what our changes would look like amongst all the resources involved.archiveArtifacts artifacts: 'terraform-graph.png': Keep a copy of the graph in the Jenkins build so we can see it later.stage("Integration Tests") {
when { expression { IS_PULL_REQUEST_BUILD || IS_MASTER_BUILD } }
steps {
sh 'kitchen verify'
}
post {
always {
sh 'kitchen destroy'
}
}
}
The integration tests will be run through Terraform Kitchen, and only if we are building a pull request, or master branch. The command to run the integration tests is simply kitchen verify. We have a trailing post block that will always run kitchen destroy to clean up any test artifacts even if the tests themselves fail. If we didn’t put this in a post / always block then if any integration tests failed, the resources would not get cleaned up.
stage("End: Verification") { steps {
milestone(label: 'End: Verification', ordinal: 1)
}}
We record a Jenkins milestone to indicate that this build has successfully reached the end of verification.
post {
cleanup {
cleanWs()
}
}
We have completed all the execution tasks for the first verification group of stages, so we will clean the Jenkins workspace to free up storage space on the local file system. If we don’t clean up, your disk storage will grow with every build and eventually you’ll run out of space.
stage("Publish Approval") {
when { expression { IS_MASTER_BUILD } }
steps {
input message: "Deploy these changes?"
}
}
This stage forms the second group of build logic and runs an input command, which causes Jenkins to pause the build pipeline and wait for user input through the Jenkins GUI. This acts as an approval gate to prevent the build pipeline from automatically continuing on. In a more realistic build pipeline there would be a bit more rigour to the approval but in our experiment we will simply wait for a human to Proceed or Abort. We are also only running this on a master branch build.
Note: There is no
agentdeclared for this stage as we do not want a paused input step to consume any real build agents while it is waiting. Never do that!!
stage("Start: Publish") {
when { expression { IS_MASTER_BUILD } }
agent { docker image: DOCKER_IMAGE_NAME, args: DOCKER_IMAGE_ARGS }
environment {
TF_IN_AUTOMATION = 'TF_IN_AUTOMATION'
}
stages {
This is the third group of build stages which will proceed with the actual publication of the Terraform code, applying its changes. We are only running this for master builds, and using our iac-env Docker image as the build agent. We need to repeat the environment variable again too.
stage("Deploy") {
steps {
milestone(label: 'Deployment started', ordinal: 2)
sh 'terraform init -input=false'
sh 'terraform plan -out=tfplan -input=false'
sh 'terraform apply -input=false tfplan'
}
}
The Deploy stage initialises Terraform, generates its plan but most importantly actually applies the changes for real. You would need to be certain you want to apply your Terraform code before this runs! Also of interest here is the following line:
milestone(label: 'Deployment started', ordinal: 2)
Recording another milestone in our build pipeline will cause all pipeline builds that haven’t reached this milestone yet to be automatically aborted. We do this on purpose to avoid older builds trying to publish themselves after a more recent build has already been published.
stage ("End: Publish") {
steps {
milestone(label: 'Deployment complete', ordinal: 3)
}
}
Another build milestone - this will record that we have successfully reached the end of our build pipeline and are done. Again, any other builds that haven’t reached this milestone yet will be automatically aborted.
post {
cleanup {
cleanWs()
}
}
We again clean up our workspace since we are done with it.
Before we continue, add a new file .gitignore to the example-project folder with the following to avoid generated files showing as Git changes:
.terraform
.terraform.tfstate.d
.kitchen
Save and close your Jenkinsfile then add all the new files to Git and push your branch to Bitbucket:
git add .
git commit -m "Adding pipeline script."
git push --set-upstream origin feature/add-project-files
Note: You may need to select
Scan Multibranch Pipeline Nowinside theInfrastructure as Code - Example Projectthe first time you add a Jenkinsfile to pick up the changes.
You will notice a new branch has appeared and after a little while the build will turn red.
Navigate into the branch build and see the stages:
Move your cursor over the Basic Validation stage and select Logs to see what the output was for the failure:
Notice that our Terraform format step failed and it is showing us the diff about how our Terraform code is not formatted properly:
We’ve found our first bug!!
Let’s fix the bug and see if the build pipeline kicks off again.
Start up iac-env in your example-project folder and format our Terraform code like so:
$ iac-env
----------------------------------------------------------------------
█ ▄▀█ █▀▀ ▄▄ █▀▀ █▄░█ █░█
█ █▀█ █▄▄ ░░ ██▄ █░▀█ ▀▄▀ v1.0.0
----------------------------------------------------------------------
https://github.com/MarcelBraghetto/iac-env
Working directory mounted at /iac-env
Enter 'iac-env-help' for help, 'exit' to leave.
bash-4.4$ terraform fmt -recursive
main.tf
bash-4.4$ exit
The terraform fmt -recursive command has looked at all our Terraform source files and updated any that weren’t formatted correctly. Add your changes to Git and push them:
git add .
git commit -m "Fixing code formatting"
git push
Another build should kick off for the feature branch, this time all builds are successful:
Note that the stages that are just white squares did not run because we conditionally only ran stages not associated with pull requests or master builds.
We can raise a pull request now that our feature branch build has succeeded which will in turn run our Jenkins pipeline again - this time including our integration tests.
Go to Bitbucket and find your feature branch and Create pull request. Complete the pull request details and Create.
Once the pull request is created you should see a new entry in the Pull requests tab in our Jenkins build:
Navigate into the pull request and you’ll see that now our pipeline included the integration tests stage as well:
Go back to Bitbucket and approve and merge the pull request - this will merge the changes into the master branch and kick off a Jenkins build which will for the first time make our master branch appear in Jenkins and will look like this:
So our master branch has finally appeared in our multi branch Jenkins project. This happened because now our master branch has a valid Jenkinsfile in it, which Jenkins looks for before creating a build pipeline. Navigate into the master branch build and you will see something like this:
The master build has completed all the validation steps and is now waiting for human input before proceeding to the next series of steps. Hover over the last stage and you will be presented with options to either Proceed or Abort:
Select Proceed and let the pipeline continue to the end:
Awesome! We successfully ran our master pipeline to completion. Of course in our example project we only used Terraform to create a local text file so the use case was probably a bit simple but at least we have the rudimentary building blocks for an automated pipeline working.
Next up we’ll provision some real Amazon resources - creating an S3 bucket that can be used later to potentially store Terraform state.
Source code can be found at: https://github.com/MarcelBraghetto/iac-env
Continue to Part 6: Provision AWS resources.
End of part 5