CI/CD Pipeline: Deploy Python App with Healthcheck Endpoint on AWS EKS using Terraform & GitHub Actions (DEV/PROD)
Table of Contents
1. Introduction
2. Prerequisites
3. Tools Used
4. Pipeline Workflow Overview
5. Setup and Configuration
6. Local Build and Testing
7. CI/CD Pipeline Implementation
8. Kubernetes Configuration
9. Security Checks
10. Branching Strategy
11. Known Risks
12. Final Testing and Final Push/Merge to Production
13. Conclusion
1. Introduction
This guide walks you through setting up a CI/CD pipeline for a containerized Python application. Using Docker, Terraform, GitHub Actions, Kubernetes, and AWS ECR, you will build, test, and deploy the application in an automated workflow. The goal is to enable automatic deployment on each new code commit.
2. Prerequisites
Before beginning, ensure you have:
1. AWS Account or Azure with permissions for ECR and EKS.
2. GitHub Repository for version control and triggering the pipeline.
3. AWS CLI or Azure CLI configured on your local machine.
4. Kubectl for Kubernetes management.
5. Terraform for provisioning infrastructure on AWS.
6.VS Code
3. Tools Used
- Docker: For containerizing the Python application.
- AWS ECR: As the Docker image registry.
- AWS EKS: To deploy the containerized application in a Kubernetes cluster.
- Terraform: For provisioning EKS, S3 bucket for state management, and related resources.
- GitHub Actions: For CI/CD, including building, testing, security scanning, and deployment.
- Trivy, Bandit, and Safety: Security tools for scanning the code and Docker images for vulnerabilities.
- Bash Script: For automating image url on k8s manifest file
4. Pipeline Workflow Overview
The CI/CD pipeline executes the following steps:
1. Code Commit: Developer commits code to GitHub.
2. Dependency Installation: Install dependencies for the Python app.
3. Lint and Unit Test: Code linting and unit tests ensure code quality.
4. Environment Configuration: Set environment variables based on the branch (development or production).
5. Build Docker Image: Containerize the application with the latest code.
6. Security Scans: Use Bandit, Safety, and Trivy to scan the code and Docker image.
7. Push to ECR: Upload the Docker image to AWS ECR.
8. Update Kubernetes Config: Update the Kubernetes configuration for the new image.
9. Deploy to Kubernetes: Roll out the updated image to the AWS EKS cluster.
10. Health Check: Verify the deployment by checking the\healthcheckendpoint.
5. Setup and Configuration
1. Clone the Repo
- Create a new repository or fork the given repo
- Open your VS Code or anyone you prefer and Clone the repo https://github.com/ougabriel/gab-test-mod.git
When this is done you may have to take some time to go through the files and understand what this project is all about.
For this project the main branch will be our prodbranch, Please check your GitHub repo, and make sure it has atlease a dev and main branch; if it doesn’t create additional one and name it accordingly.
6. Local Build and Testing
the app containsapp.py which is the main file and test_app.py which is the file that helps perform unit test and get the last_commit-sha
In the cloned repo use the provided script gab-app-local.sh to build and test the application locally. It runs tests with pytestas part of the Docker build process. Depending on your system dependencies this test may not throw any local errors because it is already a completed project as at the time of this writing.
To build the app locally: Open a terminal and make sure you are in the repo dir
cd gab-test-modDue to repo restrictions, if you can’t find the bash script in your cloned repo, you can use this script; > Create a file touch gab-app-local.sh > paste the code below into it.
#!/bin/bash# Set variables
APP_NAME="gab-app"
IMAGE_NAME="gab-app"
PORT=3000echo "---- Building Docker Image ----"
docker build -t $IMAGE_NAME .if [ $? -ne 0 ]; then
echo "Docker build failed. Exiting."
exit 1
fiecho "---- Running Docker Container ----"
docker run --rm -d -p $PORT:3000 --name $APP_NAME $IMAGE_NAMEif [ $? -eq 0 ]; then
echo "App is running at http://localhost:$PORT"
echo "To stop the app, run: docker stop $APP_NAME"
echo "To delete the app, run: docker rmi $APP_NAME"
else
echo "Failed to start the app."
fi
Run the bash script, the app will be tested automatically during docker build. You will need gitbashto run this script on your vscode. Try to install it on your system.
./gab-app-local.shTest the healthcheckendpoint: or use the browser
curl http://localhost:3000/healthcheckExpected JSON response and APP login page
Our APP Endpoint does have a last_commit_sha at this point because a commit has already been done at this point in the project.
Before we continue, stop and delete the container.
docker stop gab-app
docker rmi gab-app> Push changes to dev branch
Since our code is working perfectly in our local branch, we need to push it to the git repo.
> Make sure you have a clean git working dir
this is my git repo link https://github.com/ougabriel/gab-test-mod.git this is what i will be using as my origin, change it to the name of your repo or link
git remote -v #checks to see if the origin is the one we need
git remote remove origin #removes the origin if it is not what we needgit remote add origin https://github.com/ougabriel/gab-test-mod.git #adds the origin we need> Push the changes to dev
I don’t have the dev branch in my repo yet. I will need to commit, create the branch and push to it.
git init #initiliazes the new repogit add .
git commit -m "Added project files to the dev branch"
git checkout -b dev #creates and switches to the dev branch
git push origin dev # pushes changes to the dev branch7. CI/CD Pipeline Implementation
a. Login to AWS:
> Login to the AWS console > Click on your profile name > Click on security credentials
> Click on Create access keys > when this is created, copy and paste the AWS Access Key ID unto your terminal > copy and paste the AWS Secret Access Key to a terminal > Leave other prompts as default
Now that we are logged in to our AWS, lets continue with the deployment.
b. Use Terraform to deploy needed infrastructures:
Before we can deploy the resources, we need to first create terraform workspaces. Terraform Workspaces helps you manage multiple groups of resources with a single configuration.
cd terraform
terraform initterraform workspace new dev
terraform workspace new prodTwo terraform workspaces which are dev and prod has been created. We need to deploy first to the dev , test our APP after deployment, if everything is perfect and fine, then we destroy the resources before finally deploying to prod. This is because we want to save resource usage and extra charges it may incure.
NOTE: if terraform init throws an error it could be because the backendis cloned directly to your code. (search and comment it out). If you have the backend with no error, move to the next step. (> Ensure the Terraform backend is configured to use S3 for state management.)
> Select the dev workspace and initialize it. > Plan and Apply
terraform workspace select dev
Run the following command to deploy resources to the dev workspace
terraform init
terraform plan
terraform applyWhen this is deployed, we need to migrate our tf state files to the cloud so it can be accessed remotely. We will import it to an s3 bucket to achieve this purpose.
NOTE: If the Terraform state file is not migrated to a remote backend, we won’t be able to access the Terraform resources during our CI/CD processes. Using a remote backend also facilitates easy collaboration with other team members.
> Ensure the Terraform backend is configured to use S3 for state management.
Paste the following script inside your main.tf . In this main.tf file; files created here will have the attached names ofdev and prod suffix respectively.
terraform {
backend "s3" {
bucket = "gab-terraform-state-bucket-dev" ## actual bucket name
key = "terraform/state/default.tfstate"
region = "eu-west-2"
}
}When this is done, initialize the repo and apply the config.
terraform init
terraform applyc. GitHub Actions Workflow (`ci.yml`)
Before we can trigger the pipeline, we need to define our pipeline.yaml script and tasks. I will be doing this directly on my vscode editor, such that when I am satisfied with my pipeline script, I can do commit and push it to the dev or prodrepo. This push will trigger the pipeline to deploy the resources automatically.
Before we begin; at the root of your project directory create a .github/workflows directory > create a file named cicdpipeline.yml inside the workflows dir.
.github/
└── workflows/
└── cicdpipeline.yml> Pipeline Breakdown
a. Set environment variables based on branch
This part of the pipeline is worthy of detailed explanation, so you can understand what exactly is going on.
- name: Set environment variables based on branch
run: |
if [ "${{ github.ref }}" == "refs/heads/main" ]; then
echo "ECR_REPOSITORY=${{ secrets.ECR_REPOSITORY_PROD }}" >> $GITHUB_ENV
echo "ENVIRONMENT=prod" >> $GITHUB_ENV
else
echo "ECR_REPOSITORY=${{ secrets.ECR_REPOSITORY_DEV }}" >> $GITHUB_ENV
echo "ENVIRONMENT=dev" >> $GITHUB_ENV
fi> Purpose: This step sets up some special variables that the pipeline will use later, based on which branch of the code is being worked on (either `main` or `dev`).
> Checking the Branch:
— It looks at the current branch using `${{ github.ref }}`. This is like asking, “Am I on the `main` branch or the `dev` branch?”
> If It’s the `main` Branch:
— If the branch is `main`, it does two things:
— It sets a variable called `ECR_REPOSITORY` to a value that comes from a secret called `ECR_REPOSITORY_PROD`. This is the address of the production Docker repository.
— It sets another variable called `ENVIRONMENT` to `prod`, indicating that the code is being deployed in the production environment.
> If It’s Not the `main` Branch:
— If the branch is anything else (like `dev`), it does the following:
— It sets the `ECR_REPOSITORY` to a different value from a secret called `ECR_REPOSITORY_DEV`, which points to the development Docker repository.
— It sets `ENVIRONMENT` to `dev`, meaning the code is being worked on in a development environment.
How It Works:
— The `>> $GITHUB_ENV` part means these variable settings will be saved for later steps in the pipeline to use.
In Summary:
This code checks which branch you’re working on and sets some important variables that tell the pipeline which environment (production or development) it’s working in and where to find the correct Docker repository.
b. The entire pipeline script explained in simple terms
1. Triggers: The pipeline starts when code is pushed or a pull request is made to the `dev` or `main` branches.
2. Build and Test Job:
— Checks out the code.
— Sets up Python and installs dependencies.
— Runs unit tests and checks the code for style issues.
— Conducts security scans to find vulnerabilities in the code and dependencies.
3. Build and Push Docker Image Job:
— Depends on the successful completion of the build and test job.
— Checks out the code again.
— Configures AWS credentials.
— Logs into Amazon Elastic Container Registry (ECR) to push Docker images.
— Builds a Docker image and runs security scans on it.
— Pushes the Docker image to ECR.
4. Deploy to Kubernetes Job:
— Depends on the Docker image job.
— Checks out the code and configures AWS credentials.
— Prepares the Kubernetes configuration to connect to the correct cluster.
— Updates the deployment settings and applies them to deploy the application on Kubernetes.
— Creates secrets to access the Docker image and retrieves the status of the deployed resources.
Overall, this pipeline automates the process of testing, building, and deploying an application to ensure quality and security at every stage.
c. Deploying the pipeline
The entire pipeline script is found here, copy and paste it in the correct directory.
cd .\.github\workflowsd. Create Secrets & Variable for the pipeline
In your github repo > click on settings > In Secrets & Variables, select actions, click on New Repo Secret and Add it
Create Secrets for the following.
i. Name: AWS_ACCOUNT_ID
value: <your aws account ID>
ii. AWS_ACCESS_KEY_ID
value: <your aws secret key ID>
iii. AWS_SECRET_ACCESS_KEY
value: <your aws secret access key>
iv. AWS_REGION
value: <your aws region> (example: eu-west-2)
v. ECR_REPOSITORY_DEV
value: <your ecr repo name for dev>
example: gabapprepodev (this is defined by terraform in main.tf, if you changed it. check the ecr script or in AWS ecr)
vi. ECR_REPOSITORY_PROD
value: <your ecr repo name for prod>
example: gabapprepoprod (this is defined by terraform in main.tf, if you changed it. check the ecr script or in AWS ecr)
AWS acount ID > click on your aws profile name to find it
Access KEY and ID > on your profile name, click on security credentials > create a New secret
e. Pushing changes to dev branch.
This time, we won’t push it directly. We need to create a .gitignore folder at the root of your project directory. When it is created, copy and paste the following directories into it. These are large files which is not needed to be pushed along with your source code.
# Ignore Terraform state and provider files
.terraform/
.terraform.lock.hcl
terraform.tfstate
terraform.tfstate.d
terraform.tfstate.backup
terraform/.terraform/providers/**
terraform/terraform-provider-aws_v5.72.1_x5.exe
terraform/terraform-provider-aws*.exe
terraform/terraform-provider-*.exe
# Ignore Terraform provider binaries
terraform/.terraform/providers/terraform-provider-aws_*.exe
#git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch terraform/terraform-provider-aws_v5.72.1_x5.exe' --prune-empty --tag-name-filter cat -- --all# Ignore Ansible virtual environment folder
ansible-venv/# Ignore Python cache and virtual environments
__pycache__/
*.pyc
*.pyo
*.pyd
*.whl
Pipfile.lock
pyvenv.cfg# Ignore other IDE or OS-specific files
.vscode/
.DS_Store# Ignore any logs or temporary files
*.log
*.tmp
8. Kubernetes Configuration
Create a folder named kubernetes , this folder should have three files.
- deployment.yaml: deployes the image we need to our AWS EKS cluster
- 2. service.yaml: exposes our app to the internet using a Loadbalancer
- 3. update-k8s.yaml: helps us to dynamically monitor the ECR and populate the
deployment.yamlimage with the most recent one if there is any new push or PR in any of the branches. (this can also be done using ArgoCD or gitOps)
You can check the files in the repo link https://github.com/ougabriel/gab-test-mod.git > copy and paste the scripts into your working dirs.
9. Security Checks
In the root of your project directory, create a folder name .flake8 and paste the following into it. This will make sure that venv is not included in the security checks and to use the max line length specified.
[flake8]
exclude = venv
max-line-length = 8810. Branching Strategy
This has already been explained previously, the dev branch will push to the dev branch during deployment but the main branch is modified to be the prod branch within the pipeline. This is to enable the deployment to be seamless and also match with the naming of the terraform workspace deployment.
Go back to the root of the project directory, we need to push all the changes.
git init #reinitiliazes the repogit add .
git commit -m "Added new files to the dev branch"
git checkout -b dev #creates and switches to the dev branch
git push origin dev # pushes changes to the dev branchWhen you deploy your pipeline, we do not expect you to have any security, test or lint errors. this is because most of the security checks has already been resolved within the pipeline itself or during the set up process and also within the python code. If you encounter errors it should be from your end, try to troubleshoot it and understand what the errors is. Handling errors helps boost confidence and expands your knowledge and expertise. Please dont shy away from error logs embrace them.
If everything is right, your pipeline stages should look like
11. Known Risks
1. Dependency on AWS: Relies heavily on AWS, limiting portability.
2. Branching Limitations: Only deploys from `dev` and `main` branches.
3. Kubernetes Dependency: Assumes cluster is always healthy.
12. Final Testing
After deployment, in your aws console > search for loadbalancer > copy the DNS name and paste it into a browser
In the browser, you should have the Apps login page
Test the `/healthcheck` endpoint to confirm the application’s health.
http://<load-balancer-ip>/healthcheck— -
13. Final Push to Production
if you completed the entire dev processes; this should be simple for you.
a. In your terraform dir > go to the terraform prod workspace by running the following command terraform workspace select prod > run terraform apply to create new infrastructure needed for the prod deployment
b. Merge the dev branch with the mainbranch; > git checkout main> git pull origin main > From the main branch merge the changes git merge dev > Apply the changes git push origin main
NOTE: instead of doing a merge, you can also do a Pull Request in the github UI.
This will push all changes to prod, monitor the deployement.
14. Troubleshooting
During the deployment, you can check the pipeline logs to see if the service is running and on what external IP it is on. (the external IP takes sometime to populate, if you’d like to see it. You can run kubectl get svc directly in your EKS cluster”
It is important to check if the pods are running
In the pipeline logs you can that the pods are not running; or check the pods description . Or you can run kubectl describe pod directly in your EKS cluster
15. Clean up
In the terraform dev workspace
terraform workspace select dev
terraform destroyFor the terraform prod worspace (if you completed this stage of the project)
terraform workspace select prod
terraform destroyGo to AWS, delete any residual resources. (Note: take note of any errors that may occur during terraform destroy and delete them manually)
Delete, the Loadbalancer ( this is expensive to maintain. Make sure this is deleted manually, else you will incur expensive cost you may not like to experience)
Delete the S3 bucket, (has to be emptied and then deleted)
Delete any residual VPC if applicable
16. Conclusion
This guide provides a full setup for a CI/CD pipeline using Docker, ECR, Kubernetes, Terraform, and GitHub Actions. By following these steps, you can automate your application’s testing, deployment, and security checks, ensuring a robust production workflow.
Authors Note:
If you encounter any issues completing this project. You can email me or comment on this post and I would be glad to help.
#devops #devopsengineer #devopsticket #GabrielOkom #UgochiGabrielOkom #sreengineer #github #gitops #bash #scripting #git #seniordevopsengineer #devopsproject #devopshandsonproject #jobscenariodevops #kubernetes #aws #eks #ec2
