This page looks best with JavaScript enabled

How to Setup a GitHub to Jenkins Pipeline with WebHooks

 ·   ·  ☕ 10 min read

Introduction

In a typical CI/CD pipeline, what happens is when you push your code to a code repository (remote) like GitHub/GitLab/BitBucket, an event is triggered by the git host provider. This trigger is known as WebHook. This webhook can be used by some third-party service to provide an array of integrations. Do you want to tweet on every git push? You can do that.

But today we are not going to tweet on each push. But we are going to run a build that creates a docker image and pushes it to Docker Hub. From there one can pull the images to further use them for further consumption. You wanna use that in your Kubernetes cluster? Why not!

Index

  1. Setup Jenkins server
  2. Setup repo with Jenkinsfile
  3. Setup Multibranch Jenkins pipeline
  4. Setup WebHook on GitHub repo
  5. Test Multibranch Jenkins pipeline
  6. Publish to DockerHub (in next post)

While you read this post, take a moment to connect with me on LinkedIn.

Install Jenkins on AWS Cloud

Leasing a machine from AWS will cost you money. But if you are doing this for learning purposes and you pledge to delete the installation after the tutorial has been done then the cost should be very nominal.

If you are completely new to AWS, you can leverage AWS Free Tier. Just so that you are not in dark, this tutorial makes use of EC2 and EBS.

I’m not exactly going to show how to setup Jenkins server because I have already covered that in past. Starting with this post would be a good start if you are setting your own Jenkins server.

I also have some ansible playbooks which can help you provision your Jenkins and Nginx server.

I hope you have the Jenkins server already running by now. Let it be on your local server or on a cloud. I am following this tutorial on AWS cloud. If you are trying this on your local and you have any issues, please let me know in the comments at the end of the article.

Install Multibranch Scan Webhook Trigger

This is one extension that we’ll use with our Jenkins installation.

To install this extension, head over your Jenkins instance and go to Dashboard > Manage Jenkins > Manage Plugin.

Now search for Multibranch Scan Webhook Trigger and install it.

You might have to restart the Jenkins server to take the effect.

Setup a repo with mock Jekinsfile

After installing Jenkins and the Multibranch Scan Webhook Trigger plugin, the next step is to prepare our repo which is hosted on GitHub.

I suppose you have nothing right now.

To start with, let’s go with this simple pipeline file. This pipeline file is going to print statements to the console on the Jenkins server instead of doing any actual work. But in a typical production environment, they are replaced with real code or command.

Let’s set up a Jenkinsfile:

Jenkinsfile

pipeline {
    agent any

    stages {
        stage('Init') {
            steps {
                echo 'Initializing..'
                echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
                echo 'Running pytest..'
            }
        }
        stage('Build') {
            steps {
                echo 'Building..'
                echo 'Running docker build -t sntshk/cotu .'
            }
        }
        stage('Publish') {
            steps {
                echo 'Publishing..'
                echo 'Running docker push..'
            }
        }
        stage('Cleanup') {
            steps {
                echo 'Cleaning..'
                echo 'Running docker rmi..'
            }
        }
    }
}

Every Jenkinsfile must have a pipeline section. This section instructs the Jenkins application to commence actions defined in different stages.

Each stage must have an agent. If agent in all the stages are the same, it can go one scope above inside the pipeline. agents are basically hosts which can run a specific stage. So in a typical environment, 3 different stages can be running on 3 different compute devices.

Let’s talk a little bit more about the stages. Each stage defined here is a stage/step in CI/CD workflow. In the above pipeline:

  1. Init: This is the initializing phase. In this phase, I plan to set up the build environment for the build. Things like setting up environment variables etc.
  2. Test: The most probable reason anyone would want to establish a CI pipeline is to run unit test. So in this phase, I plan to run the unit test.
  3. Build: Build phase is for creating the artifact out of the repository. This could be a docker image or .deb file depending on the what the repository is for.
  4. Publish: Publish is for publishing the artifact to some repository. Once again, it could be Docker Hub, PYPI or npm, or anything else.
  5. Cleanup: In cleanup phase, I want to clean anything which is will build upon progressively and which can make my Jenkins server out of space.

Once again, we are not doing anything real yet. But this is sufficient enough to proceed to the next step. Above is not an exhaustive list of stages, you can have as many stage as you want. They will be executed in chronological order.

Commit this file to the repo and push it to your GitHub. Once you do that, note the project URL. We’ll need that in upcoming sections.

It will look like something like this: https://github.com/yourusername/yourrepo

Setup Multibranch Jenkins Pipeline (and manual build)

After you are running your Jenkins server successfully. You need to create a Pipeline for the project you are setting up CI/CD for. We are talking about the same project for which we set up a GitHub in the last section. Every stage in that pipeline is executed after every push of the commit on GitHub. This execution is triggered by something called a WebHook which we’ll study in the next section.

  1. On the Dashboard, select New Item.
  2. Enter a name for the pipeline and select Multibranch Pipeline. Press OK.

Once you’ve done the above, now let’s proceed with the most important fields only…

  1. Fill in a Display Name.
  2. Under Branch Sources, click Add source and select Git.
    4a. Under Project Repository, input https://github.com/yourusername/yourrepo.

Now at this point, if your repo is a private repo, you need to do some additional step here to configure credentials. I am not doing it here as it is out of the scope of this article.

  1. Under Build Configuration, select Mode to be by Jenkinsfile.
    5a. Select Script Path to be Jenkinsfile. If Jenkinsfile was in another directory, we would have put the relative from the root of the repository here.
  2. Next up.. under Scan Multibranch Pipeline Triggers, check Scan by webhook.
    6a. Under Trigger Token, input a random string. Note: make this token strong enough that it is hard to guess.

Now when you go back to your dashboard. You’ll see your project listed here. When you click on this project, you’ll see several branches here (for now, you’d only see main/master).

At this point, we have our pipeline setup. At this point, whenever you make an update to the GitHub repository, you can come to the job’s page and click on “Scan Multibranch Pipeline Now”. This would trigger a new build which would run through a defined pipeline.

In the next section, we’ll see how we can automate this build.

Automating the manual build with WebHooks

What is a WebHook?

If you are familiar with RESTful APIs, then webhooks are noshing more than a POST request made by a web service to another service. In our case, the web service which is creating this request is GitHub and the service where this request is made to is our Jenkins server running on EC2.

For a more formal definition from GeeksForGeeks:

Webhooks allow interaction between web-based applications through the use of custom callbacks. The use of webhooks allows web applications to automatically communicate with other web apps. Unlike traditional systems where one system (subject) keeps polling another system (observer) for some data, Webhooks allow the observer to push data into the subject’s system automatically whenever some event occurs.

This eliminates the need for constant checking to be done by the subject. Webhooks operate entirely over the internet and hence, all communication between systems must be in the form of HTTP messages.

By this definition, I got to understand that instead of our Jenkins polling GitHub for updates at intervals, GitHub notifies Jenkins when a new update comes in. This is more efficent.

Setup Webhook on GitHub repo

As the title says, we are going to make a connection between Github and Jenkins. Go to your repo’s settings and look for Webhooks in the left panel.

https://github.com/<user>/<repo>/settings/hooks. For example, when I go to https://github.com/santosh/cotu/settings/hooks, I see this:

Add webhook screen GitHub
Add webhook screen GitHub (click to zoom)

Click on “Add webhook” to add a new hook. And now you’ll be presented with a form. Input as directed below:

  1. Payload URL: The payload URL is divided into 4 parts. Let’s take an example of my Jenkins server.

[JENKINS_URL][/multibranch-webhook-trigger][/invoke][?token=mysecrettoken]

a. JENKINS_URL: This is the host the Jenkins is installed on. It will be different for you. For me, it’s https://ci.santoshk.dev.
b. /multibranch-webhook-trigger: This comes from Multibranch Scan Webhook Trigger we installed. You can read about this on their page.
c. /invoke: This is the endpoint GitHub will hit when there are any changes registered on their side.
d. ?token=mysecrettoken: For sake of security, you can set up a security token. So that only the one who knows the token can trigger a build.

  1. Content type: Set it to application/json.

  2. Which events would you like to trigger this webhook?: Choose Let me select individual events. This will show some more options. Choose Pushes and Pull requests.

Add webhook details
Add webhook details

Keep everything else on the default options and click Add webhook at the bottom.

Test Multibranch Jenkins pipeline

When you push a new update on the repo now, you’ll the builds being taken up.

Builds:

On this page, you’ll see that one build has already been run. This happens are part of the setup. When you click on that branch, you’d see something like this:

Jenkins build
Jenkins build

This page, also shows how much time each stage took.

Logs:

In your lower left that there is a green checkmark and besides that, there is #1. If you click on that, and then select Console Output from the left sidebar. You’d see the actual console output generated by the Jenkins when running this build pipeline.

It will look something like this.

Branch indexing
 > git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-e5842b23d97cc4456ddd15d1a300b011/.git # timeout=10
Setting origin to https://github.com/santosh/cotu
 > git config remote.origin.url https://github.com/santosh/cotu # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git --version # 'git version 2.25.1'
 > git config --get remote.origin.url # timeout=10
 > git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen branch in repository origin/setup-cicd-pipeline
Seen 2 remote branches
Obtained Jenkinsfile from 094a548654f1ee686b034557f2197de945c4ba22
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/example_setup-cicd-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/santosh/cotu
 > git init /var/lib/jenkins/workspace/example_setup-cicd-pipeline # timeout=10
Fetching upstream changes from https://github.com/santosh/cotu
 > git --version # timeout=10
 > git --version # 'git version 2.25.1'
 > git fetch --no-tags --force --progress -- https://github.com/santosh/cotu +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/santosh/cotu # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 094a548654f1ee686b034557f2197de945c4ba22 (setup-cicd-pipeline)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 094a548654f1ee686b034557f2197de945c4ba22 # timeout=10
Commit message: "Add Cleanup stage in Jenkinsfile"
First time build. Skipping changelog.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Init)
[Pipeline] echo
Initializing..
[Pipeline] echo
Running 1 on https://ci.santoshk.dev/
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
Testing..
[Pipeline] echo
Running pytest..
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] echo
Building..
[Pipeline] echo
Running docker build -t sntshk/cotu .
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Publish)
[Pipeline] echo
Publishing..
[Pipeline] echo
Running docker push..
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Cleanup)
[Pipeline] echo
Cleaning..
[Pipeline] echo
Running docker rmi..
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

To summarize, whenever this pipeline is run, Jenkins clones the repo, then runs the pipeline.

Epilogue

We have not seen pushing to DockerHub in this post. But that post is already is in my draft. Stay tuned, and share this post with your circle.

Share on

Santosh Kumar
WRITTEN BY
Santosh Kumar
Santosh is a Software Developer currently working with NuNet as a Full Stack Developer.