Recent Blog Posts

Table of Contents The Problem Code Duplication And Security A Branch Is Not A Feature Documentation The Solution Code Duplication And Security A Branch Is Not A Feature Documentation This post will describe some common problems I’ve had with Jenkins and how I solved them by developing Generic Webhook Trigger Plugin. The Problem I was often struggling with the same issues when working with Jenkins: Code duplication and security - Jenkinsfiles in every repository. A branch is not a feature - Parameterized jobs on master branch often mix parameters relevant for different features. Poorly documented trigger plugins - Proper documented services but poorly documented consuming plugins. Code Duplication And Security Having Jenkinsfiles in every Git repository allows developers to let those files diverge. Developers pushes forward with their projects and it is hard to maintain patterns to share code. I have, almost, solved code duplication with shared libraries but it does not allow me to setup a strict pattern that must be followed. Any developer can still decide to not invoke the features provided by the shared library. There is also the security aspect of letting developers run any code from the Jenkinsfiles. Developers might, for example, print passwords gathered from credentials. Letting developers execute any code on the Jenkins nodes just does not seem right to me. A Branch Is Not A Feature In Bitbucket there are projects and each project has a collection of git repositories. Something like this: PROJ_1 REPO_1 REPO_2 PROJ_2 REPO_3 Lets think about some features we want to provide for these repositories: Pull request verification Building snapshot (or pre release if you will) Building releases If the developers are use to the repositories being organized like this in Bitbucket, should we not organize them the same way in Jenkins? And if they browse Jenkins should they not find one job per feature, like pull-request, snapshot and release? Each job with parameters only relevant for that feature. I think so! Like this: / - Jenkins root /PROJ_1 - A folder, lists git repositories /PROJ_1/REPO_1 - A folder, lists jobs relevant for that repo. /PROJ_1/REPO_1/release - A job, performs releases. /PROJ_1/REPO_1/snapshot - A job, performs snapshot releases. /PROJ_1/REPO_1/pull-request - A job, verifies pull requests. …​ In this example, both snapshot and release jobs might work with the same git branch. The difference is the feature they provide. Their parameters can be well documented as you don’t have to mix parameters relevant for releases and those relevant for snapshots. This cannot be done with Multibranch Pipeline Plugin where you specify parameters as properties per branch. Documentation Webhooks are often well documented in the services providing them. See: Bitbucket Cloud Bitbucket Server GitHub GitLab Gogs and Gitea Assembla Jira It bothered me that, even if I understood these webhooks, I was unable to use them. Because I needed to perform development in the plugin I was using in order to provide whatever value from the webhook to the build. That process could take months from PR to actual release. Such a simple thing should really not be an issue. The Solution My solution is pretty much back to basics : We have an automation server (Jenkins) and we want to trigger it on external webhooks. We want to gather information from that webhook and provide it to our build. In order to support it I have created the Generic Webhook Trigger Plugin. The latest docs are available in the repo and I also have a fully working example with GitLab implemented using configuration-as-code. See the repository here. Code Duplication And Security I establish a convention that all developers must follow. Instead of letting the developers explicitly invoke the infrastructure from Jenkinsfiles. There are rules to follow, like: All git repositories should be built from the root of the repo. If it contains a gradlew Build is done with./gradlew build Release is done with./gradlew release …​ and so on If it contains a package.json Build is done with npm run build Release is done with npm run release …​ and so on With these rules, pipelines can be totally generic and no Jenkinsfiles are needed in the repositories. Some git repositories may, for some reason, need to disable test cases. That can be solved by allowing repositories to add a special file, perhaps jenkins-settings.json, let the infrastructure discover and act on its content. This also helps the developers even when not doing CI. When they clone a new, to them unknown, repository they will know what commands can be issued and their semantics. A Branch Is Not A Feature I implement: Jenkins job configurations - With Job DSL. Jenkins build process - With Pipelines and Shared Library. By integrating with the git service from Job DSL I can automatically find the git repositories. I create jobs dynamically organized in folders. Also invoking the git service to setup webhooks triggering those jobs. The jobs are ordinary pipelines, not multibranch, and they don’t use Jenkinsfile from Git but instead Jenksinfile configured in the job using Job DSL. So that all job configurations and pipelines are under version control. This is all happening here. Documentation The plugin uses JSONPath, and also XPath, to extract values from JSON and provide them to the build. Letting the user pick whatever is needed from the webhook. It also has a regular expression filter to allow not triggering for some conditions. The plugin is not very big, just being the glue between the webhook, JSONPath / XPath and regular expression. All these parts are very well documented already and I do my best supporting the plugin. That way this is a very well documented solution to use!

In this post I wanted to show how you can run Jenkins behind a firewall (which could be a corporate firewall, a NAT’ed network like you have at home) but still receive webhooks in real time from GitHub.com. You can generalise this to other services too - such as BitBucket or DockerHub, or anything really that emits webhooks, but the instructions will be for GitHub projects hosted on github.com. What are webhooks Just a very quick refresher on what webhooks are: Messages (often JSON, but not always) typically posted by HTTP(S) from a server to a client that is listening for events. The events flow left to right, Jenkins sits there happily listing on paths like /github-webhook/ or /dockerhub-webhook/ etc for some HTTP request to tell it to wake up and do some work. GitHub/BitBucket may be reporting a new commit or PR, or DockerHub reporting an upstream image has changed. What all these things have in common is that they push to Jenkins, and expect to be able to push to it (ie that Jenkins is visible to them). This works great when the network is open - say GitHub Enterprise, or Jenkins is listening on the web. Not on the web The trick is when something gets in the middle, say a firewall: ( As is industry standard, all firewalls have to be a wall on fire. Please don’t somehow set bricks on fire in your organisation) This is just the same when you fire up Jenkins on your laptop, and want to receive webhooks from github.com (a legitimate thing, perhaps to test out your setup, perhaps to run builds for iOS on a mac, or some corner of a network that is not exposed to the web). Unless your laptop is addressable to the whole web that is (not likely), or your network is configured just right, the webhooks won’t be able to flow. This is fine - we can fall back to polling for changes. Except this is terrible. You burn through API quotas, and you don’t get changes in real time, and really no one is happy. Some problems are opportunities We can both solve this problem, but also, view this is an opportunity. Having things not addressable on the web, or locked down in some default way is a feature, not a bug. You massively reduce your attack surface, and can have defence in depth: A Webhook forwarding service Enter the memorably named Smee. This is an OSS project provided by GitHub and also helpfully hosted as a service by GitHub. This can capture and forward webhooks for you. I’ll try to explain it with a diagram: GitHub pushes an event (via HTTPS/json in this case) to Smee.io (the funny thing with circles, which is on the public web and accessible from GitHub.com) - and Jenkins in turn subscribes to Smee with an outgoing connection from a client. Note the direction of the arrows: Jenkins only makes an outbound connection. This is the important point: this will work as long as the firewall is one way (like a NAT typically is, and many networks). If the Jenkins side can’t connect to anything on the outside world - well, this won’t help with that of course (but that is not often the case). Setting it up Step 1: Firstly - go to https://smee.io/ and click “Start a new channel”: This will give you a unique URL (which you should copy for later use): Next you should install the smee client next to where you have the Jenkins server running: npm install --global smee-client (This will make the smee client/command available to receive and forward webhooks). Now start the smee client and point it to your Jenkins server. In this case I have it running on port 8080 (the default if you fire it up on your laptop, change both the port and the smee URL as needed): smee --url https://smee.io/GSm1B40sRfBvSjYS --path /github-webhook/ --port 8080 This says to connect to the smee service, and forward webhooks to /github-webhook/ (that trailing slash is important, don’t miss it). Once this is running, you will see it log that it is connected and forwarding webhooks. Leave this command running for as long as you want to receive webhooks. Next, you need to configure a pipeline that makes use of github. In this case I set up one from scratch. You can skip this if you already have a pipeline setup: I then chose “GitHub” as the where the code is: Then choose your repository. This will set things up ready to receive webhooks from GitHub. (also if you have an existing pipeline setup, and it is using GitHub as the SCM source, that is also fine). The final step is to tell GitHub to post webhook events for that repository (or organization, you can do that too) to Smee (which ultimately means Jenkins will receive them). Go to the settings tab for your GitHub repository, and then click “add webhook”: Next, configure the webhook: Paste in the “smee” URL you copied from the step above. Choose application/json as the content type Tell it to send everything (you can pick and choose what events, but I just did that as simpler). Press Add Webhook (or update) It should look something like this: OK - webhooks should be flowing now. You can make a change to your repository, and check that a build starts soon after: Good luck!