Recent Blog Posts

Today we published a security advisory that mostly informs about issues in Jenkins plugins that have no fixes. What’s going on? The Jenkins security team triages incoming reports both to Jira and our non-public mailing list. Once we’ve determined it is a plugin not maintained by any Jenkins security team members, we try to inform the plugin maintainer about the issue, offering our help in developing, reviewing, and publishing any fixes. Sometimes the affected plugin is unmaintained, or maintainers don’t respond in a timely manner to the notifications or the followup emails we send. In such cases, we publish security advisories informing users about these issues, even if there’s no new release with a fix. Doing so allows administrators to make an informed decision about the continued use of plugins with unresolved security vulnerabilities. Today’s advisory is overwhelmingly such an advisory. See a plugin you love on this list and want to help out? Learn about adopting plugins.

In the Jenkins project, we ask that people report security issues to our private issue tracker. This allows us to review issues and prepare fixes in private, often resulting in better, safer security fixes. As a side effect of that, we also learn about common misconceptions and usability problems related to security in Jenkins. This post is intended to address one of those: The goal and limitations of credentials masking. The Problem One very common example of that is the role of credentials masking in Jenkins, typically involving a pipeline snippet that looks like this: // Scripted // withCredentials([usernamePassword(credentialsId: 'topSecretCredentials', passwordVariable: 'PWD', usernameVariable: 'USR')]) sh './deploy.sh' // requires PWD and USR to be set } // Declarative // Credentials that are in scope are made available to the pipeline without limitation. To prevent accidental exposure in the build log, credentials are masked from regular output, so an invocation of env (Linux) or set (Windows), or programs printing their environment or parameters would not reveal them in the build log to users who would not otherwise have access to the credentials. The misconception here is that Jenkins will prevent other, perhaps deliberate ways to reveal the password. Some examples: // Scripted // withCredentials([usernamePassword(credentialsId: 'topSecretCredentials', passwordVariable: 'PWD', usernameVariable: 'USR')]) sh 'echo $PWD | base64' // will print e.g. dDBwczNjcjN0Cg= which is trivially converted back to the top secret password } // Declarative // // Scripted // withCredentials([usernamePassword(credentialsId: 'topSecretCredentials', passwordVariable: 'PWD', usernameVariable: 'USR')]) sh 'echo $PWD > myfile' archiveArtifacts 'myfile' // then browse archived artifacts from the Jenkins UI } // Declarative // Both of these snippets circumvent credentials masking in the build log, and show that people with control over the build script can use credentials in ways not necessarily intended or approved by admins. Obviously these are just the most straightforward examples illustrating the problem. Others could involve the proc file system, sending it to an HTTP server in response to a 401 authentication challenge, embedding it in the (otherwise legitimate) build result, etc. It would be great if Jenkins could allow the flexible use of credentials with no risk of exposing them through straightforward build script modifications, but realistically, it is impossible for Jenkins to police use of the credential by a build script without the support of a very specific environment setup (e.g. restrictive network configuration). It should also be noted that credentials aren’t just at risk from users able to control the pipeline, typically by editing the Jenkinsfile. Actual build scripts invoked by pipelines, either shell scripts as in the example above, or more standard build tools such as Maven (controlled by pom.xml) are just as much of a risk if they are run inside a withCredentials block, or executing on the same agent as another block that passed such credentials. Disclosure of secrets can also happen inadvertently: Jenkins will prevent exact matches of the password or other secret to appear in the log file. Consider that the secret may contain shell metacharacters that bash +x would escape by adding a \ before those characters. The sequence of characters to be printed is no longer identical to the secret, so would not be masked. The Solution Credentials can be defined in different scopes: Credentials defined on the root Jenkins store (the default) will be available to all jobs on the instance. The only exception are credentials with System scope, intended for the global configuration only, for example, to connect to agents. Credentials defined in a folder are only available within that folder (transitively, i.e. also in folders inside this folder). This allows defining sensitive credentials, such as deployment credentials, on specific folders whose contents only users trusted with those credentials are allowed to configure: Directly in Jenkins using Matrix Authorization Plugin and by limiting write access to repositories defining pipelines as code. Pipelines inside this folder can use the (e.g. deployment) credentials without limitation, while they’re inaccessible to pipelines outside the folder. Those would need to use the build step or similar approaches to invoke the pipelines inside the folder to deploy their output. Caveats While the previous section outlines a solution to the problem of restricting access to credentials, care needs to be taken so that credentials are not captured anyway. For example, a deployment pipeline that allows its users to define where to deploy to as a build parameter might still be used to send credentials to a maliciously set up host to capture them. A blog post explaining the design of some Jenkins project infrastructure discusses some of these concerns around trust. It should also be noted that credential domains are a UI hint only — defining a credential to only be valid for github.com does not actually prevent its use elsewhere.

Close to two years ago, we announced in New, safer CLI in 2.54 that the traditional “Remoting” operation mode of the Jenkins command-line interface was being deprecated for a variety of reasons, especially its very poor security record. Today in Jenkins 2.165 support for this mode is finally being removed altogether, in both the server and bundled jenkins-cli.jar client. The projected June 5th LTS release will reflect this removal, at which point the Jenkins project will no longer maintain this feature nor investigate security vulnerabilities in it. This change makes the code in Jenkins core related to the CLI considerably simpler and more maintainable. (There are still two transports —HTTP(S) and SSH—but they have similar capabilities and behavior.) It also reduces the “attack surface” the Jenkins security team must consider. Among other issues, a compromised server could freely attack a developer’s laptop if -remoting were used. The 2.46.x upgrade guide already urged administrators to disable Remoting mode on the server. Those Jenkins users who rely on the CLI for remote scripting (as opposed to the HTTP(S) REST APIs) would be affected only if they were still using the -remoting CLI flag, since the default has long been to use HTTP(S) mode. Most CLI features have long worked fine without -remoting, in some cases using slightly different syntax such as requiring shell redirects to access local files. As part of this change, some CLI commands, options, and option types in Jenkins core have been removed, other than -remoting itself: The login and logout commands, and the --username and --password options. The -p option to select a proxy. (The CLI in default -http mode accesses Jenkins no differently than any other HTTP client.) The install-tool, set-build-parameter, and set-build-result commands relied on a fundamentally insecure idiom that is no longer supportable. Command options or arguments which took either a local file or = for standard input/output (e.g., install-plugin, build -p, support) now only accept the latter. Some features of relatively little-used plugins will no longer work, such as: DistFork Remote Terminal Access Build Env Propagator

In this post I wanted to show how you can run Jenkins behind a firewall (which could be a corporate firewall, a NAT’ed network like you have at home) but still receive webhooks in real time from GitHub.com. You can generalise this to other services too - such as BitBucket or DockerHub, or anything really that emits webhooks, but the instructions will be for GitHub projects hosted on github.com. What are webhooks Just a very quick refresher on what webhooks are: Messages (often JSON, but not always) typically posted by HTTP(S) from a server to a client that is listening for events. The events flow left to right, Jenkins sits there happily listing on paths like /github-webhook/ or /dockerhub-webhook/ etc for some HTTP request to tell it to wake up and do some work. GitHub/BitBucket may be reporting a new commit or PR, or DockerHub reporting an upstream image has changed. What all these things have in common is that they push to Jenkins, and expect to be able to push to it (ie that Jenkins is visible to them). This works great when the network is open - say GitHub Enterprise, or Jenkins is listening on the web. Not on the web The trick is when something gets in the middle, say a firewall: ( As is industry standard, all firewalls have to be a wall on fire. Please don’t somehow set bricks on fire in your organisation) This is just the same when you fire up Jenkins on your laptop, and want to receive webhooks from github.com (a legitimate thing, perhaps to test out your setup, perhaps to run builds for iOS on a mac, or some corner of a network that is not exposed to the web). Unless your laptop is addressable to the whole web that is (not likely), or your network is configured just right, the webhooks won’t be able to flow. This is fine - we can fall back to polling for changes. Except this is terrible. You burn through API quotas, and you don’t get changes in real time, and really no one is happy. Some problems are opportunities We can both solve this problem, but also, view this is an opportunity. Having things not addressable on the web, or locked down in some default way is a feature, not a bug. You massively reduce your attack surface, and can have defence in depth: A Webhook forwarding service Enter the memorably named Smee. This is an OSS project provided by GitHub and also helpfully hosted as a service by GitHub. This can capture and forward webhooks for you. I’ll try to explain it with a diagram: GitHub pushes an event (via HTTPS/json in this case) to Smee.io (the funny thing with circles, which is on the public web and accessible from GitHub.com) - and Jenkins in turn subscribes to Smee with an outgoing connection from a client. Note the direction of the arrows: Jenkins only makes an outbound connection. This is the important point: this will work as long as the firewall is one way (like a NAT typically is, and many networks). If the Jenkins side can’t connect to anything on the outside world - well, this won’t help with that of course (but that is not often the case). Setting it up Step 1: Firstly - go to https://smee.io/ and click “Start a new channel”: This will give you a unique URL (which you should copy for later use): Next you should install the smee client next to where you have the Jenkins server running: npm install --global smee-client (This will make the smee client/command available to receive and forward webhooks). Now start the smee client and point it to your Jenkins server. In this case I have it running on port 8080 (the default if you fire it up on your laptop, change both the port and the smee URL as needed): smee --url https://smee.io/GSm1B40sRfBvSjYS --path /github-webhook/ --port 8080 This says to connect to the smee service, and forward webhooks to /github-webhook/ (that trailing slash is important, don’t miss it). Once this is running, you will see it log that it is connected and forwarding webhooks. Leave this command running for as long as you want to receive webhooks. Next, you need to configure a pipeline that makes use of github. In this case I set up one from scratch. You can skip this if you already have a pipeline setup: I then chose “GitHub” as the where the code is: Then choose your repository. This will set things up ready to receive webhooks from GitHub. (also if you have an existing pipeline setup, and it is using GitHub as the SCM source, that is also fine). The final step is to tell GitHub to post webhook events for that repository (or organization, you can do that too) to Smee (which ultimately means Jenkins will receive them). Go to the settings tab for your GitHub repository, and then click “add webhook”: Next, configure the webhook: Paste in the “smee” URL you copied from the step above. Choose application/json as the content type Tell it to send everything (you can pick and choose what events, but I just did that as simpler). Press Add Webhook (or update) It should look something like this: OK - webhooks should be flowing now. You can make a change to your repository, and check that a build starts soon after: Good luck!

We just released security updates to Jenkins, versions 2.154 and LTS 2.150.1, that fix multiple security vulnerabilities. Since 2.150.1 is the first release in the new LTS line, we also released 2.138.4, a security update for the previous LTS line. This allows administrators to install today’s security fixes without having to upgrade to the new LTS line immediately. For an overview of what was fixed, see the security advisory. For an overview on the possible impact of these changes, see our LTS 2.138.4 upgrade guide. A note on previously released changes related to this fix In the Jenkins core security updates released in August and October, we also included security improvements that can be disabled by setting various system properties. Those changes are an essential part of the SECURITY-595 fix, so we strongly recommend not disabling them for any reason. Previously published documentation has been updated.

In a past blog post, Delivery Pipelines, with Jenkins 2, SonarQube, and Artifactory, we talked about pipelines which result in binaries for development versions, and in Delivery pipelines, with Jenkins 2: how to promote Java EE and Docker binaries toward production, we examined ways to consistently promote applications toward production. In this blog post, I continue on both by discussing more details on security related quality gates and bringing this together with the handling of Docker images. Use case: Foster security on given, containerized business application Security is an overloaded term with varying meaning in different contexts. For this contribution, I consider security as the sum of rules regarding vulnerabilities (Common Vulnerability and Exposure, CVE), in binaries. In a past blog post, we’ve identified SonarQube already, as a very helpful tool to identify flaws in source code, particularly concerning reliability (bugs), vulnerabilities (security, e.g. CWE, that is common weakness enumaration, and OWASP, that is the Open Web Application Security Project), and maintainability (code smells). Now it is a good time to add another tool to the chain, that is Twistlock, for inspection binaries for security issues. Features of Twistlock include Compliance and vulnerability management, transitively Runtime defense Cloud-native CI/CD support Broad coverage of supported artifact types and platforms API, dashboards, and Jenkins integration, with strong configuration options The underlying use case can be derived from several real-world security initiatives, in enterprises, based on given containerized applications. In practice, it is not a surprise that after adding such new quality gates, you identify historically grown issues. However, there are many good reasons to do so. You don’t need any Word documents to check any governance criteria manually, rather execution and reporting are done automatically and also part of the actions are taken automatically. And above all, of course, your application is quality assured regarding known vulnerability issues, aligned with the DevOps approach: development is interested in quick feedback whether their change would introduce any vulnerabilities, and operations is interested in insights whether and how running applications are affected if a new CVE is discovered. The term DevSecOps was coined to explicitely add security concerns to DevOps. In my opinion, security is already inherent part of DevOps. Thus, there is no strong reason to introduce a new word. Surely, new words are catchy. But they have limits. Or have you ever experienced NoDev, the variant of DevOps where features are suddenly falling from the sky and deployed to production automatically? Conceptually, container inspection is now part of the delivery pipeline and Twistlock processing is now triggered once we have produced our Docker images, see below, in order to get fast feedback. Software is staged over different environments by configuration, without rebuilding. All changes go through the entire staging process, although defined exception routines may be in place, for details see Michael Hüttermann, Agile ALM (Manning, 2012). The staged software consists of all artifacts which make up the release, consistently, including the business application, test cases, build scripts, Chef cookbooks, Dockerfiles, Jenkins files to build all that in a self-contained way, for details see Michael Hüttermann, DevOps for Developers (Apress, 2012). This blog post covers sample tools. Please note, that there are also alternative tools available, and the best target architecture is aligned with concrete requirements and given basic conditions. Besides that, the sample toolchain is derived from couple of real world success stories, designed and implemented in the field. However, this blog post simplifies and abstracts them in order to stay focussed while discussing the primitives of delivery units. For example, aggregating multiple Docker images with ASCII files, does not change the underlying primitives and their handlings. For more information on all parts of the blog post, please consult the respective documentation, good books or attend fine conferences. Or go to the extremes: talk to your colleagues. In our sample process, we produce a web application that is packaged in a Docker image. The produced Docker images are distributed only if the dedicated quality gate passes. A quality gate is a stage in the overall pipeline and a sum of defined commitments, often called requirements, the unit of work must pass. In our case, the quality gate comprises inspection of produced binaries and it fails if vulnerabilities of severity 'critical' are found. We can configure Twistlock according to our requirements. Have a look how we’ve integrated it into our Jenkins pipeline, with focus on detecting vulnerabilities. Jenkinsfile (excerpt): Twistlock inspection triggered stage('Twistlock: Analysis') { (1) String version = readFile('version.properties').trim() (2) println "Scanning for version: ${version}" twistlockScan ca: '', cert: '', compliancePolicy: 'critical', \ dockerAddress: 'unix:///var/run/docker.sock', \ ignoreImageBuildTime: false, key: '', logLevel: 'true', \ policy: 'critical', repository: 'huttermann-docker-local.jfrog.io/michaelhuettermann/alpine-tomcat7', \ (3) requirePackageUpdate: false, tag: "$version", timeout: 10 } stage('Twistlock: Publish') { (4) String version = readFile('version.properties ').trim() println "Publishing scan results for version: ${version}" twistlockPublish ca: '', cert: '', \ dockerAddress: 'unix:///var/run/docker.sock', key: '', \ logLevel: 'true', repository: 'huttermann-docker-local.jfrog.io/michaelhuettermann/alpine-tomcat7', tag: "$version", \ timeout: 10 } 1 Twistlock inspection as part of the sequence of stages in Jenkinsfile 2 Nailing down the version of the to be inspected image, dynamically 3 Configuring analysis including vulnerability severity level 4 Publishing the inspection results to Twistlock console, that is the dashboard Now let’s start with the first phase to bring our application in shape again, that is gaining insight about the security related flaws. Phase 1: Gain insights about security related flaws After we’ve introduced the new quality gate, it failed, see image above. As integration with other tools, Jenkins is the automation engine and does provide helpful context information, however, those cannot replace features and data the dedicated, triggered tool does offer. Thus, this is the moment to switch to the dedicated tool, that is Twistlock. Opening the dashboard, we can navigate to the Jenkins build jobs, that is the specific run of the build, and the respective results of the Twistlock analysis. What we see now is a list of vulnerabilities, and we need to fix those of severity critical in order to pass the quality gate, and get our changes again toward production. The list shows entries of type jar, that is a finding in a binary as part of the Docker image, in our case the WAR file we’ve deployed to a web container (Tomcat), and of type OS, those are issues of the underlying image itself, the operating system, either part of the base image, or as a package added/changed in our Dockerfile. We can now easily zoom in and examine the vulnerabilities of the Docker layers. This really helps to structure work and identify root causes. Since, typically, a Docker image extends a Docker base image, the findings in the base image are shown on the top, see next screenshot, grouped by severity. Other Docker layers were added to the base image, and those can add vulnerabilities too. In our case, the packaged WAR file obviously contains a vulnerability. The next image shows how we examine that finding, while this time expanding the Twistlock wizard (that is the plus sign) to directly see the list of found vulnerabilities. Finding and visualizing the issues are a very good first step, and we’ve even made those findings actionable, so we now have to take action and address them. Phase 2: Address the findings To address the findings, we need to split our initiative into two parts: Fixing the critical vulnerabilities related to the Docker image (in our case largely the base image) Fixing the critical vulnerabilities related to the embedded deployment unit (in our case the WAR) Let’s proceed bottom up, first coping with the Docker base image. This is an easy example covering multiple scenarios particularly identifying and fixing vulnerabilities in transitive binaries, i.e. binaries contained in other binaries, e.g. a Docker image containing a WAR file that in turn contains libraries. To expand this vertical feasibility spike, you can easily add more units of each layer, or add more abstractions, however, the idea can always be nailed down to the primitives, covered in this blog post. Let’s now have a look at the used Docker image by looking at the used Dockerfile. Dockerfile: The Dockerfile based on Alpine, running OpenJDK 8 FROM openjdk:8-jre-alpine (1) LABEL maintainer "michael@huettermann.net" # Domain of your Artifactory. Any other storage and URI download link works, just change the ADD command, see below. ARG ARTI ARG VER # Expose web port EXPOSE 8080 # Tomcat Version ENV TOMCAT_VERSION_MAJOR 9 (2) ENV TOMCAT_VERSION_FULL 9.0.6 # Download, install, housekeeping RUN apk add --update curl &&\ (3) apk add bash &&\ #apk add -u libx11 &&\ (4) mkdir /opt &&\ curl -LO ${ARTI}/list/generic-local/apache/org/tomcat/tomcat-${TOMCAT_VERSION_MAJOR}/v${TOMCAT_VERSION_FULL}/bin/apache-tomcat-${TOMCAT_VERSION_FULL}.tar.gz &&\ gunzip -c apache-tomcat-${TOMCAT_VERSION_FULL}.tar.gz | tar -xf - -C /opt &&\ rm -f apache-tomcat-${TOMCAT_VERSION_FULL}.tar.gz &&\ ln -s /opt/apache-tomcat-${TOMCAT_VERSION_FULL} /opt/tomcat &&\ rm -rf /opt/tomcat/webapps/examples /opt/tomcat/webapps/docs &&\ apk del curl &&\ rm -rf /var/cache/apk/* # Download and deploy the Java EE WAR ADD http://${ARTI}/list/libs-release-local/com/huettermann/web/${VER}/all-${VER}.war /opt/tomcat/webapps/all.war (5) RUN chmod 755 /opt/tomcat/webapps/*.war # Set environment ENV CATALINA_HOME /opt/tomcat # Start Tomcat on startup CMD ${CATALINA_HOME}/bin/catalina.sh run 1 Base image ships OpenJDK 8, on Alpine 2 Defined version of web container 3 Applying some defined steps to configure Alpine, according to requirements 4 Updating package itself would address one vulnerability already 5 Deploying the application By checking available versions of the official OpenJDK Alpine image, we see that there’s a newer version 8u181 which we could use. We can zoom in and study release notes and contents, or we just pragmatically switch the base image to a more recent version. Often it is a good idea to upgrade versions regularly, in defined intervals. This leads to the following change in the Dockerfile. Dockerfile (excerpt): The Dockerfile based on Alpine, running OpenJDK 8u181 FROM openjdk:8u181-jre-alpine (1) LABEL maintainer "michael@huettermann.net" 1 Base image is now OpenJDK 8u181, on Alpine There are more options available to fix the issues, but let’s proceed to the second part, the vulnerabilities in the deployment unit. Before we push this change to GitHub, we also address the vulnerability issue in the deployment unit, that is jetty-io. Here we are a bit unsure about why, in this specific use case, the library is used. To retrieve more information about dependencies, we run a dependency:tree command on our Maven based project. We now see that jetty-io is transitively referenced by org.seleniumhq.selenium:htmlunit-driver. We can surely discuss why this is a compile dependency and the libraries are shipped as part of the WAR, but let’s consider this to be given according to requirements, thus we must take special attention now to version 2.29.0 of the specific library. Also here we can browse release notes and content (particularly how those libs are built themselves), and come to the conclusion to switch from the used version, that is 2.29.0, to a newer version of htmlunit-driver, that is 2.31.1. pom.xml (excerpt): Build file (1) org.seleniumhq.selenium selenium-java 3.14.0 org.seleniumhq.selenium (2) htmlunit-driver 2.31.1 junit junit 4.7 1 Part of the underlying POM defining dependencies 2 Definition of the dependency, causing the vulnerability finding; we use a newer version now OK, now we are done. We push the changes to GitHub, and our GitHub webhook directly triggers the workflow. This time the quality gate passes, so it looks like our fixes did address the root causes and eliminated those with the configured threshold severity. Finally, after running through our entire workflow, that is made up of different pipelines, our inspected and quality assured container does successfully run in our production runtime environment, that is on Oracle Cloud. Crisp, isn’t it? Summary This closes our quick walkthrough of how to inject security related quality gates into a Jenkins based delivery pipeline. We’ve discussed some concepts and how this can look like with sample tools. In the center of our efforts, we used Jenkins, the swiss army knife of automation. We enriched our ecosystem by integrating couple of platforms and tools, above all Twistlock. After this tasty appetizer you are ready to assess your own delivery pipelines, concepts and tools, and to possibly invest even more attention to security. References 'Agile ALM', Manning, 2011 'DevOps for Developers', Apress, 2012 Docker, the standard to develop and ship set of changes Docker images, shipping OpenJDK Oracle Cloud Infrastructure, for containers Alpine Linux SonarQube, the language/platform agnostic Continuous Inspection tool Twistlock, the container security platform Sources on GitHub ASCII, commonly used standard to work on primitives, such as Docker (and their aggregations) Common Vulnerabilities and Exposures Holistic pipelines, Live 15-minute Jenkins Demos, Part 1, on YouTube Holistic pipelines, Live 15-minute Jenkins Demos, Part 2, on YouTube Delivery Pipelines, with Jenkins 2, SonarQube, and Artifactory Delivery pipelines, with Jenkins 2: how to promote Java EE and Docker binaries toward production

We just released security updates to Jenkins, versions 2.146 and 2.138.2, that fix multiple security vulnerabilities. For an overview of what was fixed, see the security advisory. For an overview on the possible impact of these changes on upgrading Jenkins LTS, see our LTS upgrade guide. Further improvements In addition to the security fixes listed in the security advisory, we also applied multiple improvements that make future security vulnerabilities more difficult, or even impossible to exploit. One such improvement concerns cross-site scripting vulnerabilities, and comes with a risk of regressions. Jenkins uses a fork of Jelly for the vast majority of the views it renders. Since 2011, it includes a feature that lets view authors opt in or out of automatic escaping of variable values for rendering in HTML, and since 2016, the plugin build tooling requires that views explicitly specify whether to apply this automatic escaping. Details are available in the developer documentation. Until now, if views do not declare whether to automatically escape, they were rendered without automatic escaping, and developers were expected to explicitly escape every variable reference that was not supposed to contain markup. This has resulted in a number of cross-site scripting (XSS) vulnerabilities, most recently SECURITY-1130 in Job Config History Plugin. For that reason, we have decided to enable this automatic escaping by default if plugins do not specify a preference. This can result in problems with some plugins if they need their output to remain unescaped. We expect that those plugins will adapt pretty quickly to this change, as the fix is typically straightforward. We track known affected plugins and their status on the Jenkins wiki. In the mean time, users can set the system property org.kohsuke.stapler.jelly.CustomJellyContext.escapeByDefault to false to disable this additional protection.

We just released security updates to Jenkins, versions 2.138 and 2.121.3, that fix multiple security vulnerabilities. For an overview of what was fixed, see the security advisory. For an overview on the possible impact of these changes on upgrading Jenkins LTS, see our LTS upgrade guide. Subscribe to the jenkinsci-advisories mailing list to receive important notifications related to Jenkins security.