Earlier this week the Jenkins infrastructure team identified a successful attack against our deprecated Confluence service. We responded immediately by taking the affected server offline while we investigated the potential impact. At this time we have no reason to believe that any Jenkins releases, plugins, or source code have been affected. Thus far in our investigation, we have learned that the Confluence CVE-2021-26084 exploit was used to install what we believe was a Monero miner in the container running the service. From there an attacker would not be able to access much of our other infrastructure. Confluence did integrate with our integrated identity system which also powers Jira, Artifactory, and numerous other services. The trust and security in Jenkins core and plugin releases is our highest priority. We do not have any indication that developer credentials were exfiltrated during the attack. At the moment we cannot assert otherwise and are therefore assuming the worst. We are taking actions to prevent releases at this time until we re-establish a chain of trust with our developer community. We have reset passwords for all accounts in our integrated identity system. We are improving the password reset system as part of this effort. At this time, the Jenkins infrastructure team has permanently disabled the Confluence service, rotated privileged credentials, and taken proactive measures to further reduce the scope of access across our infrastructure. We are working closely with our colleagues at the Linux Foundation and the Continuous Delivery Foundation to ensure that infrastructure which is not directly managed by the Jenkins project is also scrutinized. In October 2019 we made the Confluence server read-only effectively deprecating it for day-to-day use within the project. At that time, we began migrating documentation and changelogs from the wiki to GitHub repositories. That migration has been ongoing, with hundreds of plugins and many other documentation pages moved from the wiki to GitHub repositories. We are grateful for those of you who followed our responsible disclosure procedure and reached out to us about this vulnerability affecting the Jenkins project. We will continue to take proactive measures to improve the security of our infrastructure and encourage you to follow us on Twitter for further updates.

When I first wrote about Jenkins Evergreen, which was then referred to as "Jenkins Essentials", I mentioned a number of future developments which in the subsequent months have become reality. At this year’s DevOps World - Jenkins World in San Francisco, I will be sharing more details on the philosophy behind Jenkins Evergreen, show off how far we have come, and discuss where we’re going with this radical distribution of Jenkins. As discussed in my first blog post, and JEP-300, the first two pillars of Jenkins Evergreen have been the primary focus of our efforts. Automatically Updated Distribution Perhaps unsurprisingly, implementing the mechanisms necessary for safely and automatically updating a Jenkins distribution, which includes core and plugins, was and continues to be a sizable amount of work. In Baptiste’s talk he will be speaking about the details which make Evergreen "go" whereas I will be speaking about why an automatically updating distribution is important. As continuous integration and continuous delivery have become more commonplace, and fundamental to modern software engineering, Jenkins tends to live two different lifestyles depending on the organization. In some organizations, Jenkins is managed and deployed methodically with automation tools like Chef, Puppet, etc. In many other organizations however, Jenkins is treated much more like an appliance, not unlike the office wireless router. Installed and so long as it continues to do its job, people won’t think about it too much. Jenkins Evergreen’s distribution makes the "Jenkins as an Appliance" model much better for everybody by ensuring the latest feature updates, bug and security fixes are always installed in Jenkins. Additionally, I believe Evergreen will serve another group we don’t adequately serve at the moment: those who want Jenkins to behave much more like a service. We typically don’t consider "versions" of GitHub.com, we receive incremental updates to the site and realize the benefits of GitHub’s on-going development without ever thinking about an "upgrade." I believe Jenkins Evergreen can, and will provide that same experience. Automatic Sane Defaults The really powerful thing about Jenkins as a platform is the broad variety of patterns and practices different organizations may adopt. For newer users, or users with common use-cases, that significant amount of flexibility can result in a paradox of choice. With Jenkins Evergreen, much of the most common configuration is automatically configured out of the box. Included is Jenkins Pipeline and Blue Ocean, by default. We also removed some legacy functionalities from Jenkins while we were at it. We are also utilizing some of the fantastic Configuration as Code work, which recently had its 1.0 release, to automatically set sane defaults in Jenkins Evergreen. Status Quo The effort has made significant strides thus far this year, and we’re really excited for people to start trying out Jenkins Evergreen. As of today, Jenkins Evergreen is ready for early adopters. We do not yet recommend using Jenkins Evergreen for a production environment. If you’re at DevOps World - Jenkins World in San Francisco please come see Baptiste’s talk Wednesday at 3:45pm in Golden Gate Ballroom A. Or my talk at 11:15am in Golden Gate Ballroom B. If you can’t join us here in San Francisco, we hope to hear your feedback and thoughts in our Gitter channel!

Jenkins Essentials has been renamed to Jenkins Evergreen since this was written. It’s been far too long since we posted an update on Jenkins Essentials. While it’s not quite ready for users to start trying it out, we continue hacking away on all manner of changes to support the safe and automatic upgrades of a running Jenkins environment. In the meantime, Jenkins contributor Baptiste Mathus took some time to introduce and demonstrate Jenkins Essentials at the recently held EclipseCon France, From the talk’s abstract: The Jenkins Project is working on providing its users with a brand new, strongly opinionated, and continuously delivered distribution of Jenkins: Jenkins Essentials. Constantly self-updating, including auto-rollback, with an aggressive subset of verified plugins. In this talk, we will detail how this works: how we run and upgrade Jenkins itself. How instances are continuously sending health data back to help automated decision-making about the quality of given new release, and decide to generalize a given version of Jenkins to the whole fleet, or roll it back. We will end giving an overview of the status of the project: how it’s managed in a fully open manner, from design to code and its infrastructure, and all the radical solutions to imagine and the upcoming challenges for the next months. I hope you enjoy the video You can learn more about Jenkins Essentials from GitHub repository, or join us on our Gitter channel.

One common pattern for automated releases I have seen and used relies on Git tags as the catalyst for a release process. The immutable nature of releases and the immutable nature of tags can definitely go hand in hand, but up until few months ago Jenkins Pipeline was not able to trigger effectively off of Git tags. In this post I want to briefly share how to use tags to drive behaviors in Jenkins Pipeline. Consider the following contrived Jenkinsfile, which contains the three basic stages of Build, Test, and Deploy: pipeline { agent any stages { stage('Build') { steps { sh 'make package' } } stage('Test') { steps { sh 'make check' } } stage('Deploy') { when { tag "release-*" } steps { echo 'Deploying only because this commit is tagged...' sh 'make deploy' } } } } Of particular note is the when condition on the "Deploy" stage which is applying the tag criteria. This means the stage would only execute when the Pipeline has been triggered from a tag in Git matching the release-* Ant-style wildcard. In practice, this means that all pull requests, and branch-based Pipeline Runs result in the stage being skipped: When I push a release-1.0 tag, the Pipeline will then be triggerd and run the "Deploy" stage: Out of the box, Pipelines won’t trigger off of the presence of tags, which means that a Multibranch Pipeline must have a configuration update to know that it must Discover Tags. Configuring From the configuration screen of a Multibranch Pipeline (or GitHub Organization Folder), Discovering tags can be enabled by adding the appropriate "Behavior" to the Branch Source configuration: With these changes, the Jenkinsfile in the tagged versions of my source repository can now drive distinct deployment behavior which is not otherwise enabled in the Pipeline.

Jenkins Essentials has been renamed to Jenkins Evergreen since this was written. A couple weeks ago, I wrote about the Jenkins Essentials effort, on which we’ve been making steady progress. Personally, the most exciting challenge of this project is defining the machinery to drive automatic updates of Jenkins Essentials, which viewed from a high level, are classic continuous delivery challenges. In this post, I wanted to dive into a bit of the gritty details of how we’re going to be delivering Jenkins Essentials with automatic updates, which has some really interesting requirements for the development of Jenkins itself. The traditional Jenkins core and plugin development workflow involves a developer working on changes for some amount of time, then when they’re ready, they "create a release" which typically involves publishing artifacts to our Artifactory, and then on a timer (typically every 15 minutes) the Update Center will re-generate a file called update-center.json. Once the new Update Center has been generated, it is published and consumed by Jenkins installations within 24 hours. Of course, only after Jenkins administrators recognize that there is an update available, can they install it. All in all, it can take quite a long time from when a developer publishes a release, to when it is successfully used by an end-user. With our desire to make Jenkins Essentials updates seamless and automatic, the status quo clearly was not going to work. Our shift in thinking has required a couple simultaneous efforts to make this more continuously delivered approach viable. Developer Improvements Starting from the developer’s workflow, Jesse Glick has been working on publishing "incremental builds" of artifacts into a special Maven repository in Artifactory. Much of his work is described in the very thorough Jenkins Enhancement Proposal 305. This support, which is now live on ci.jenkins.io allows plugin developers to publish versioned changes from pull requests and branches to the incrementals repository. Not only does this make it much easier for Jenkins Essentials to deliver changes closer to the HEAD of master branches, it also unlocks lots of flexibility for Jenkins developers who coordinate changes across matrices of plugins and core, as occasionally is necessary for Jenkins Pipeline, Credentials, Blue Ocean, and a number of other foundational components of a modern Jenkins install. In a follow-up blog post, Jesse is going to go into much more detail on some of the access control and tooling changes he had to solve to make this incrementals machinery work. Of course, incremental builds are only a piece of the puzzle, with those artifacts, Jenkins Essentials has to be able to do something useful with them! Update Improvements The number one requirement, from my perspective, for the automatically updated distribution is that it is safe. "Safe" means that a user doesn’t need to be involved in the update process, and if something goes wrong, the instance recovers without the user needing to do anything to remediate a "bad code deploy." In my previous post on the subject, I mentioned Baptiste’s work on Jenkins Enhancement Proposal 302 which describes the "data safety" system for safely applying updates, and in case of failure, rolling back. The next obvious question is "what’s failure?" which Baptiste spent some time exploring and implementing in two more designs: JEP-304: Essentials Client Error Telemetry Logging JEP-306: Essentials Instance Client Health Checking On the server side, of which there is substantial work for Jenkins Essentials, these concepts integrate with the concept of an Update Lifecycle between the server and client. In essence, the server side must be able to deliver the right updates to the right clients, and avoid delivering tainted updates (those with known problems) to clients. While this part of the work is still on-going, tremendous progress has been made over the past couple weeks in ensuring that updates can be safely, securely, and automatically delivered. With the ability to identify "bad code deploys", and having a mechanism for safely rolling back, not only does Jenkins Essentials allow seamless updates, but it enables Jenkins developers to deliver features and bugfixes much more quickly than our current distribution model allows. While Jenkins Essentials does not have a package ready for broad consumption yet, we’re rapidly closing in on the completion of our first milestone which ties all of these automatic update components together and builds the foundation for continuous delivery of all subsequent improvements. You can follow our progress in the jenkins-infra/evergreen repository, or join us in our Gitter chat!

Jenkins Essentials has been renamed to Jenkins Evergreen since this was written. In his presentation at the 2017 Jenkins World Contributor Summit, Kohsuke challenged us to continue the work started with Jenkins 2 of making Jenkins easier to install and easier to use. "A user should be successful with Jenkins in under five minutes and five clicks." At that same Contributor Summit, a few of us discussed the idea of a distribution which had "batteries included", which Andrew proudly named "Jenkins Essentials." At the time I was certainly not as excited about the project as I am now, I thought to myself "we built a Setup Wizard in Jenkins 2, nobody needs a Setup Wizard++." As Kohsuke and I continued to discuss the idea, more and more ideas came up. Towards the end of 2017 the picture became much clearer: Jenkins Essentials would be a comprehensive, low-maintenance distribution to help new and existing users be successful with Jenkins, without needing to be Jenkins experts. This will of course not replace the existing distribution of Jenkins core and its plugins, which allow many of us large amounts of flexibility, but rather it will make Jenkins easier for users who don’t want to "build it themselves." The more I thought about it, the more excited about the idea I became: Jenkins Essentials could open the door to new improvements and features in Jenkins which had been left in the "idea and design" phase going back almost two years! Really, I checked, some of the concepts adopted into the design of Jenkins Essentials were first conceived of in early 2016! Kohsuke briefly discussed the project in his previous blog post but in post I wanted to expand on what Jenkins Essentials is, and our progress has been in its development. What’s in Jenkins Essentials A few months ago I prepared this presentation for the FOSDEM 2018 Jenkins Contributor Summit, which outlines the following "pillars" or Jenkins Essentials, which are also described in JEP-300 : Automatically Updated Distribution Automatic Sane Defaults Connected Obvious Path to User Success Automatically Updated Distribution In order to provide an easier-to-use and easier-to-manage Jenkins environment, Jenkins Essentials will be distributed as an automatically self-updating distribution, containing Jenkins core and a version-locked set of plugins considered "essential." Rather than attempting to mirror the existing Weekly and LTS release lines for core, plus some plugin version matrix, Jenkins Essentials will update in a manner similar to Google Chrome. This automatically updating distribution will mean that Jenkins Essentials will require significantly less overhead to manage, receiving improvements and bug fixes without any user involvement. From the user perspective, their Jenkins will appear to automatically improve over time. There is really interesting work being pioneered by Baptiste Mathus with JEP-302 to ensure that these automatic upgrades can be performed safely. Automatic Sane Defaults Providing a core along with "essential" plugins is a good first step to helping Jenkins users successfully automate their CI/CD workloads, but requires additional "smoothing" over some of the numerous options and configurations plugins. Jenkins Essentials will perform some amount of "automatic environment-based self-configuration." For example, clicking a "Launch Stack" button from the Download page would launch an AWS-flavored Jenkins Essentials which, out of the box attempts to set up AWS-specific configuration with S3 and EC2 services. Connected In order to provide a more seamless experience for end-users, and ensure that Jenkins project developers receive useful error and usage telemetry to drive further improvements in Jenkins, Jenkins Essentials must necessarily be viewed as a " Connected" application. This means some yet-to-be-specified number of server-side applications to coordinate updates, receive and process telemetry, broker 3rd-party service authentications, relay webhooks, etc. Obvious Path to User Success The final pillar in Jenkins Essentials, is to ensure that Jenkins provides an obvious path for a user to configure and use it successfully. This largely entails in-application documentation, examples, and disabling legacy functionality within the application. All with the end goal of preventing users from inadvertently choosing legacy, or poorly supported, options when configuring their CI/CD workloads. Progress thus far Suffice it to say, Jenkins Essentials is a hugely ambitious project! We have been making steady progress however, as you can see in the jenkins-infra/evergreen repository on GitHub. We have been adamantly following the Jenkins Enhancement Proposal process, and have been making sure our designs and implementations are clear as we build them. Thus far we’ve written designs and implemented: JEP-300: Jenkins Essentials JEP-301: Evergreen packaging for Jenkins Essentials JEP-302: Evergreen snapshotting data safety system JEP-303: Evergreen Client Registration and Authentication JEP-304: Essentials Client Error Telemetry Logging Unfortunately we don’t yet have the first parts of the Automatically Updated Distribution working, which means you cannot download Jenkins Essentials today and get started with it. We’re still building the Jenkins-side and server-side components necessary to make the full feedback loop operate, without which we would not be able to safely deliver new upgrades to Jenkins Essentials installations. If you’re interested in getting involved, you can check out our Gitter channel or our Jira issues board. Jenkins Essentials is just one major initiative going on in the Jenkins project this year, so I hope you’re as excited as I am for the future of Jenkins!

At Jenkins World last month, we continued the tradition of "lunch-time demos" in the Jenkins project’s booth which we started in 2016. We invited a number of Jenkins contributors to present brief 10-15 minute demos on something they were working on, or considered themselves experts in. Continuing the post-Jenkins World tradition, we also just hosted a "Jenkins Online Meetup" featuring a selection of those lunch-time demos. I would like to thank Alyssa Tong for organizing this online meetup, Liam Newman for acting as the host, and our speakers: Oleg Nenashev Michael Hüttermann Thorsten Scherler Stephen Donner Mark Waite Keith Zantow Below are some links from the sample projects demonstrated and the direct links to each session. Developing Pipeline Libraries Locally Video link If you have ever tried developing Pipeline Libraries, you may have noticed how long it takes to deploy a new version to server to discover just another syntax error. I will show how to edit and test Pipeline libraries locally before committing to the repository (with Configuration-as-Code and Docker). Slides Source Code Demo container Delivery Pipelines with Jenkins Video link Showing off how to set up holistic Delivery Pipelines with the DevOps enabler tool Jenkins. Demo application Pimp my Blue Ocean Video link How to customize Blue Ocean, where I create a custom plugin and extending Blue Ocean with custom theme and custom components. Presentation and demo code Deliver Blue Ocean Components at the Speed of Light Video link Using storybook.js.org for Blue Ocean frontend to speed up the delivery process - validate with PM and designer the UX. Showing how quickly you develop your components. Presentation and demo code Mozilla’s Declarative + Shared Libraries Setup Video link How Mozilla is using Declarative Pipelines and shared libraries together. Google Doc with links Shared Library source code Documentation for the shared library See also the #fx-test IRC channel on irc.mozilla.org Git Tips and Tricks Video link Latest capabilities in the git plugin, like large file support, reference repositories and some reminders of existing tips that can reduce server load, decrease job time, and decrease disc use. Visual Pipeline Creation in Blue Ocean Video link We will show how to use Blue Ocean to build a real-world continuous delivery pipeline using the visual pipeline editor. We will coordinate multiple components of a web application across test and production environments, simulating a modern development and deployment workflow.

While at Jenkins World, Kohsuke Kawaguchi presented two long-time Jenkins contributors with a " Small Matter of Programming" award: Andrew Bayer and Jesse Glick. "Small Matter of Programming" being: a phrase used to ironically indicate that a suggested feature or design change would in fact require a great deal of effort; it often implies that the person proposing the feature underestimates its cost. — Wikipedia In this context the "Small Matter" relates to Jenkins Pipeline and a very simple snippet of Scripted Pipeline: [1, 2, 3].each { println it } For a long time in Scripted Pipeline, this simply did not work as users would expect it. Originally filed as JENKINS-26481 in 2015, it became one of the most voted for, and watched, tickets in the entire issue tracker until it was ultimately fixed earlier this year. At least some closures are executed only once inside of Groovy CPS DSL scripts managed by the workflow plugin. — Original bug description by Daniel Tschan At a high level, what has been confusing for many users is that Scripted Pipeline looks like a Groovy, it quacks like a Groovy, but it’s not exactly Groovy. Rather, there’s an custom Groovy interpreter ( CPS) that executes the Scripted Pipeline in a manner which provides the durability/resumability that defines Jenkins Pipeline. Without diving into too much detail, refer to the pull requests linked to JENKINS-26481 for that, the code snippet above was particularly challenging to rectify inside the Pipeline execution layer. As one of the chief architects for Jenkins Pipeline, Jesse made a number of changes around the problem in 2016, but it wasn’t until early 2017 when Andrew, working on Declarative Pipeline, started to identify a number of areas of improvement in CPS and provided multiple patches and test cases. As luck would have it, combining two of the sharpest minds in the Jenkins project resulted in the "Small Matter of Programming" being finished, and released in May of this year with Pipeline: Groovy 2.33. Please join me in congratulating, and thanking, Andrew and Jesse for their diligent and hard work smashing one of the most despised bugs in Jenkins history :).