We recently released version 1.3 of Declarative Pipelines, which includes a couple significant new features. We’re going to cover these features in separate blog posts. The next post will show the new ability to restart a completed Pipeline run starting from a stage partway through the Pipeline, but first, let’s look at the new sequential stages feature. Sequential Stages In Declarative 1.2, we added the ability to define stages to run in parallel as part of the Declarative syntax. Now in Declarative 1.3, we’ve added another way to specify stages nested within other stages, which we’re calling "sequential stages". Running Multiple Stages in a Parallel Branch One common use case is running build and tests on multiple platforms. You could already do that with parallel stages, but now you can run multiple stages in each parallel branch giving you more visibility into the progress of your Pipeline without having to check the logs to see exactly which step is currently running where, etc. You can also use stage directives, including post, when, agent, and all the others covered in the Pipeline Syntax reference in your sequential stages, letting you control behavior for different parts of each parallel branch. In the example below, we are running builds on both Windows and Linux, but only want to deploy if we’re on the master branch. pipeline { agent none stages { stage("build and deploy on Windows and Linux") { parallel { stage("windows") { agent { label "windows" } stages { stage("build") { steps { bat "run-build.bat" } } stage("deploy") { when { branch "master" } steps { bat "run-deploy.bat" } } } } stage("linux") { agent { label "linux" } stages { stage("build") { steps { sh "./run-build.sh" } } stage("deploy") { when { branch "master" } steps { sh "./run-deploy.sh" } } } } } } } } Running Multiple Stages with the Same agent, or environment, or options While the sequential stages feature was originally driven by users wanting to have multiple stages in parallel branches, we’ve found that being able to group multiple stages together with the same agent, environment, when, etc has a lot of other uses. For example, if you are using multiple agents in your Pipeline, but would like to be sure that stages using the same agent use the same workspace, you can use a parent stage with an agent directive on it, and then all the stages inside its stages directive will run on the same executor, in the same workspace. Another example is that until now, you could only set a timeout for the entire Pipeline or an individual stage. But by using a parent stage with nested stages, you can define a timeout in the parent’s options directive, and that timeout will be applied for the execution of the parent, including its nested stages. You may also want to conditionally control the execution of multiple stages. For example, your deployment process may be spread across multiple stages, and you don’t want to run any of those stages unless you’re on a certain branch or some other criteria is satisified. Now you can group all those related stages together in a parent stage, within its stages directive, and have a single when condition on that parent, rather than having to copy an identical when condition to each of the relevant stages. One of my favorite use cases is shown in the example below. In Declarative 1.2.6, we added the input directive for stages. This will pause the execution of the Pipeline until a user confirms that the Pipeline should continue, using the Scripted Pipeline input step. The input directive is evaluated before the stage enters its agent, if it has one specified, and before the stage’s when condition, if specified, is evaluated. But if you’re using a top-level agent for most of your stages, you’re still going to be using that agent’s executor while waiting for input, which can be a waste of resources. With sequential stages, you can instead use agent none at the top-level of the Pipeline, and group the stages using a common agent and running before the stage with the input directive together under a parent stage with the required agent specified. Then, when your Pipeline reaches the stage with input, it will no longer be using an agent’s executor. pipeline { agent none stages { stage("build and test the project") { agent { docker "our-build-tools-image" } stages { stage("build") { steps { sh "./build.sh" } } stage("test") { steps { sh "./test.sh" } } } post { success { stash name: "artifacts", includes: "artifacts/**/*" } } } stage("deploy the artifacts if a user confirms") { input { message "Should we deploy the project?" } agent { docker "our-deploy-tools-image" } steps { sh "./deploy.sh" } } } } These are just a few example of the power of the new sequential stages feature in Declarative 1.3. This new feature adds another set of significant use cases that can be handled smoothly using Declarative Pipeline. In my next post, I’ll show the another highly requested feature - the new ability to restart a Pipeline run from any stage in that Pipeline.

Last week we released the latest version of Declarative Pipelines, version 1.2.8. With that out, we thought now would be a good time to introduce you to the new features and options that have been added to Declarative since the beginning of 2018. These are all available now in the Update Center, with version 1.2.8. Declarative Directive Generator This is something we’re really happy about - if you go to the "Pipeline Syntax" link from your Pipeline’s page in Jenkins, you’ll see a couple new links on the left, including "Declarative Directive Generator". The Directive Generator is much like the Snippet Generator that’s been in Pipeline for a couple years now, but where the Snippet Generator is just for filling out a form for a step and generating the Pipeline code that configuration maps to, the Directive Generator is built to help you write your Declarative Pipeline directives, like agent, options, stage, and more! This is the first release to include the Directive Generator, and it’s definitely going to see more polish going forward, but we think it should be quite helpful for you already. We’ll be putting up another blog post looking at the Directive Generator in more detail in the near future. New when conditions We’ve added a number of new when conditions, providing you more control over whether your stages get executed. equals - Compares two values - strings, variables, numbers, booleans - and returns true if they’re equal. I’m honestly not sure how we missed adding this earlier! You can do "not equals" comparisons using the not { equals …​ } combination too. changeRequest - In its simplest form, this will return true if this Pipeline is building a change request, such as a GitHub pull request. You can also do more detailed checks against the change request, allowing you to ask "is this a change request against the master branch?" and much more. buildingTag - A simple condition that just checks if the Pipeline is running against a tag in SCM, rather than a branch or a specific commit reference. tag - A more detailed equivalent of buildingTag, allowing you to check against the tag name itself. In addition, we’ve added a new option to when : beforeAgent. This allows you to specify that the when conditions should be evaluated before entering the agent for the stage, rather than the normal behavior of evaluating when conditions after entering the agent. When beforeAgent true is specified, you will not have access to the agent’s workspace, but you can avoid unnecessary SCM checkouts and waiting for a valid `agent to be available. This can speed up your Pipeline’s execution in some cases. New post conditions The changed condition has always been a bit confusing, and to be honest, it wasn’t our best work. changed will fire any time the current run’s status is different than the previous run’s status - whether the current run is healthier than the previous one, or the other way around. That’s…​not actually very useful. So now we’ve added two new post conditions that should provide you with a lot more value than changed has. fixed - This will check to see if the current run is successful, and if the previous run was either failed or unstable. regression - This will check to see if the current run’s status is worse than the previous run’s status. So if the previous run was successful, and the current run is unstable, this will fire and its block of steps will execute. It will also run if the previous run was unstable, and the current run is a failure, etc. New options The options directive in Declarative can contain a number of different kinds of configuration: traditional Jenkins job properties, like buildDiscarder, wrapper steps to execute the entire Pipeline within, like timeout, and Declarative-specific options that can switch from some default behaviors of Declarative execution. We’ve added two new Declarative-specific options in the last few releases. checkoutToSubdirectory - Allows you to override the location that the automatic SCM checkout will use. Using checkoutToSubdirectory("foo"), your Pipeline will checkout your repository to"$WORKSPACE/foo", rather than the default of"$WORKSPACE". newContainerPerStage - If you’re using a top-level docker or dockerfile agent, and want to ensure that each of your stages run in a fresh container of the same image, you can use this option. Any stage without its own agent specified will run in a new container using the image you’ve specified or built, on the same computer and with access to the same workspace. Stage options Sometimes, you may only want to disable automatic checkout of your repository, using the skipDefaultCheckout(true) option, for one specific stage in your Pipeline. Or perhaps you want to have a timeout that covers an entire stage, including time spent waiting for a valid agent, post condition execution, or the new input directive for stages (see further down for more details on that!). To make those things possible, we’ve added a new options direction to stage. You can use a subset of the top-level options content in a stage’s `options - wrapper steps, and Declarative-specific options that are marked as legal in a stage. Input You’ve always been able to run the input step inside a stage’s `steps block, but we’ve found that approach can lose out on some of the value that the input step provides. To help with that, we’ve added a new input directive to stage, with the same parameters as the input step. When you use the stage input directive rather than using the step directly, any parameters you’ve specified for the input will be made available in the stage’s environment, meaning you can reference parameters from the `input in when conditions, or in environment variables. // Declarative // pipeline { agent none stages { stage('Example') { input { message "Should we continue?" ok "Yes, we should." submitter "alice,bob" parameters { string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?') } } agent any steps { echo "Hello, ${PERSON}, nice to meet you." } } } } // Script // Also, the input directive is evaluated before you enter any agent specified on this stage, so if you are using a top-level agent none and each stage has its own agent specified, you can avoid consuming an executor while waiting for the input to be submitted. Lastly, you can use timeout in the stage options, as mentioned above, to time-out the input if too much time has passed without a response. I hope you find these new features and options for Declarative Pipelines helpful, and I look forward to the rest of 2018 as we continue to invest and improve in Jenkins Pipeline!

After a few months of work on its key features, I’m happy to announce the 1.2 release of Declarative Pipeline! On behalf of the contributors developing Pipeline, I thought it would be helpful to discuss three of the key changes. Parallel Stages First, we’ve added syntax support for parallel stages. In earlier versions of Declarative Pipeline, the only way to run chunks of Pipeline code in parallel was to use the parallel step inside the steps block for a stage, like this: /* .. snip .. */ stage('run-parallel-branches') { steps { parallel( a: { echo "This is branch a" }, b: { echo "This is branch b" } ) } } /* .. snip .. */ While this works, it doesn’t integrate well with the rest of the Declarative Pipeline syntax. For example, to run each parallel branch on a different agent, you need to use a node step, and if you do that, the output of the parallel branch won’t be available for post directives (at a stage or pipeline level). Basically the old parallel step required you to use Scripted Pipeline within a Declarative Pipeline. But now with Declarative Pipeline 1.2, we’ve introduced a true Declarative syntax for running stages in parallel: Jenkinsfile pipeline { agent none stages { stage('Run Tests') { parallel { stage('Test On Windows') { agent { label "windows" } steps { bat "run-tests.bat" } post { always { junit "**/TEST-*.xml" } } } stage('Test On Linux') { agent { label "linux" } steps { sh "run-tests.sh" } post { always { junit "**/TEST-*.xml" } } } } } } } You can now specify either steps or parallel for a stage, and within parallel, you can specify a list of stage directives to run in parallel, with all the configuration you’re used to for a stage in Declarative Pipeline. We think this will be really useful for cross-platform builds and testing, as an example. Support for parallel stages will be in the soon-to-be-released Blue Ocean Pipeline Editor 1.3 as well. You can find more documentation on parallel stages in the User Handbook. Defining Declarative Pipelines in Shared Libraries Until the 1.2 release, Declarative Pipelines did not officially support defining your pipeline blocks in a shared library. Some of you may have tried that out and found that it could work in some cases, but since it was never an officially supported feature, it was vulnerable to breaking due to necessary changes for the supported use cases of Declarative. But with 1.2, we’ve added official support for defining pipeline blocks in src/.groovy files in your shared libraries. Within your src/.groovy file’s call method, you can call pipeline { …​ }, or possibly different pipeline { …​ } blocks depending on if conditions and the like. Note that only one pipeline { …​ } block can actually be executed per run - you’ll get an error if a second one tries to execute! Major Improvements to Parsing and Environment Variables Hopefully, you’ll never actually care about this change, but we’re very happy about it nonetheless. The original approach used for actually taking the pipeline { …​ } block and executing its contents was designed almost two years ago, and wasn’t very well suited to how you all are actually using Declarative Pipelines. In our attempts to work around some of those limitations, we made the parsing logic even more complicated and fragile, resulting in an impressive number of bugs, mainly relating to inconsistencies and bad behavior with environment variables. In Declarative 1.2, we’ve replaced the runtime parsing logic completely with a far more robust system, which also happens to fix most of those bugs at the same time! While not every issue has been resolved, you may find that you can use environment variables in more places, escaping is more consistent, Windows paths are no longer handled incorrectly, and a lot more. Again, we’re hoping you’ve never had the misfortune to run into any of these bugs, but if you have, well, they’re fixed now, and it’s going to be a lot easier for us to fix any future issues that may arise relating to environment variables, when expressions, and more. Also, the parsing at the very beginning of your build may be about 0.5 seconds faster. =) More to Come! While we don’t have any concrete plans for what will be going into Declarative Pipelines 1.3, rest assured that we’ve got some great new features in mind, as well as our continuing dedication to fixing the bugs you encounter and report. So please do keep opening tickets for issues and feature requests. Thanks!

This is a guest post by Andrew Bayer, who is one of the authors of the Declarative Pipeline plugin, and is a software engineer on the Pipeline team at CloudBees, Inc. A year ago at Jenkins World 2016, we unveiled Declarative Pipeline, a structured way to define your Pipeline. It’s been a great year for Declarative and Pipeline in general, with the release of Declarative Pipeline 1.0 in February, multiple releases since then, the introduction of documentation on Pipeline at jenkins.io, with a focus on Declarative, and more. Given everything that’s happened over the last year, we thought it’d be good to let you all know what you can expect to see and hear about Declarative Pipeline at this year’s Jenkins World. First, on Thursday, August 31, I’ll be giving a talk on Declarative Pipeline with Robert Sandell, one of my coworkers here at CloudBees and another author of Declarative Pipeline. We’ll be covering what’s happened with Declarative over the last year, new features added since the 1.0 release, such as the libraries directive and more when conditions, what’s planned for the upcoming 1.2 release (which is planned for shortly after Jenkins World!), including parallel stage s, and what’s on the roadmap for the future. In addition, we’ll be demoing some of the features in 1.2, and providing some pointers on best practices for writing your Declarative Pipeline. Also on Thursday, Stephen Donner from Mozilla will be giving a demo showing Mozilla’s usage of Declarative Pipeline and shared libraries at the Community Booth - Mozilla has been doing great work with Declarative, and I’m excited to see their usage in more detail and hear Stephen talk about their experience! In addition, Robert, Stephen, and myself will all be at Jenkins World both days of the main sessions, and Robert and myself will also be at the Contributor Summit on Tuesday. We’d love to hear your thoughts on Declarative and will be happy to answer any questions that we can. Looking forward to seeing you all! Andrew Bayer and Robert Sandell will be talking about the latest on Declarative Pipeline in Jenkins at Jenkins World in August, register with the code JWFOSS for a 30% discount off your pass.

This week, we released the second beta of the new Declarative Pipeline syntax, available in the Update Center now as version 0.8.1 of Pipeline: Model Definition. You can read more about Declarative Pipeline in the blog post introducing the first beta from December, but we wanted to update you all on the syntax changes in the second beta. These syntax changes are the last compatibility-breaking changes to the syntax before the 1.0 release planned for February, so you can safely start using the 0.8.1 syntax now without needing to change it when 1.0 is released. A full syntax reference is available on the wiki as well. Syntax Changes Changed "agent" configuration to block structure In order to support more detailed and clear configuration of agents, as well as making agent syntax more consistent with the rest of the Declarative Pipeline syntax, we’ve moved the agent configuration into blocks. The agent any and agent none configurations work the same as previously, but label, docker and dockerfile now look like the following: Just specifying a label is simple. // Declarative // agent { label "some-label" } // Script // If you’re just specifying a Docker image, you can use this simple syntax. // Declarative // agent { docker "ubuntu:16.04" } // Script // When you are specifying a label or other arguments, docker looks like this: // Declarative // agent { docker { image "ubuntu:16.04" label "docker-label" args "-v /tmp:/tmp -p 8000:8000" } } // Script // When you’re building an image from "Dockerfile" in your repository and don’t care what node is used or have additional arguments, you can again use a simple syntax. // Declarative // agent { dockerfile true } // Script // When you’re building an image from a different file, or have a label or other arguments, use the following syntax: // Declarative // agent { dockerfile { filename "OtherDockerfile" label "docker-label" args "-v /tmp:/tmp -p 8000:8000" } } // Script // Improved "when" conditions We introduced the when section a couple releases ago, but have made some changes to its syntax here in 0.8.1. We wanted to add some simpler ways to specify common conditions, and that required we re-work the syntax accordingly. Branch One of the most common conditions is running a stage only if you’re on a specific branch. You can also use wildcards like "*/master". // Declarative // when { branch "master" } // Script // Environment Another built-in condition is the environment condition, which checks to see if a given environment variable is set to a given value. // Declarative // when { environment name: "SOME_ENV_VAR", value: "SOME_VALUE" } // Script // Expression Lastly, there’s the expression condition, which resolves an arbitrary Pipeline expression. If the return value of that expression isn’t false or null, the stage will execute. // Declarative // when { expression { echo "Should I run?" return "foo" == "bar" } } // Script // "options" replaces "properties" and "wrappers" We’ve renamed the properties section to options, due to needing to add new Declarative-specific options and to cut down on confusion. The options section is now where you’ll put general Pipeline options like buildDiscarder, Declarative-specific options like skipDefaultCheckout, and block-scoped steps that should wrap the execution of the entire build, like timeout or timestamps. // Declarative // options { buildDiscarder(logRotator(numToKeepStr:'1')) skipDefaultCheckout() timeout(time: 5, unit: 'MINUTES') } // Script // Heading towards 1.0! While we may still add more functionality to the Declarative Pipeline syntax, we won’t be making any changes to existing syntax for the 1.0 release. This means that any pipelines you write against the 0.8.1 syntax will keep working for the foreseeable future without any changes. So if you’re already using Declarative Pipelines, make sure to update your `Jenkinsfile`s after upgrading to 0.8.1, and if you haven’t been using Declarative Pipelines yet, install the Pipeline: Model Definition plugin and give them a try!

This is a guest post by Jenkins World speaker Andrew Bayer, Jenkins developer at CloudBees. Over the last couple years, Pipeline as code has very much become the future of Jenkins - in fact, at this point, I’d say it’s pretty well established as the present of Jenkins. But that doesn’t mean it’s done, let alone that it’s perfect. While many developers enjoy the power and control that they get from writing Pipelines using scripting, not everyone feels the same way. A lot of developers want to specify their build as configuration and get on with building software. Pipeline scripts haven’t been a good way to do that…​until now. With new changes to Jenkins Pipeline, you are now able to define your Pipeline from configuration in your Jenkinsfile by installing the new Pipeline Model Definition plugin. It’s available today for you to try via the update center. Be sure to check the documentation for examples on how to get started for a variety of languages and platforms. Here’s a quick example based on the plugin’s own Jenkinsfile : pipeline { // Make sure that the tools we need are installed and on the path. tools { maven "Maven 3.3.9" jdk "Oracle JDK 8u40" } // Run on any executor. agent label:"" // The order that sections are specified doesn't matter - this will still be run // after the stages, even though it's specified before the stages. postBuild { // No matter what the build status is, run these steps. There are other conditions // available as well, such as "success", "failed", "unstable", and "changed". always { archive "target/**/*" junit 'target/surefire-reports/*.xml' } } stages { // While there's only one stage here, you can specify as many stages as you like! stage("build") { sh 'mvn clean install -Dmaven.test.failure.ignore=true' } } } It’s still early days for this feature, with a lot of further functionality planned to make it easier and more intuitive to define your Pipelines. All of that functionality lives on top of Pipeline scripting, so we’ll also keep improving Pipeline steps and syntax outside of the model! And perhaps most exciting, the Pipeline model will be used by an in-the-works visual editor that will be part of the Blue Ocean project - while the editor isn’t ready yet, the Pipeline Model Definition plugin will be bundled with the Blue Ocean beta for you to try out. I’ll be going into all of this and more at my talk on Thursday, September 15th, at 3:45pm at Jenkins World, and showing off the same day at the lunchtime demo theater. I can’t wait to see you all there and hear what you think of all this! Andrew will be presenting more of this concept at Jenkins World in September. Register with the code JWFOSS for 20% off your full conference pass.

On January 20, the first San Francisco JAM (Jenkins Area Meetup) of the new year was held at Mesosphere ’s offices. We had two speakers - myself, and Roger Ignazio, an infrastructure automation engineer at Mesosphere. Around forty people attended and enjoyed the food and drinks Mesosphere provided for us. Links to the talks are below: Elastic Jenkins with Mesos and DCOS, by Roger Ignazio Who is Jenkins?, by Andrew Bayer More JAMs will be happening in the coming months - for example, the first Los Angeles JAM is tentatively planned for early March, 2016! You can always find out the latest on JAMs around the world at the Jenkins Area Meetup page.

A lot of people are using the Workflow plugin, but as with any scripting environment, users often have to start from scratch and learn the same lessons and shortcuts that other users have already learned. While there are blog posts from developers and users in various places, and some samples in the Workflow plugin documentation, more examples and tips and tricks are always, always useful. To help with that, we’ve created the workflow-examples repository on GitHub, as a place to store community-developed Workflow scripts that can help new users get started, show how to accomplish some non-trivial goals, and find tips and trick for taking your Workflow pipeline to the next level. The repository has four directories: docs/ - documentation, guides, and more. Including a Best Practices document. We’d love to see more contributions to that doc, as well as any new ones that would be helpful to Workflow users! workflow-examples/ - general Workflow examples, showing how to use a given plugin with Workflow, quirks of the Workflow DSL syntax, and more. global-library-examples/ - examples of how to write code for the Workflow global library. jenkinsfile-examples/ - Sample Jenkinsfiles or other Workflow scripts from SCM . During Hacksgiving some initial content was added, but not everything is covered yet, which is why I’m posting this - more is needed. We’d love to see your tips, examples, gotchas and more. If you’ve got Workflow scripts you’d like to contribute, please read the README and send a pull request. Thanks!