Recent Blog Posts

This is a guest post by Carl Caum, who works at Puppet and created the Puppet Enterprise Pipeline plugin. During PuppetConf 2016, myself and Brian Dawson from CloudBees announced the plugin:puppet-enterprise-pipeline[Puppet Enterprise plugin for Jenkins Pipeline]. Let’s take a look at how the plugin makes it trivial to use Puppet to perform some or all of the deployment tasks in continuous delivery pipelines. Jenkins Pipeline introduced an amazing world where the definition for a pipeline is managed from the same version control repository as the code delivered by the pipeline. This is a powerful idea, and one I felt complemented Puppet’s automation strengths. I wanted to make it trivial to control Puppet Enterprise’s orchestration and infrastructure code management capabilities, as well as set hierarchical configuration data and use Puppet’s inventory data system as a source of truth – all from a Pipeline script. The result was the Puppet Enterprise plugin, which fully buys into the Pipeline ideals by providing methods to control the different capabilities in Puppet Enterprise. The methods provide ways to query PuppetDB, set Hiera key/value pairs, deploy Puppet code environments with Code Management, and kick off orchestrated Puppet runs with the Orchestrator. The Puppet Enterprise for Jenkins Pipeline plugin The Puppet Enterprise for Jenkins Pipeline plugin itself has zero system dependencies. You need only to install the plugin from the update center. The plugin uses APIs available in Puppet Enterprise to do its work. Since the PuppetDB query, Code Management, and Orchestrator APIs are all backed by Puppet Enterprise’s role-based access control (RBAC) system, it’s easy to restrict what pipelines are allowed to control in Puppet Enterprise. To learn more about RBAC in Puppet Enterprise, read the docs here. Configuring Configuring the plugin is fairly straight forward. It takes three simple steps: Set the address of the Puppet server Create a Jenkins credential with a Pupppet Enterprise RBAC authentication token Configure the Hiera backend Set the Puppet Enterprise Server Address Go to Jenkins > Manage Jenkins > Puppet Enterprise page. Put the DNS address of the Puppet server in the Puppet Master Address text field. Click the Test Connection button to verify the server is reachable, the Puppet CA certificate is retrievable, and HTTPS connections are successful. Once the test succeeds, Click Save. Create a Jenkins Credentials Entry The plugin uses the Jenkins built-in credentials system (the plain-credentials plugin) to store and refer RBAC tokens to Puppet Enterprise for authentication and authorization. First, generate an RBAC token in Puppet Enterprise by following the instructions on the docs site. Next, create a new Jenkins Credentials item with Kind Secret text and the Secret value the Puppet Enterprise RBAC token. It’s highly recommended to give the credential an ID value that’s descriptive and identifiable. You’ll use it in your Pipeline scripts. In your Jenkinsfile, use the puppet.credentials method to set all future Puppet methods to use the RBAC token. For example: puppet.credentials 'pe-team-token' Configure the Hiera Backend The plugin exposes an HTTP API for performing Hiera data lookups for key/value pairs managed by Pipeline jobs. To configure Hiera on the Puppet compile master(s) to query the Jenkins Hiera data store backend, use the hiera-http backend. On the Puppet Enterprise compile master(s), run the following commands: /opt/puppetlabs/puppet/bin/gem install hiera-http /opt/puppetlabs/bin/puppetserver gem install hiera-http Now you can configure the /etc/puppetlabs/puppet/hiera.yaml file. The following configuration instructs Hiera to first look to the Hiera yaml files in the Puppet code’s environment, then fall back to the http backend. The http backend will first query the Hiera data store API looking for the key in the scope with the same name as the node. If nothing’s found, look for the key in the node’s environments. You can use any Facter fact to match scope names. :backends: - yaml - http :http: :host: jenkins.example.com :port: 8080 :output: json :use_auth: true :auth_user: :auth_pass: :cache_timeout: 10 :failure: graceful :paths: - /hiera/lookup?path=%{clientcert}&key=%{key} - /hiera/lookup?path=%{environment}&key=%{key} Finally, restart the pe-puppetserver process to pick up the new configs: /opt/puppetlabs/bin/puppet resource service pe-puppetserver ensure=stopped /opt/puppetlabs/bin/puppet resource service pe-puppetserver ensure=running Hiera HTTP Authentication If Jenkins' Global Security is configured to allow unauthenticated read-only access, the 'use_auth', 'auth_pass', and 'auth_user' parameters are unnecessary. Otherwise, create a local Jenkins user that has permissions to view the Hiera Data Lookup page and use that user’s credentials for the hiera.yaml configuration. Querying the infrastructure PuppetDB is an extensive data store that holds every bit of information Puppet generates and collects across every system Puppet is installed on. PuppetDB provides a sweet query language called PQL. With PQL, you can ask complex questions of your infrastructure such as "How many production Red Hat systems are there with the openssl package installed?" or "What us-west-2c nodes with the MyApp role that were created in the last 24 hours?" This can be a powerful tool for parts of your pipeline where you need to perform specific operations on subsets of the infrastructure like draining a loadbalancer. Here’s an example using the puppet.query method: results = puppet.query ''' inventory[certname] { facts.os.name = "RedHat" and facts.ec2_metadata.placement.availability-zone = "us-west-2c" and facts.uptime_hours < 24 }''' The query returns an array of matching items. The results can be iterated on, and even passed to a series of puppet.job calls. For example, the following code will query all nodes in production that experienced a failure on the last Puppet run. results = puppet.query 'nodes { latest_report_status = "failed" and catalog_environment = "production"}' Note that once you can use closures in Pipeline scripts, doing the above example will be much simpler. Creating an orchestrator job The orchestration service in Puppet Enterprise is a tool to perform orchestrated Puppet runs across as broad or as targeted an infrastructure as you need at different parts of a pipeline. You can use the orchestrator to update applications in an environment, or update a specific list of nodes, or update nodes across a set of nodes that match certain criteria. In each scenario, Puppet will always push distributed changes in the correct order by respecting the cross-node dependencies. To create a job in the Puppet orchestrator from a Jenkins pipeline, use the puppet.job method. The puppet.job method will create a new orchestrator job, monitor the job for completion, and determine if any Puppet runs failed. If there were failures, the pipeline will fail. The following are just some examples of how to run Puppet orchestration jobs against the infrastructure you need to target. Target an entire environment: puppet.job 'production' Target instances of an application in production: puppet.job 'production', application: 'Myapp' Target a specific list of nodes: puppet.job 'production', nodes: ['db.example.com','appserver01.example.com','appserver02.example.com'] Target nodes matching a complex set if criteria: puppet.job 'production', query: 'inventory[certname] { facts.os.name = "RedHat" and facts.ec2_metadata.placement.availability-zone = "us-west-2c" and uptime_hours < 24 }' As you can see, the puppet.job command means you can be as broad or as targeted as you need to be for different parts of your pipeline. There are many other options you can add to the puppet.job method call, such as setting the Puppet runs to noop, or giving the orchestrator a maximum concurrency limit. Learn more about the orchestrator here. Updating Puppet code If you’re using Code Management in Puppet Enterprise (and you should), you can ensure that all the modules, site manifests, Hiera data, and roles and profiles are staged, synced, and ready across all your Puppet masters, direct from your Jenkins pipeline. To update Puppet code across all Puppet masters, use the puppet.codeDeploy method: puppet.codeDeploy 'staging' Learn more Code Management in Puppet Enterprise here. Setting Hiera values The plugin includes an experimental feature to set Hiera key/value pairs. There are many cases where you need to promote information through a pipeline, such as a build version or artifact location. Doing so is very difficult in Puppet, since data promotion almost always involves changing Hiera files and committing to version control. The plugin exposes an HTTP API endpoint that Hiera can query using the hiera-http backend. With the backend configured on the Puppet master(s), key/value pairs can be set to scopes. A scope is arbitrary and can be anything you like, such as a Puppet environment, a node’s certname, or the name of a Facter fact like operatingsystem or domain. To set a Hiera value from a pipeline, use the puppet.hiera method. puppet.hiera scope: 'staging', key: 'build-version', value: env.BUILD_ID Now you can set the same key with the same value to the production scope later in the pipeline, followed by a call to puppet.job to push the change out. Examples The plugin’s Github repository contains a set of example Pipeline scripts. Feel free to issue pull requests to add your own scripts! What’s next I’m pretty excited to see how this is going to help simplify continuous delivery pipelines. I encourage everyone to get started with continuous delivery today, even if it’s just a simple pipeline. As your practices evolve, you can begin to add automated tests, automate away manual checkpoints, start to incorporate InfoSec tests, and include phases for practices like patch management that require lots of manual approvals, verifications and rollouts. You’ll be glad you did.

This is a guest post by R. Tyler Croy, who is a long-time contributor to Jenkins and the primary contact for Jenkins project infrastructure. He is also a Jenkins Evangelist at CloudBees, Inc. When the Ruby on Rails framework debuted it changed the industry in two noteworthy ways: it created a trend of opinionated web application frameworks ( Django, Play, Grails) and it also strongly encouraged thousands of developers to embrace test-driven development along with many other modern best practices (source control, dependency management, etc). Because Ruby, the language underneath Rails, is interpreted instead of compiled there isn’t a "build" per se but rather tens, if not hundreds, of tests, linters and scans which are run to ensure the application’s quality. With the rise in popularity of Rails, the popularity of application hosting services with easy-to-use deployment tools like Heroku or Engine Yard rose too. This combination of good test coverage and easily automated deployments makes Rails easy to continuously deliver with Jenkins. In this post we’ll cover testing non-trivial Rails applications with Jenkins Pipeline and, as an added bonus, we will add security scanning via Brakeman and the Brakeman plugin. Topics Preparing the app Preparing Jenkins Writing the Pipeline Security scanning Deploying the good stuff Wrap up For this demonstration, I used Ruby Central 's cfp-app : A Ruby on Rails application that lets you manage your conference’s call for proposal (CFP), program and schedule. It was written by Ruby Central to run the CFPs for RailsConf and RubyConf. I chose this Rails app, not only because it’s a sizable application with lots of tests, but it’s actually the application we used to collect talk proposals for the " Community Tracks" at this year’s Jenkins World. For the most part, cfp-app is a standard Rails application. It uses PostgreSQL for its database, RSpec for its tests and Ruby 2.3.x as its runtime. If you prefer to just to look at the code, skip straight to the Jenkinsfile. Preparing the app For most Rails applications there are few, if any, changes needed to enable continuous delivery with Jenkins. In the case of cfp-app, I added two gems to get the most optimal integration into Jenkins: ci_reporter, for test report integration brakeman, for security scanning. Adding these was simple, I just needed to update the Gemfile and the Rakefile in the root of the repository to contain: Gemfile # .. snip .. group :test do # RSpec, etc gem 'ci_reporter' gem 'ci_reporter_rspec' gem "brakeman", :require => false end Rakefile # .. snip .. require 'ci/reporter/rake/rspec' # Make sure we setup ci_reporter before executing our RSpec examples task :spec => 'ci:setup:rspec' Preparing Jenkins With the cfp-app project set up, next on the list is to ensure that Jenkins itself is ready. Generally I suggest running the latest LTS of Jenkins; for this demonstration I used Jenkins 2.7.1 with the following plugins: Pipeline plugin Brakeman plugin CloudBees Docker Pipeline plugin I also used the GitHub Organization Folder plugin to automatically create pipeline items in my Jenkins instance; that isn’t required for the demo, but it’s pretty cool to see repositories and branches with a Jenkinsfile automatically show up in Jenkins, so I recommend it! In addition to the plugins listed above, I also needed at least one Jenkins agent with the Docker daemon installed and running on it. I label these agents with "docker" to make it easier to assign Docker-based workloads to them in the future. Any Linux-based machine with Docker installed will work, in my case I was provisioning on-demand agents with the Azure plugin which, like the EC2 plugin, helps keep my test costs down. If you’re using Amazon Web Services, you might also be interested in this blog post from earlier this year unveiling the EC2 Fleet plugin for working with EC2 Spot Fleets. Writing the Pipeline To make sense of the various things that the Jenkinsfile needs to do, I find it easier to start by simply defining the stages of my pipeline. This helps me think of, in broad terms, what order of operations my pipeline should have. For example: /* Assign our work to an agent labelled 'docker' */ node('docker') { stage 'Prepare Container' stage 'Install Gems' stage 'Prepare Database' stage 'Invoke Rake' stage 'Security scan' stage 'Deploy' } As mentioned previously, this Jenkinsfile is going to rely heavily on the CloudBees Docker Pipeline plugin. The plugin provides two very important features: Ability to execute steps inside of a running Docker container Ability to run a container in the "background." Like most Rails applications, one can effectively test the application with two commands: bundle install followed by bundle exec rake. I already had some Docker images prepared with RVM and Ruby 2.3.0 installed, which ensures a common and consistent starting point: node('docker') { // .. 'stage' steps removed docker.image('rtyler/rvm:2.3.0').inside { (1) rvm 'bundle install' (2) rvm 'bundle exec rake' } (3) } 1 Run the named container. The inside method can take optional additional flags for the docker run command. 2 Execute our shell commands using our tiny sh step wrapper rvm . This ensures that the shell code is executed in the correct RVM environment. 3 When the closure completes, the container will be destroyed. Unfortunately, with this application, the bundle exec rake command will fail if PostgreSQL isn’t available when the process starts. This is where the second important feature of the CloudBees Docker Pipeline plugin comes into effect: the ability to run a container in the "background." node('docker') { // .. 'stage' steps removed /* Pull the latest `postgres` container and run it in the background */ docker.image('postgres').withRun { container -> (1) echo "PostgreSQL running in container ${container.id}" (2) } (3) } 1 Run the container, effectively docker run postgres 2 Any number of steps can go inside the closure 3 When the closure completes, the container will be destroyed. Running the tests Combining these two snippets of Jenkins Pipeline is, in my opinion, where the power of the DSL shines: node('docker') { docker.image('postgres').withRun { container -> docker.image('rtyler/rvm:2.3.0').inside("--link=${container.id}:postgres") { (1) stage 'Install Gems' rvm "bundle install" stage 'Invoke Rake' withEnv(['DATABASE_URL=postgres://postgres@postgres:5432/']) { (2) rvm "bundle exec rake" } junit 'spec/reports/*.xml' (3) } } } 1 By passing the --link argument, the Docker daemon will allow the RVM container to talk to the PostgreSQL container under the host name 'postgres'. 2 Use the withEnv step to set environment variables for everything that is in the closure. In this case, the cfp-app DB scaffolding will look for the DATABASE_URL variable to override the DB host/user/dbname defaults. 3 Archive the test reports generated by ci_reporter so that Jenkins can display test reports and trend analysis. With this done, the basics are in place to consistently run the tests for cfp-app in fresh Docker containers for each execution of the pipeline. Security scanning Using Brakeman, the security scanner for Ruby on Rails, is almost trivially easy inside of Jenkins Pipeline, thanks to the Brakeman plugin which implements the publishBrakeman step. Building off our example above, we can implement the "Security scan" stage: node('docker') { /* --8 (1) publishBrakeman 'brakeman-output.tabs' (2) /* --8 1 Run the Brakeman security scanner for Rails and store the output for later in brakeman-output.tabs 2 Archive the reports generated by Brakeman so that Jenkins can display detailed reports with trend analysis. As of this writing, there is work in progress ( JENKINS-31202) to render trend graphs from plugins like Brakeman on a pipeline project’s main page. Deploying the good stuff Once the tests and security scanning are all working properly, we can start to set up the deployment stage. Jenkins Pipeline provides the variable currentBuild which we can use to determine whether our pipeline has been successful thus far or not. This allows us to add the logic to only deploy when everything is passing, as we would expect: node('docker') { /* --8 (1) sh './deploy.sh' (2) } else { mail subject: "Something is wrong with ${env.JOB_NAME} ${env.BUILD_ID}", to: 'nobody@example.com', body: 'You should fix it' } /* --8 1 currentBuild has the result property which would be 'SUCCESS', 'FAILED', 'UNSTABLE', 'ABORTED' 2 Only if currentBuild.result is successful should we bother invoking our deployment script (e.g. git push heroku master) Wrap up I have gratuitously commented the full Jenkinsfile which I hope is a useful summation of the work outlined above. Having worked on a number of Rails applications in the past, the consistency provided by Docker and Jenkins Pipeline above would have definitely improved those projects' delivery times. There is still room for improvement however, which is left as an exercise for the reader. Such as: preparing new containers with all their dependencies built-in instead of installing them at run-time. Or utilizing the parallel step for executing RSpec across multiple Jenkins agents simultaneously. The beautiful thing about defining your continuous delivery, and continuous security, pipeline in code is that you can continue to iterate on it!

This is a guest post by Brian Dawson at CloudBees, where he works as a DevOps Evangelist responsible for developing and sharing continuous delivery and DevOps best practices. He also serves as the CloudBees Product Marketing Manager for Jenkins. Last fall CloudBees asked attendees at the Jenkins User Conference – US West (JUC), and other in the Jenkins community to take a survey. Almost 250 people did – and thanks to their input, we have results which provided interesting insights into how Jenkins is being used. Back in 2012, at the time of the last community survey, 83% of respondents felt that Jenkins was mission-critical. By 2015, the percentage saying that Jenkins was mission-critical was 92%. Additionally, echoing the importance of Jenkins, 89% of respondents said their use of Jenkins had increased over the last year, while 11% said it had stayed the same. 0% said that it had decreased. The trend in the industry over the last couple of years has been to adopt continuous delivery (CD), thus pushing automation further down the pipeline – from development all the way into production. Jenkins being an automation engine applicable to any phase of the software delivery lifecycle, is readily supporting this trend. Jenkins' extensible architecture and unparalleled plugin ecosystem enables integration with and orchestration of practically any tool in any phase of software delivery. The trend towards adoption of CD is clearly reflected amongst the community: 59% of respondents are using Jenkins for continuous integration (CI), but an additional 30% have extended CI into CD and are manually deploying code to production. Finally, 11% are practicing continuous deployment – they have extended CI to CD and are deploying code automatically into production. Another trend tied to the adoption of CD and DevOps is the frequent deployment of incremental releases to production. 26% of those respondents using continuous delivery practices are deploying code at least once per day. Another 37% are deploying code at least once per week. In keeping with the move to CD, 30% of survey takers are already using the relatively new Pipeline plugin to automate their software delivery pipelines. Of those not using the Pipeline plugin, 79% plan to adopt it in the next 12 months. Survey respondents are also using Jenkins for many different activities. 97% of survey takers use it for "build" – no surprise, since that is where Jenkins got its start - but 58% now also use it for their deployment. When the 2012 community survey was conducted, container technology was not as well understood as it is today, and many didn’t know what a “Docker” was. A short four years later, 96% of survey respondents who use Linux containers are using Docker. Container technology has seen impressive adoption and arguably is revolutionizing the way application infrastructure is delivered. When coupled with Jenkins as an automation engine, containers help accelerate software delivery by providing rapid access to lightweight environments. The Jenkins community has recognized and embraced the power of containers by providing plugins for Docker and Kubernetes. The Jenkins improvements which survey respondents desired the most were quality/timely bug fixes, a better UI and more documentation/examples. Interestingly, Jenkins 2.0 - which is just about to officially launch, provides UI improvements and the new Jenkins.io website provides improved, centralized documentation. Finally, the respondents favorite Star Wars character was R2-D2, followed by Obi-Wan and Darth Vader. Yoda and Han Solo also got a fair amount of votes. The votes for Jar-Jar Binks and Jabba the Hutt left us puzzled. Notably, BB-8 had a write-in vote despite the fact the new Star Wars movie hadn’t been released yet. As to where the community is headed, our prediction is that by the next Jenkins Community Survey: More Jenkins users will have transitioned from just continuous integration to continuous delivery with some evening practicing continuous deployment Pipeline plugin adoption and improvements will continue, leading to pipeline-as-code becoming an essential solution for automating the software (and infrastructure) delivery process There will be a significant increase in use of the Docker plugin to support elastic Jenkins infrastructure and continuous delivery of containers using software development best practices BB-8 will be the next favorite Star Wars character! <3</p> See you at Jenkins World, September 13-15, in Santa Clara, California! Register now for the largest Jenkins event on the planet in 2016 – and get the Early Bird discount. The Call for Papers is still open – so submit a talk and share your knowledge with the community about Jenkins. 2015 Community Survey Results (PDF) State of Jenkins Infographic (PDF)