This is a guest post by Guillaume Laforge, Developer Advocate for Google Cloud Last week, I had the pleasure to speak at the Jenkins Community Day conference, in Paris, organized by my friends from JFrog, provider of awesome tools for software management and distribution. I covered how to scale Jenkins with Kubernetes on Google Container Engine. For the impatient, here are the slides of the presentation I’ve given: But let’s step back a little. In this article, I’d like to share with you why you would want to run Jenkins in the cloud, as well as give you some pointers to interesting resources on the topic. Why running Jenkins in the cloud? So why running Jenkins in the cloud? First of all, imagine your small team, working on a single project. You have your own little server, running under a desk somewhere, happily building your application on each commit, a few times a day. So far so good, your build machine running Jenkins isn’t too busy, and stays idle most of the day. Let’s do some bottom of the napkin calculations. Let’s say you have a team of 3 developers, committing roughly 4 times a day, on one single project, and the build takes roughly 10 minutes to go. 3 developers * 4 commits / day / developer * 10 minutes build time * 1 project = 1 hour 20 minutes So far so good, your server indeed stays idle most of the day. Usually, at most, your developers will wait just 10 minutes to see the result of their work. But your team is growing to 10 persons, the team is still as productive, but the project becoming bigger, the build time goes up to 15 minutes: 10 developers * 4 commits / day / developer * 15 minutes build time * 1 project = 10 hours You’re already at 10 hours build time, so your server is busy the whole day, and at times, you might have several build going on at the same time, using several CPU cores in parallel. And instead of building in 15 minutes, sometimes, the build might take longer, or your build might be queued. So in theory, it might be 15 minutes, but in practice, it could be half an hour because of the length of the queue or the longer time to build parallel projects. Now, the company is successful, and has two projects instead of one (think a backend and a mobile app). Your teams grow further up to 20 developers per project. The developers are a little less productive because of the size of the codebase and project, so they only commit 3 times a day. The build takes more time too, at 20 minutes (in ideal time). Let’s do some math again: 20 developers * 3 commits / day / developer * 20 minutes build time * 2 projects = 40 hours Woh, that’s already 40 hours of total build time, if all the builds are run serially. Fortunately, our server is multi-core, but still, there are certainly already many builds that are enqueued, and many of them, perhaps up to 2-3 or perhaps even 4 could be run in parallel. But as we said, the build queue increases further, the real effective time of build is certainly longer than 30 minutes. Perhaps at times, developers won’t see the result of their developments before at least an hour, if not more. One last calculation? With team sizes of 30 developers, decreased productivity of 2 commits, 25 build time, and 3 projects? And you’ll get 75 hours total build time. You may start creating a little build farm, with a controller and several build agents. But you also increase the burden of server management. Also, if you move towards a full Continuous Delivery or Continuous Deployment approach, you may further increase your build times to go up to deployment, make more but smaller commits, etc. You could think of running builds less often, or even on a nightly basis, to cope with the demand, but then, your company is less agile, and the time-to-market for fixes of new features might increase, and your developers may also become more frustrated because they are developing in the blind, not knowing before the next day if their work was successful or not. With my calculations, you might think that it makes more sense for big companies, with tons of projects and developers. This is quite true, but when you’re a startup, you also want to avoid taking care of local server management, provisioning, etc. You want to be agile, and use only compute resources you need for the time you need them. So even if you’re a small startup, a small team, it might still make sense to take advantage of the cloud. You pay only for the actual time taken by your builds as the build agent containers are automatically provisioned and decommissioned. The builds can scale up via Kubernetes, as you need more (or less) CPU time for building everything. And this is why I was happy to dive into scaling Jenkins in the cloud. For that purpose, I decided to go with building with containers, with Kubernetes, as my app was also containerized as well. Google Cloud offers Container Engine, which is basically just Kubernetes in the cloud. Useful pointers I based my presentation and demo on some great solutions that are published on the Google Cloud documentation portal. Let me give you some pointers. Overview of Jenkins on Container Engine Setting up Jenkins on Container Engine Configuring Jenkins for Container Engine Continuous Deployment to Container Engine using Jenkins Lab: Build a Continuous Deployment Pipeline with Jenkins and Kubernetes The latter one is the tutorial I actually followed for the demo that I presented during the conference. It’s a simple Go application, with a frontend and backend. It’s continuously build, on each commit (well, every minute to check if there’s a new commit), and deployed automatically in different environments: dev, canary, production. The sources of the project are stored in Cloud Source Repository (it can be mirrored from Github, for example). The containers are stored in Cloud Container Registry. And both the Jenkins controller and agents, as well as the application are running inside Kubernetes clusters in Container Engine. Summary and perspective Don’t bother with managing servers! Quickly, you’ll run out of CPU cycles, and you’ll have happier developers with builds that are super snappy! And for the record, at Google, dev teams are also running Jenkins! There was a presentation ( video and slides available) given last year by David Hoover at Jenkins World talking about how developers inside Google are running hundreds of build agents to build projects on various platforms.

This is a guest post by Guillaume Laforge. Well known for his contribution to the Apache Groovy project, Guillaume is also the "Product Ninja and Advocate" of Restlet, a company focusing on Web APIs: with DHC (an API testing client), Restlet Studio (an API designer), APISpark (an API platform in the cloud), and the Restlet Framework open source project for developing APIs. Modern mobile apps, single-page web sites and applications, are more and more relying on Web APIs, as the nexus of the interaction between the frontend and the backend services. Web APIs are also central to third-party integration, when you want to share your services with others, or when you need to consume existing APIs to build your own solution on top of their shoulders. With APIs being a key element of your architecture and big picture, it’s obviously important to assess that this API is functioning the way it should, thanks to proper testing. Your framework of choice, regardless of the technology stack or programming language used, will hopefully offer some facilities for testing your code, whether in the form of unit tests, or ideally with integration tests. Coding Web API tests From a code perspective, as I said, most languages and frameworks provide approaches to testing APIs built with them. There’s one I wanted to highlight in particular, which is one developed with a DSL approach (Domain-Specific Language), using the Apache Groovy programming language, it’s AccuREST. To get started, you can have a look at the introduction, and the usage guide. If you use the contract DSL, you’ll be able to write highly readable examples of requests you want to issue against your API, and the assertions that you expect to be true when getting the response from that call. Here’s a concrete example from the documentation: GroovyDsl.make { request { method 'POST' urlPath('/users') { queryParameters { parameter 'limit': 100 parameter 'offset': containing("1") parameter 'filter': "email" } } headers { header 'Content-Type': 'application/json' } body '''{ "login" : "john", "name": "John The Contract" }''' } response { status 200 headers { header 'Location': '/users/john' } } } Notice that the response is expected to return a status code 200 OK, and a Location header pointing at /users/john. Indeed, a very readable way to express the requests and responses! Tooling to test your APIs From a tooling perspective, there are some interesting tools that can be used to test Web APIs, like Paw (on Macs), Advanced REST client, Postman or Insomnia. But in this article, I’ll offer a quick look at DHC, a handy visual tool, that you can use both manually to craft your tests and assertions, and whose test scenarios you can export and integrate in your build and continuous integration pipeline, thanks to Maven and Jenkins. At the end of this post, you should be able to see the following reporting in your Jenkins dashboard, when visualising the resulting API test execution: Introducing DHC DHC is a Chrome extension, that you can install from the Chrome Web Store, in your Chrome browser. There’s also an online service available, with some limitations. For the purpose of this article, we’ll use the Chrome extension. In the main area, you can create your request, define the URL to call, specify the various request headers or params, chose the method you want to use, and then, you can click the send button to issue the request. In the left pane, that’s where you’ll be able to see your request history, create and save your project in the cloud, or also set context variables. The latter is important when testing your Web API, as you’ll be able to insert variables like for example {localhost} for testing locally on your machine or {staging} and {prod} to run your tests in different environments. In the bottom pane, you have access to actual raw HTTP exchange, as well as the assertions pane. Again, a very important pane to look at! With assertions, you’ll be able to ensure that your Web API works as expected. For instance, you can check the status code of the call, check the payload contains a certain element, by using JSON Path or XPath to go through the JSON or XML payload respectively. Beyond assertions, what’s also interesting is that you can chain requests together. A call request can depend on the outcome of a previous request! For example, in a new request, you could pass a query parameter whose value would be the value of some element of the JSON payload of a previously executed request. And by combining assertions, linked requests and context variables together, you can create full-blown test scenarios, that you can then save in the cloud, but also export as a JSON file. To export that test scenario, you can click on the little export icon in the bottom left hand corner, and you’ll be able to select exactly what you want to export: Running your Web API tests with Maven Now things become even more interesting, as we’ll proceed to using Maven and Jenkins! As the saying goes, there’s a Maven plugin for that! For running those Web API tests in your build! Even if your Web API is developed in another technology than Java, you can still create a small Maven build just for your Web API tests. And the icing on the cake, when you configure Jenkins to run this build, as the plugin outputs JUnit-friendly test reports, you’ll be able to see the details of your successful and failed tests, just like you would see JUnit’s! Let’s sketch your Maven POM: 4.0.0 com.example my-first-api-test 1.2.3 com.restlet.dhc dhc-maven-plugin 1.1 test test companies-scenario.json restlet-maven Restlet public Maven repository Release Repository https://maven.restlet.com Visualizing Web API test executions in Jenkins Once you’ve configured your Jenkins server to launch the test goal of this Maven project, you’ll be able to see nice test reports for your Web API scenarios, like in the screenshot in introduction of this article! Next, you can easily run your Web API tests when developers commit changes to the API, or schedule regular builds with Jenkins to monitor an online Web API. For more information, be sure to read the tutorial on testing Web APIs with DHC. There are also some more resources like a screencast, as well as the user guide, if you want to learn more. And above all, happy testing!