Recent Blog Posts

This year, the Jenkins organization participated in the Google Summer of Code Mentor Summit at the Google office in Sunnyvale on Oct 12, 13 and 14, 2018. The GSoC Mentor Summit is where mentors of all organizations participating in the GSoC program are invited each year to learn and network with mentors from other organization, and make GSoC a better program. This is the second time Jenkins mentors have participated in the summit, the first time was in 2016. Exceptionally, three Jenkins GSoC mentors were invited to the summit this year. Normally only two mentors are invited, but when there are cancellations, Google draws a name at random from the waiting list, and the Jenkins organization was lucky enough and sent an extra mentor this year! The mentors participating this year were Oleg Nenashev, Jeff Pearce and Martin d’Anjou. It is worth mentioning that the Mentor Summit is not a typical conference where you go sit and listen at what speakers have to say, quite the contrary. The Mentor Summit is an unconference where participants are invited to fill empty time slots with their own topic of discussion. Friday Oct 12 Pre-conference meeting The mentors had a short pre-conference meeting to reflect on the Jenkins participation in the 2018 GSoC program, and to plan for 2019. We were joined at this meeting by Lloyd Chang, whom we had met at Jenkins World 2018. Thank you Lloyd for joining us! A few ideas we had for 2019 are: Move project proposals to individual Google Documents Create a template for project proposed by potential mentors and by project champions Create an Organization Administrator Guide for future Jenkins GSoC project admins Other preparations we agreed to work on include a review of the 2018 feedback and the creation of an Epic capturing the action items in preparation for 2019. We are also planning on making progress on the GSoC Budget process described in JEP-8. Summit Starts! The summit started by a welcoming dinner at the Google Cafeteria and an evening session where we were explained how the unconference would work. We proposed a few topics: dealing with CPT lost slots, motivating mentors, and Open Source Hardware ASIC/FPGA. On thing to say is that everyone at the conference had heard of Jenkins, or was already using Jenkins. Lots of people came to tell us their Jenkins experience. I noticed this too - made me feel proud to be part of the Jenkins project. — Jeff Pearce Jenkins GSoC mentor in 2018 Saturday Oct 13 The morning started with a couple of announcements from Google. The first one was that Google is thinking of creating a program called "Google Season of Docs" (GSoD for short), where technical writers would be paired with Open Source Organizations to help them write documentation such as: High-impact tutorials Set of How-To Guides Contributor’s Guide Documentation refactoring Plain documentation We have additional details regarding this in the GSoC Mentor Summit Notes and we quickly concluded that if this program comes to life, Jenkins should be a participating organization. The other announcement made by the GSoC administrators is that GSoC may take a different form in 2020. However, not much more information has been made available at this time. The program has been operating for 13 years, and in 2020 it will have been 15 years. The announcements were followed by a series of morning lightning talks. This is where organizations showcase what their students accomplished during the program. This is when we had a bit of a surprise…​ Oleg who had signed up for the evening lightning talks, was watching the talks while casually preparing slides for his evening presentation. But something unusual happened: many talks were shorter than the 3-minute allotted, and suddenly we were ahead of schedule. That’s when Oleg was called to the stage. I had no idea whether his slides were ready or not since he had just leaned over to me to say that he wanted to talk about all 3 projects we had this year. Not knowing how far he had gotten into refactoring the slides, this was going to be…​ interesting. Being an experienced presenter, Oleg pulled it off brilliantly. The slides were effectively ready (how he managed that I have no idea), but you can see the slides of his lightning talk here: Jenkins Remoting over Apache Kafka. Then there were the unconference sessions. Some of the sessions we attended are: Documentation Attracting and retaining mentors (facilitated by Martin) Organizing and motivating volunteers and mentors Getting students from coding/boot camps involved in open source Retaining students after GSoC Open Event management System GSoC Feedback We have notes for all the session in the main document. Some sessions were captured in separated documents which are linked from the main document, or from this blog post. There were lots of good ideas in those sessions, and we will do what we can next year to implement some of them. Some organizations have said that the key for student retention is to give them responsibilities and tasks after the program is over. We have certainly seen that this year, with one of our students asking for more responsibilities and wanting to know how his plugin project could continue to grow within the Jenkins project (while at the same time help out on another GSoC plugin!). In the evening was the second round of lightning talks. Jeff Pearce presented the Code Coverage API Plugin lightning talk, (he was not caught by surprise). After the lightning talks, we were invited to hang out at the cafeteria and on the patio, to exchange stickers, network with mentors of other organizations, and enjoy late evening snacks, music and of course the chocolate table! Sunday Oct 14 On Sunday, the sessions continued. An interesting session was "Beyond GSoC, What can Google do?". One person got a big round of applause when he said: "Cloud credits". It turns out the GSoC program admins have been trying to get that for us for about 3 years. Google may be big and powerful, but some things are hard and remain hard in the corporate world. An interesting suggestion was made by Oleg, and it would be to have a program with smaller, shorter term commitments, something that would encourage more granular contributions but would not require a 4-month long commitment. This was noted by the GSoC program admins. Then we attended a number of sessions: Open Hardware ASIC/FPGA (facilitated by Martin) Community Metrics Outreachy (Jenkins is participating in this program) Failing a student Burnout (of mentors and org admins) Then the day came to an end with some last words by Google thanking all the mentors and volunteers who run this program in their organizations. Return trip I would now like to add a personal note. After the summit, like many others I fly back home, so I spend the evening at the SFO international terminal waiting for my late night flight. That is where I get to meet more mentors, as some of us still wear our badges and T-Shirts, and also recognize each other from being at the conference. And funny enough, there are so many geeks at that terminal that we may have recruited, among the passengers, a mentor to another org for next year! Want a GSoC student to work on your project in 2019? We have already started the preparations for GSoC 2019. And we cannot do this without the participation of the Jenkins community. We are already looking for: Mentors from the Jenkins Special Interest Groups Mentors from any background and any provenance (being a Jenkins developer is NOT required) Project proposals Students and their proposals Lots of people are afraid that mentoring a student will take a lot of their time. If you feel that way, you are not alone. It does take some time. In my case, I spend 5 to 8 hours per week on mentor tasks (more at the start, less at the end). To make it easier on mentors who likely have full time jobs and life commitments, we define different mentor roles: Project champion co-mentor : this is the mentor who proposes the idea, but may not have all the Jenkins code expertise needed. This mentor works with the student to define the project and acts mostly as a "customer" of the project. This mentor usually know enough about coding to comment on pull-requests with regards to the over quality, style and features of the code. Technical co-mentor : this is the mentor who knows enough about the Jenkins code to guide the student on coding, and to provide Jenkins specific code reviews on pull-requests, but has limited involvement outside the coding activity of the student. There is a third role which is: Subject Matter Expert : these individuals are not mentors, but we reach out to them 3-4 times during the project for advice and guidance, and sometimes complicated programming challenges. If you have questions or are curious about the program, contact us on the GSoC Gitter SIG chat. We would like to emphasize that project proposals are not limited to "big projects". For example, it is perfectly fine to have a proposal that is a collection of related Jira issues that aim to improve your project, or a list of tasks that need to be done for your project. Writing documentation is outside the scope of GSoC, but automating documentation generation, as long as it is mostly about writing code, is within the scope of GSoC. We look forward working with the Jenkins community on GSoC 2019!

It has been a while since the last blogpost about Google Summer of Code in Jenkins. GSoC 2018 has officially finished on August 23, and we had a Jenkins Online Meetup where we had final presentations of the GSoC projects. It is never late to provide more context, so I would like to summarize the results and provide updates of what was happening in Jenkins GSoC Special Interest Group over last 2 months. In this blogpost you can find project status overviews and updates from the Jenkins GSoC SIG. But first of all, I would like to thank all our students, their mentors and to all other contributors who proposed project ideas, participated in student selection, community bonding and further reviews. Google Summer of Code is a major effort which would not be possible without active participation of the Jenkins community Summary This year we started preparing for Google Summer of Code in early December. 14 project ideas and 12 potential mentors we published on our website, and we got dozens of students reaching out to us during the application period. After processing applications, we have selected 4 applications for GSoC. Unfortunately one project got cancelled due to student eligibility issues. So, we had the following projects: Code Coverage API plugin, Remoting over Apache Kafka, and Simple Pull-Request Job Plugin (also known as Pipeline as YAML). All these projects have a significant value to the Jenkins community. They were focused on areas which have been discussed in the community for a long time, but which had no progress so far. Google Summer of Code allowed us to kick-start these projects, and to make significant progress there. All projects have been released and made available in the Jenkins community (common or experimental update centers). In total there were 9 blogposts about GSoC projects on jenkins.io, and also 2 Jenkins Online Meetups. GSoC results have been also presented at DevOps World - Jenkins World conference and the contributor summit. Code Coverage API Plugin Student: Shenyu Zheng, Henan University, China Mentors: Jeff Pearce, Steven Christou, Oleg Nenashev, Supun Wanniarachchi Project page There are many code coverage plugins in Jenkins: Cobertura, JaCoCo, Emma, etc., etc. The problem with these plugins is that each of them implements all code coverage features on their own. So you get different feature sets, UIs, CLI commands and REST APIs. The idea of this project was to unify the existing functionality and offer a new API plugin which other plugins could extend. It would help to simplify existing plugin and to create new plugins for coverage tools. The project has started really well, and we had the first demo after a week of coding. Then Shenyu continued extending the plugin’s functionality over coding periods. Here is the list of the key features offered by the plugin: Flexible data structure for defining and storing coverage metrics within Jenkins Coverage charts and trends Source code navigation REST API for retrieving coverage stats and trends Report aggregation for parallel steps Extension points which allow integrating other plugins In addition to the Code Coverage API Plugin, Shenyu added integration to the Cobertura Plugin and also created a new llvm-cov plugin which is expected to be released soon. After GSoC Shenyu continued contributing to the Jenkins project. He works on the Code Coverage API plugin and also participates in the Chinese Localization SIG. Simple Pull-Request Job Plugin Student: Abhishek Gautam, Visvesvaraya National Institute of technology, India Mentors: Martin d’Anjou, Jeff Knurek, Kristin Whetstone Project page This project focused on introducing a way to easily define pull-request build job definitions in YAML. This project has been shaped a lot during the application period and community bonding, so that the project fit the existing Jenkins Ecosystem better. Finally it was decided to build the new plugin on the top of Pipeline: Multi-Branch Plugin. There was also an idea to offer extra syntax sugar, templating and automatic resolution for common flows, so that users need less time to define Pipelines for common use-cases. The plugin allows defining Pipeline jobs as YAML being stored in SCM. Original design presumed a new job type, but during community bonding and Phase 1 prototyping it was decided to build the plugin on the top of the existing Pipeline ecosystem and extension points. Currently the plugin generates Declarative Pipeline code from YAML so that it gets a lot of Pipeline features out-of-the box. In addition to that, Simple Pull Request Job Plugin uses a an engine provided by the Configuration as Code plugin to convert YAML snippets to Pipeline step definitions. The plugin has been well described by Abhishek in his Pipeline as YAML blogpost in August. Currently it is available in the Experimental Update Center as an alpha version. Pham Vu Tuan, one of our GSoC students, have also joined the plugin team. At the DevOps World - Jenkins World hackfest we had discussions with the Jenkins Pipeline team, and we have a plan towards making this plugin available as an Incubated Pipeline project. The final implementation may change, but in any case the project gave us a working prototype and a lot of information about obstacles we need to resolve. Remoting over Apache Kafka Student: Pham Vu Tuan, Nanyang Technological University, Singapore Mentors: Oleg Nenashev, Supun Wanniarachchi Project page Last but not least, Remoting over Kafka is another challenging project we had. To implement communication between its controllers and agents, Jenkins widely uses home-grown protocol implementations based on TCP ( JNLP 1..4 protocols). There are some performance and stability implementations, and there have been discussions about using an industry-standard message bus or queue. Pham Vu Tuan proposed to use Apache Kafka for it, and after some experiments during community bonding and first coding phase we agreed to go forward with this implementation. During his project Vu Tuan extended Jenkins Core and Remoting to allow implementing an agent communication channel in a plugin. Then he has created a new Remoting over Kafka plugin which is now available in the main Jenkins Update cente. Once the plugin is installed, it is possible to connect to agents over Apache Kafka and execute all types of Jenkins jobs there. There are also official jenkins/remoting-kafka-agent images available on DockerHub. Vu Tuan continued contributing to the Jenkins project after GSoC, currently he maintains the Remoting over Kafka plugin. He visited the DevOps World - Jenkins World US conference in September, presented his GSoC project at the Jenkins Contributor Summit. You can find his slides here. After the conference he also participated in the hackfest where he helped to migrate Jenkins' DNS services to Microsoft Azure. What could we do better? After the end of GSoC we had a Retrospective with GSoC students and mentors. We discussed the issues we encountered during the projects, and ways to improve the student and mentor experience. Main takeaways for us: GSoC projects should be aligned with Jenkins Special Interest Groups (SIGs) or subprojects in order to get a wider list of stakeholders Projects should be aligned with SIG priorities when possible In addition to GSoC SIG meetings and Jenkins Online Meetups during student evaluation, we should also run regular status updates within SIGs so that there more contributors involved in projects We should invest more time into forming mentor teams before the application period starts. This year there were changes in mentor teams after the community bonding started, and it complicated the work We should pay more attention to student eligibility. This year we started from 4 projects, but unfortunately one project (EDA plugins for Jenkins) got cancelled due to the visa limitations the student had. We should do regular office hours for mentors/students so that it is possible to exchange information between GSoC projects within the organization. This year we cancelled them at the end of phase and relied only on regular project meetings and mailing lists, but this is not enough. For me personally the main takeaway is also to reduce direct involvement into the project as a mentor and technical advisor. Doing org administration, logistics and mentorship is not good from a bus factor PoV, and I believe I was pushing my vision too hard in few cases. Will do my best to prevent it next year. If you want to share your feedback and ideas, please reach out to us using the GSoC mailing list. What’s next? In order to improve GSoC organization in Jenkins, we have have created a GSoC Special Interest Group which will be running non-stop as other SIGs in Jenkins. The objective of the SIG is to organize GSoC, work with potential students/mentors, and to help students stay involved in the community after GSoC ends. In this SIG we will have monthly meetings to sync-up on GSoC. If you are interested to contribute, please join the SIG. According to the Retrospective, next year we plan to invest more into communication with mentors. We will also try to tie new project proposals to Jenkins Special Interest Groups so that the students become a part of ongoing coordinated efforts. This weekend Martin d’Anjou, Jeff Pearce and me are participating in the GSoC Mentor summit to share experiences and to study from other GSoC organizations. On October 17 we will have a GSoC SIG meeting to discuss our experience and to discuss next steps. In addition to that, Jenkins Google Summer of Code will be presented at DevOps World - Jenkins World Nice and at the contributor summit. If you plan to visit the conference and you are interested to participate in Google Summer of Code and other community activities, please join us at the contributor summit or stop by at the community booth. And, elephant in the room…​ GSoC 2019. Of course we are going to apply, stay tuned for new announcements. We have already started collecting project ideas for the next year. If you are interested to participate as a student or mentor, please reach out to us using the GSoC SIG mailing list.

I am happy to announce availability of Code Coverage API. These plugins have been recently released as 1.0, and they are now available in the Jenkins Update Center. In this blogpost I will introduce the features and project structure of Code Coverage API plugin. My name is Shenyu Zheng, and I am an undergraduate student in Computer Science and Technology at Henan University from China. Overview Code Coverage API plugin is one of GSoC 2018 Jenkins projects. There are a lot of plugins which currently implement code coverage; however, they all use similar config, charts, and content. So it would be much better if we could have an API plugin which does the most repeated work for those plugins and offers a unified API which can be consumed by other plugins and external tools. My mentors are Steven Christou, Supun Wanniarachchi, Jeff Pearce and Oleg Nenashev. Supported Coverage Formats Embedded JaCoCo Other plugins as an Extension of Code Coverage API plugin Cobertura ( Cobertura Plugin) llvm-cov ( llvm-cov Plugin) Features Modernized coverage chart Coverage trend Source code navigation Parallel pipeline support Reports combining REST API Failed conditions and flexible threshold setting Other small features Modernized Coverage Chart In the summary chart we can see the coverage summary of current coverage metric. In the child summary chart, we can see the coverage summary of each child, also, we can use the range handler to filter item we want to see to reduce the chart size. If we want to see coverage details of the child, we can click the child name to see more information. Coverage Trend We also support coverage trend to show coverage metrics changing between builds. Source Code Navigation You can enable source code navigation by specifying Source File Storing Level to save last build source files (enable source files navigation in current and last build) or save all build source files (enable source files navigation in all builds). You can see source file with coverage information on File level coverage page. Parallel Pipeline Support We support parallel pipeline. You can call the Code Coverage API plugin in different branches like this: node { parallel firstBranch: { publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')] }, secondBranch: { publishCoverage adapters: [jacocoAdapter('jacoco.xml')] } } Reports Combining You can add tag on publishCoverage and Code Coverage API plugin will combine reports have same tag node { parallel firstBranch: { publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')], tag: ‘t’ }, secondBranch: { publishCoverage adapters: [jacocoAdapter('jacoco.xml')], tag: ‘t’ } } REST API We provide a REST API to retrieve coverage data: Coverage result:…​/{buildNumber}/coverage/…​/result/api/\{json|xml\} Trend result:…​/{buildNumber}/coverage/…​/trend/api/\{json|xml\} Coverage result of last build:…​/{buildNumber}/coverage/…​/last/result/api/\{json|xml\} Trend result of last build:…​/{buildNumber}/coverage/…​/last/trend/api/\{json|xml\} Failed Conditions and Flexible Threshold Setting You can set different failed conditions and threholds to control build result. If the thresholds satisfy the failed conditions, it will fail the build. Other Small Features We also have other small features like auto detecting reports, coverage filters, etc. You can find more information about these features in the plugin documentation. Architecture This API plugin will mainly do these things: Find coverage reports according to the user’s config. Use adapters to convert reports into the our standard format. Parse standard format reports, and aggregate them. Show parsed result in a chart. So, we can implement code coverage publishing by simply writing an adapter, and such adapter only needs to do one thing - convert a coverage report into the standard format. The implementation is based on extension points, so new adapters can be created in separate plugins. In order to simplify conversion for XML reports, there is also an abstraction layer which allows creating XSLT-based converters. The below diagram show the architecture of Code Coverage API plugin Implementing a New Coverage Plugin We can implement a coverage plugin by implementing CoverageReportAdapter extension point. For example, by using the provided abstract layer, we can implement JaCoCo simple like this: public final class JacocoReportAdapter extends JavaXMLCoverageReportAdapter { @DataBoundConstructor public JacocoReportAdapter(String path) { super(path); } /** * {@inheritDoc} */ @Override public String getXSL() { return "jacoco-to-standard.xsl"; } /** * {@inheritDoc} */ @Override public String getXSD() { return null; } @Symbol("jacoco") @Extension public static final class JacocoReportAdapterDescriptor extends JavaCoverageReportAdapterDescriptor { public JacocoReportAdapterDescriptor() { super(JacocoReportAdapter.class); } @NonNull @Override public String getDisplayName() { return Messages.JacocoReportAdapter_displayName(); } } } All we need is to extend an abstract layer for XML-based Java report and provide an XSL file to convert the report to our standard format. There are also other extension points which are under development. If you want implement a new coverage format that we did not provide abstract layer, you need to register `CoverageElement`s and implement an simple parser. See llvm-cov Plugin to get more details. Future Tasks Support more coverage tools ( JENKINS-52467, JENKINS-52469 and etc.) Make the UI extensible ( JENKINS-51738) Improve performance ( JENKINS-52982) Phase 3 Presentation Slides Phase 3 Presentation Video Links JIRA Component Project Page Project Repository

About me I am Abhishek Gautam, 3rd year student from Visvesvaraya National Institute of technology, India, Nagpur. I was a member of ACM Chapter and Google student developer club of my college. I am passionate about automation. Project Summary This is a GSoC 2018 project. This project aims to develop a pull request Job Plugin. Users should be able to configure job type using YAML file placed in root directory of the Git repository being the subject of the pull request. The plugin should interact with various platforms like Bitbucket, Github, Gitlab, etc whenever a pull request is created or updated. Plugin detects the presence of certain types of reports at conventional locations, and publish them automatically. If the reports are not present at their respective conventional location, the location of the report can be configured in the YAML file. My mentors are Oleg Nenashev (Org Admin), Martin d’Anjou, Kristin Whetstone, Jeff Knurek Project Repository Project repository Code changes All the pull requests made can be found here List of major pull requests. Phase 1 PR-5 : Git wrappers like clone, pull, checkout, pullChangesOfPullrequest, merge, deleteBranch and merge added. PR-6 : Yaml to Declarative Pipeline code generation. Please see Phase 1 blog post Phase 2 PR-11 : Implemented StepConfigurator using Jenkins configuration as code plugin. PR-19 : Unit tests created for agent and yaml to pipeline generation. Please see Phase 2 blog post Phase 3 PR-25 : Declarative pipeline code generator code exported to extensions for extensibility and support of custom sections Jenkinsfile.yaml example Documentation of Jenkinsfile.yaml and yaml format can be found here Tasks completed in Coding Phase 3 Add unit tests, JenkinsRule tests JENKINS-52495 Refactor snippet generator to extensions ( JENKINS-52491) Plugin overview (Present in README.md) Future tasks Phase 3 Jira Epic Release 1.0 ( JENKINS-52519) Support the “when” Declarative Pipeline directive ( JENKINS-52520) Nice2have: Support hierarchical report types ( JENKINS-52521) Acceptance Test Harness tests JENKINS-52496 Automatic Workspace Cleanup when PR is closed ( JENKINS-51897) Test Multi-Branch Pipeline features support: Support for webhooks ( JENKINS-51941) Check if trusted people have approved a pull request and start build accordingly ( JENKINS-52517) Finalize documentation ( JENKINS-52518) Test the integration with various platforms Bitbucket, Gitlab, Github. Phase 3 evaluation presentation video Video: Link to video evaluation Phase 3 evaluation presentation slides Link to presentation slides My GSoC experience Student applications started on March 12 16:00 UTC and ended on March 27 16:00 UTC. Application period allowed me to explore many new technology and platforms that are making peoples life easy. Before starting of the application period I did not know anything about Jenkins. I found Jenkins organisation on the GSoC organisations page and came to know that I is a CI/CD platform that is used automate various things related to software development. I studied about Jenkins online and went through the problem statements provided by some mentors. I decided that to work on Simple Pull-Request Job Plugin project. Then I wrote a draft proposal for this project and received many comments to refactor the proposal and enhance its quality from the mentors, then finally I submitted my final proposal to Google. I was able to complete most of the tasks decided in Phase 1 and 2. After Phase 2 I was not able to give time to the project because of the placement season in the my college. I modified the code so that other plugin developers can contribute to it by Jenkins extensions. All the mentors made themselves available for most of the weekly calls and provided many valuable suggestions during the entire period of GSoC. Sometimes I was not able to communicate effectively. As communication is the key while working remotely, mentors suggested to communicate more thorough gitter chat. My overall experience of GSoC was good and all the mentors helped me as they can all times. This project allowed me to explore Jenkins and the services offered by it. I am allowed to work on the project after GSoC ends (This is a good thing). How to reach me Email: gautamabhishek46@gmail.com Gitter room: https://gitter.im/Jenkinsci/simple-pull-request-job-plugin References Project repository Project page Gitter chat Bug Tracker Demo Repository

I am very excited to announce that we have recently released 1.0 version of Remoting Kafka Plugin under Jenkins Plugin. You can check the CHANGELOG to see the features included in this release. About me My name is Pham Vu Tuan, I am a final year undergraduate student from Singapore. This is the first time I participate in Google Summer of Code and contribute to an open-source organization. Mentors I have GSoC mentors who help me in this project Oleg Nenashev and Supun Wanniarachchi. Besides that, I also receive great support from developers in remoting project Devin Nusbaum and Jeff Thompson. Overview Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability. This project aims to develop a plugin in order to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins. Benefits to the community Provide a new method to connect agent to controller using Kafka besides existing methods such as JNLP or SSH Build Agents plugin. Help to resolve the existing issues with the TCP protocol between controller and agent communication in Jenkins. Help to resolve traffic prioritization and multi-agent communications issue in Jenkins. Why Kafka? When planning for this project, we want to use traditional message queue system such as ActiveMQ or RabbitMQ. However, after some discussion, we decided to have a try with Kafka with more suitable features with this project: Kafka itself is not a queue like ActiveMQ or RabbitMQ, it is a distributed, replicated commit log. This helps to remove message delivery complexity we have in traditional queue system. We need to support data streaming as a requirement, and Kafka is good at this aspect, which RabbitMQ is lack of. Kafka is said to have a better scalability and good support from the development community. Architecture Overview The project consists of multiple components: Kafka Client Library - new command transport implementation, producer and consumer client logic. Remoting Kafka Plugin - plugin implementation with KafkaGlobalConfiguration, KafkaComputerLauncher and KafkaSecretManager. Remoting Kafka Agent - A custom JAR agent with remoting JAR packaged together with a custom Engine implementation to setup a communication channel with Kafka. The agent is also packaged as a Docker image in DockerHub. All the components are packaged together with Docker Compose. The below diagram is the overview of the current architecture: With this design, controller is not communicating with agent using direct TCP communication anymore, all the communication commands are transfered with Kafka. Features The project is now under the third coding phase and we have some features available in 1.0 release. 1. Kafka Global Configuration with support of credentials plugin to store secrets. 2. Launch agent with Kafka Launcher. 3. Launch agent from CLI using agent JAR with secret provided to ensure security. 4. Run jobs, pipeline using Kafka agent. 5. Kafka communication between controller and agent. Remoting operations are being executed over Kafka. In the log you may see: Command execution (SlaveInstallerFactoryImpl.isWindows()) Classloading (Classloader.fetch()) Log streaming (Pipe.chunk()) How to run demo We have setup a ready-to-fly demo for this plugin. You can try to run a demo of the plugin by following this instruction. Features in the demo: Docker Compose starts preconfigured controller and agent instance, they connect automatically using Kafka launcher. Kafka is secured and encrypted with SSL. There few demo jobs in the instance so that a user can launch a job on the agent. Kakfa Manager supported in localhost:9000 to support monitoring of Kafka cluster. Phase 2 Presentation Slides Phase 2 Presentation Video Links GitHub Repository Wiki Plugin Site Project Info Introduction Blogpost Phase 1 Evaluation Slides Phase 2 Evaluation Slides Phase 1 Evaluation Video Phase 2 Evaluation Video

About me I am Abhishek Gautam, 3rd year student from Visvesvaraya National Institute of technology, India, Nagpur. I was a member of ACM Chapter and Google student developer club of my college. I am passionate about automation. Project Summary This is a GSoC 2018 project. This project aims to develop a pull request Job Plugin. Users should be able to configure job type using YAML file placed in root directory of the Git repository being the subject of the pull request. The plugin should interact with various platforms like Bitbucket, Github, Gitlab, etc whenever a pull request is created or updated. Plugin detects the presence of certain types of reports at conventional locations, and publish them automatically. If the reports are not present at their respective conventional location, the location of the report can be configured in the YAML file. My mentors are Oleg Nenashev (Org Admin), Martin d’Anjou, Kristin Whetstone, Jeff Knurek Benefits to the community Project administrators will be able to handle pull request builds more easily. Build specifications for pull requests can be written in a concise declarative format. Build reports will be automatically published to Github, Bitbucket, etc. Build status updates will be sent to git servers automatically. Users will not have to deal with pipeline code. If there will be no merge conflicts or build failures, the PR can be merged into target branch. Phase 1 blog post Please see Phase 1 blog post Implementations till now Alpha version of the plugin is released. It supports all features of Multi-Branch Pipeline and offers the following features. Build description is defined via YAML file stored within the SCM repo. This plugin will depend on GitHub plugin, Bitbucket plugin, Gitlab plugin if users will be using respective platforms for their repositories. Conversion of YAML to Declarative Pipeline: A class YamlToPipeline is written which will load the "Jenkinsfile.yaml" and make use of PipelineSnippetGenerator class to generate Declarative pipeline code. Reporting of results, only xml report types is supported for now. Use of Yaml file (Jenkinsfile.yaml) from target branch. Git Push step: To push the changes of pull request to the target branch. This is implemented using git-plugin, PushCommand is used for this from git-plugin. credentialId, branch name and repository url for interacting with Github, Bitbucket, etc will be taken automatically from "Branch-Source" (Users have to fill this details of branch source in job configuration UI). (You can see How to run the demo) StepConfigurator: To generate pipeline code for all supported steps in Jenkins. This is using Jenkins configuration-as-code plugin (JCasC plugin) to configure a particular step object and then that step object is passed to Snippetizer.object2Groovy() method to generate the script of that step. Jenkinsfile.yaml example For the phase 1 prototype demonstration, the following yaml file was used. Note that this format is subject to change in the next phases of the project, as we formalise the yaml format definition. # Docker image agent example agent: label: my_label customWorkspace: path_to_workspace dockerImage: maven:3-alpine args: -v /tmp:/tmp tools: maven : maven_3.0.1 jdk : jdk8 configuration: # Push PR changes to the target branch if the build succeeds. # default value is false pushPrOnSuccess: false # Trusted user to approve pull requests prApprovers: - username1 - username2 - username3 environment: variables: variable_1: value_1 variable_2: value_2 # Credentials contains only two fields. credentialId must be present in the Jenkins Credentials credentials: - credentialId : fileCredentialId variable : FILE # In user scripts Username and Password can be accessed by LOGIN_USR and LOGIN_PSW # respectively as environment variales - credentialId : dummyGitRepo variable : LOGIN stages: - name: stage1 agent: any steps: - sh: "scripts/hello" - sleep: time: 2 unit: SECONDS - sleep: 2 - junit: testResults: "target/**.xml" allowEmptyResults: true testDataPublishers: - AutomateTestDataPublisher - JunitResultPublisher: urlOverride: "urlOverride" # Post section for "stage1". All Conditions which are available in Jenkins # declarative pipeline are supported post: failure: - sh: "scripts/hello" # Outer post section. Just like declarative pipeline. post: always: - sh: "scripts/hello" Coding Phase 2 plans (Completed) Decide a proper YAML format to use for Jenkinsfile.yaml Create Step Configurator for SPRP plugin. JENKINS-51637. This will enable users to use Pipeline steps in Jenkinsfile.yaml. Automatic indentation generation in the generated PipelineSnippetGenerator class. Write tests for the plugin. Coding Phase 3 plans Test Multi-Branch Pipeline features support: Support for webhooks ( JENKINS-51941) Check if trusted people have approved a pull request and start build accordingly ( JENKINS-52517) Finalize documentation ( JENKINS-52518) Release 1.0 ( JENKINS-52519) Plugin overview blog post Coding Phase 3 plans after release Support the “when” Declarative Pipeline directive ( JENKINS-52520) Nice2have: Support hierarchical report types ( JENKINS-52521) Add unit tests, JenkinsRule tests, and ATH tests ( JENKINS-52495, JENKINS-52496) Automatic Workspace Cleanup when PR is closed ( JENKINS-51897) Refactor snippet generator to extensions ( JENKINS-52491) Phase 3 Jira Epic Phase 2 evaluation presentation video Video: Phase 2 evaluation presentation slides How to reach me Email: gautamabhishek46@gmail.com Gitter room: https://gitter.im/Jenkinsci/simple-pull-request-job-plugin References Project repository Project page Gitter chat Bug Tracker Demo Repository Phase 2 Presentation video (July 12, 2018) Phase 2 Presentation Slides (July 12, 2018)

I am happy to announce that we have recently released an alpha version of Remoting Kafka Plugin to the Experimental Update Center. You can check the CHANGELOG to see the features included in this initial release. Overview Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability. Remoting Kafka Plugin is a plugin developed under Jenkins Google Summer of Code 2018. The plugin is developed to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins. A quick introduction of the project can be found in this introduction blogpost. How to use the plugin? The instructions to run the plugin in alpha version are written here. Feel free to have a try and let us know your feedback on Gitter or the mailing list. Links Alpha Changelog Introduction Blogpost GitHub Repository Project Page Phase 1 Presentation Video Phase 1 Presentation Slides

About me My name is Pham Vu Tuan, I am a final year undergraduate student from Singapore. This is the first time I participate in Google Summer of Code and contribute to an open-source organization. I am very excited to contribute this summer. Mentors I have GSoC mentors who help me in this project Oleg Nenashev and Supun Wanniarachchi. Besides that, I also receive great support from developers in remoting project Devin Nusbaum and Jeff Thompson. Overview Current versions of Jenkins Remoting are based on the TCP protocol. If it fails, the agent connection and the build fails as well. There are also issues with traffic prioritization and multi-agent communications, which impact Jenkins stability and scalability. This project aims to develop a plugin in order to add support of a popular message queue/bus technology (Kafka) as a fault-tolerant communication layer in Jenkins. Why Kafka? When planning for this project, we want to use traditional message queue system such as ActiveMQ or RabbitMQ. However, after some discussion, we decided to have a try with Kafka with more suitable features with this project: Kafka itself is not a queue like ActiveMQ or RabbitMQ, it is a distributed, replicated commit log. This helps to remove message delivery complexity we have in traditional queue system. We need to support data streaming as a requirement, and Kafka is good at this aspect, which RabbitMQ is lack of. Kafka is said to have a better scalability and good support from the development community. Current State The project is reaching the end of the first phase and here are things we have achieved so far: Setup project as a set of Docker Compose components: Kafka cluster, Jenkins controller (with plugin) and a custom agent (JAR). Create a PoC with new command transport implementation to support Kafka, which involves of command invocation, RMI, classloading and data streaming. Make neccessary changes in Remoting and Jenkins core to make them extensible for the use of this project. Decide to use Kafka as a suitable final implementation. We planned to release an alpha version of this plugin by the end of this phase, but decided to move this release to the second phase because we need to wait for remoting and core patches to be released. Architecture Overview The project consists of multiple components: Kafka Client Library - new command transport implementation, producer and consumer client logic. Remoting Kafka Plugin - plugin implementation with KafkaGlobalConfiguration and KafkaComputerLauncher. Remoting Kafka Agent - A custom JAR agent with remoting JAR packaged together with a custom Engine implementation to setup a communication channel with Kafka. All the components are packaged together with Docker Compose. The below diagram is the overview of the current architecture: With this design, controller is not communicating with agent using direct TCP communication anymore, all the communication commands are transfered with Kafka. Features 1. Kafka Global Configuration 2. Custom agent start up as a JAR User can start running an agent with the following command: 3. Launch agents with Kafka 4. Commands transferred between controller and agent over Kafka Remoting operations are being executed over Kafka. In the log you may see: Classloading (Classloader.fetch()) Log streaming (Pipe.chunk()) 5. Run jobs with remoting Kafka It is possible to run jobs on Agents connected over Kafka Next Phase Plan Here are the tasks planned for the next phase: Support security for controller-agent connection: Kafka authentication/authorization ( JENKINS-51472, JENKINS-51473). Agent secrets ( JENKINS-51470). Improve Kafka producer-consumer model to ensure reliability ( JENKINS-51942). Bug fixing. Release alpha version and address feedback ( JENKINS-51713). How to run demo You can try to run a demo of the plugin by following the instruction. Links GitHub Repository Project Page Phase 1 Presentation Video Phase 1 Presentation Slides