Hello everyone, This is the final blog post for the Custom Distribution Service project during the Google Summer of Code timeline. I have mixed feelings since we are almost near the finish line for one of the most amazing open source programs out there. However, it is time to wrap things up for this project and achieve a state where the project can be built upon and extended further. This phase has been super busy with respect to the bug fixes, testing and getting the project hosted, so let us get straight into the phase 3 updates. Fixes and Code quality assurance Set Jenkinsfile agent to linux We realised that the build was failing on windows and that there was not really a use-case for running it on windows for right now. Maybe it could be on a future roadmap. Therefore, we decided to shift the testing to only linux agents with respect to running the tests on the jenkins server. Pull Request #116 Backend port error message Spring boot has a default message on the port:8080 and therefore we wanted to change it to a custom message on the backend. So the major takeaway here is that we needed to implement the Error Controller interface and include a custom message in it. This was technical debt from the last phase and was completed and merged during this phase. Pull Request #92 PMD Analysis In order to enhance the quality of the code, the PMD source code analyser was applied to the project. It helped me catch tons of errors. When the initial PMD check was run and we found approximately 162 PMD errors. We realised some of them were not relevant and some of them could be fixed later. Pull Request #102 Findbugs Analysis Another tool to improve code quality that we included in this phase was findbugs. It did catch around 5-10 bugs in my code which I immediately resolved. Most of them were around the Closeable HTTP Request and an easy fix was the try with resources. Pull Request #118 Jacoco Code Coverage We needed to make sure most of the code we write had proper coverage for all branches and lines. Therefore we decided to include a JaCoco Code Coverage reporter that helped us find the uncovered lines and areas we need to improve coverage on. Pull Request #103 Remove JCasC generation While developing the service we quickly realised that the generation of the war package broke if we included a configuration as code section but did not provide a path to the corresponding required yml file. Therefore we took a decision to remove the casc section all together. Maybe it will comeback in a future patch Pull Request link: #127 Issue link: #65 Minor Fixes Logging Fix: #99 Docs Fix : link: #120 Update Center Dump Fix : link: #125 Class Path Fix: link: #126 Release Drafter Addition: link: #136 Front end Community Config Navigation link There was no community configuration link present for navigation which was added here. Now it is easier to navigate to the community page from the home page itself. Pull Request #100 Docker updates Build everything with Docker This was one of the major changes this phase with respect to making the service very easy to spin up locally, this change will greatly help community adoption since it eliminates the tools one needs to install locally. Initially the process was to run maven locally, generate all of the files and then copy all of its contents into the container. However, with this change we are going to generate all of the files inside the docker container itself. Allowing the user to just run a couple of commands to get the service up and running. So some of the major changes we did with respect to the dockerfile was: a) Copy all of the configuration files and pom.xml into the container. b) Run the command mvn clean package inside the container which generates the jar. c) Run the jar inside the container. Pull Request #104 Hosting updates This process was supposed to be a future roadmap, however the infra team approved and was super helpful in making this process as smooth as possible. Thanks to Gavin, Tim and Oblak for making this possible. Here is the google group dicussion The project has now been hosted here as a preview. It still needs some fixes to be fully functional. Infra Docker PR #131 Infra Project Addition PR link: #393 Testing Updates Unit test the services With respect to community hosting and adoption, testing of the service one of the most important and major milestones for this phase was to test the majority of the code and we have completed the testing with flying colors. All of the services have been completely unit tested, which is a major accomplishment. For the testing of the service we decided to go with wiremock which can be used to mock external services. Kezhi’s comment helped us to understand what we needed to do since he had done something quite similar in his Github Checks API project. So we basically wiremocked the update-center url and made sure we were getting the accurate response with appropriate control flow logic tested. wireMockRule.stubFor(get(urlPathMatching("/getUpdateCenter")) .willReturn(aResponse() .withStatus(200) .withHeader("Content-Type", "application/json") .withBody(updateCenterBody))); Pull Request #105 Add Update Center controller tests Another major testing change involved testing the controllers. For this we decided to use the wiremock library in java to mock the server response when the controllers were invoked. For example: If I have a controller that serves in an api called /api/plugin/getPluginList wiremock can be used to stub out its response when the system is under test. So we use something like this to test it out. when(updateService.downloadUpdateCenterJSON()).thenReturn(util.convertPayloadToJSON(dummyUpdateBody)) When the particular controller is called the underlying service is mocked and it returns a response according to the one provided by us. To find more details the PR is here. Pull Request #106 Add Packager Controller Tests Along with the update center controller tests another controller that needed to be tested was the packager controller. Also we needed to make sure that all the branches for the controllers were properly tested. Additional details can be found in the PR below. Pull Request #133 Docker Compose Tests One problem that we faced the entire phase was the docker containers. We regularly found out that due to some changes in the codebase the docker container build sometimes broke, or even sometimes the inner api’s seemed to malfunction. In order to counteract that we decided to come up with some tests locally. So what I did was basically introduce a set of bash scripts that would do the following: a) Build the container using the docker-compose command. b) Run the container. c) Test the api’s using the exposed port. d) Teardown the running containers. Pull Request #131 User Documentation We also included a user docs guide so that it makes it super easy to get started with the service. Pull Request #145 Future Roadmap This has been a super exciting project to work on and I can definitely see this project being built upon and extended in the future. I would like to talk about some of the features that are left to come in and can be taken up in a future roadmap discussion a) JCasC Support : Description: Support the generation of a Jenkins Configuration as Code file asking the user interactively for the plugins they select what would be the configuration they would want eg: If the user selects the slack plugin we need to ask him questions like what is the slack channel? what is the token? etc, and on the basis of this generate a casc file. This feature was initially planned to go into the service but we realised this is a project in its own capacity. b) Auto Pull Request Creation : Description: Allow users to create a configuration file and immediately open a pull request on github without leaving the user interface. This was originally planned using a github bot and we started the work on it. But we were in doubt if the service would be hosted or not and therefore put the development on hold. You can find the pull requests here: Github Controller #72 Pull Request Creation Functions #66 c) Synergy with Image Controller Description: This feature requires some planning, some of the questions we can ask are: a) Can we generate the images (i.e Image Controller). b) Can we have the service as a multipurpose generator ? Statistics This phase has been the busiest of all phases and it has involved a lot of work, more than I had initially expected in the phase. Although lines of code added is not an indication of work done, however 800 lines of Code added is a real personal milestone for me. Pull Requests Opened 26 Lines of Code Added 1096 Lines of Docs Added 200 Other links Gitter Channel Link GSoC Proposal Design Document Daily Notes

Hello everyone, It is time to wrap up another successfull phase for the custom distribution service project, and we have incorporated most of the features that we had planned at the start of the phase. It has been an immense learning curve for me and the entire team. To understand what the project is about and the past progress, please refer to the phase one blogpost here. Front-End Filters for Plugins In the previous phase we implemented the ability to add plugins to the configuration, and the ability to search these plugins via a search bar. Sometimes though we would like to filter these plugins based on their usage, popularity, stars etc. Hence we have added a certain set of filters to these plugins. We support only four major filters for now. They are: Title Most installed Relevance Trending Filter implementation The major heavy lifting is done by the plugin api which takes in the necessary parameters and returns the relevant plugins in the form of a json object, here is an example of the api call url: const url = https://plugins.jenkins.io/api/plugins?$params . For details, see: Feature request #9 Pull Request #76 Community Configurations One major deliverable for the project was the ability for users to share the configurations developed by them, so that they can be used widely within the community. For example we see quite a lot of jenkins configurations involve being run on AWS and kubernetes and so on. Therefore it would be really good for the community to have a place to find and run these configurations right out of the box. Design Decision The major design decision taken here was whether to include the configurations inside the repository or to have them in a completely new repository. Let us talk about both these approaches. Having the configurations in the current repository: This allows us to have all of the relevant configurations inside the repository itself, and so users would not have to go fetch this in different repositories. We could have issues with the release cycle and dependencies since, it would have to happen along with the custom distribution service project releases. Having the configurations in a different repository: This allows us to manage all of the configurations and the relevant dependencies separately and easily, thus avoiding any release conflict with the current repository. However it would be a bit difficult if users were to not find this repository. Decision : We still cannot quite agree on what is the best method so for now, I have included the url from which the community configurations are picked up as a configuration variable in the.env file which can be configured later and therefore it can be up to the user to configure. Another advantage of having it configurable, is that the user can decide to load configurations which are private to his organization as well. For details, see: Issue #6161 Pull Request #73 Back-End War Generation The ability to generate and download war files has finally been achieved, the reason this feature took so long to complete is because we had some difficulty in implementing the war generation and its tests. However this has been completed and can now be tested successfully. Things to take care while generating war files In its current state the war generation cannot include casc.yml or groovy files if they are included in the configuration they would have to be added externally. There is an issue opened here. The war file generation would yell at you if you tried to build a war file with a jcasc file configuration. For details, see: Issue #60 Pull Request #68 Pull Request Creation This feature was included in the design document that I created after my GSoC selection. It involves the ability to create pull requests via the front-end of the service. The User Story behind this feature was that If I want to share a configuration with the community and I do not quite know how to use github or I do not want to do it via the terminal. This feature includes creation of a bot that handles the creation of pull requests in the repository. This bot would have to be installed by the jenkins organization in this repository and the bot would handle the rest. For details, see: Issue #59 Pull Request #72 Disclaimer: This feature has however been put on the back-burner for now because we are focusing on getting the project to be self hosted and therefore would like to implement this once we have a clear path for the project to be hosted by the jenkins-infra team.If you would like to participate in the discussion here are the links for the pull requests, PR 1 and link: PR 2, or you can even jump in our gitter channel. If you have been following my posts, I mentioned in my second week blog post that pulling in the json file consisting of more than 1600 plugins took a bit more time that my liking. We managed to solve that issue using a caching mechanism, so now the files are pulled in the first time you start the service and downloaded in a temporary folder. The next time you want to view the plugin cards they are pulled in directly from the temp directory bam ! thereby reducing time. For details see Pull Request #90 Fixes and improvements Port 8080 Port 8080 now does have a message instead of a whitelabel error message which is present by default in the spring-boot tomcat server setup. Turns out it requires overriding a particular class, and inserting a custom message For details, see: Pull Request #92 War Generation Till now while you were generating the war file, if something went wrong during genration the service would not complain it would just swallow the error and throw back a corrupted war file, however now we have added an error support feature that will alert you when something goes wrong, the error is not very informative as of now, but we are working on making it more informative in the future. For details, see: War generation error handling #91 Add Github controller and jwt helper #66 Dockerfile One of the major milestones of this phase was to have a project that can be self hosted, needless to say we needed the dockerfile i.e docker-compose.yml to spin the project with a few commands. The major issue we faced here was that there was a bit of a problem making the two containers talk to each other. Let me give you a little bit of context here. Our docker-compose is constructed using two separate dockerfiles one for the backend of the service and the other for the front-end. The backend makes api calls to the front-end via the proxy url i.e localhost:8080. We now had to change this since the network bridge between the two containers spoke to each other via the backend-server name i.e app-server. To brige that gap we have this PR that ensured that the docker compose works flawlessly. For details, see: Pull Request #82 However there is a minor draw-back of the above approach was now the entire project just relied on the docker compose and could not run using the simple combination of npm and maven since the proxy was different. In order to fix this I decided to follow a multiple environment approach, where we have multiple environment files that pick up the correct proxy and insert it at build time, to elaborate further we have two environment files, (using the env-cmd library ).env and the docker.env and we insert, the correct file depending on how you want to build the project. For instance if you want to run it using the dockerfile the command that is run under the hood is something along these lines — npm --env-cmd -f docker.env start scripts. For details, see: Pull Request #88 Other links Gitter Channel Link GSoC Proposal Design Document Daily Notes Demo

Hello, After an eventful community bonding period we finally entered into the coding phase. This blog post will summarize the work done till the midterm of the coding phases i.e. week 6. If some of the topics here require a more detailed explanation, I will write a separate blog post. These blogs posts will not have a very defined format but would cover all of the user stories or features implemented. Project Summary The main idea behind the project is to build a customizable jenkins distribution service that could be used to build tailor-made jenkins distributions. The service would provide users with a simple interface to select the configurations they want to build the instance with eg: plugins, authorization matrices etc. Furthermore it would include a section for sharing community created distros so that users can find and download already built jenkins war/configuration files to use out of the box. Quick review Pull Requests Opened 38 Github Issues completed 36 Details I have written separate blog posts for every week in GSoC and the intricate details for each of them can be found at their respective blog pages. I am including a summary for every phase supported with the respective links. Community Bonding This year GSoC had a longer community bonding than any of the previous editions due to the Coronavirus pandemic and therefore this gave me a lot of time to explore, so I spent it by building a prototype for my project. I realised some of the blockages I might face early on, and therefore it gave me more clarity in terms of how I can proceed. I also spent this time preparing a design document which you can find here. Community Bonding Blog Week 1 In week one, I spent time getting used to the tech stack I would be using, I was pretty familiar with Spring Boot but React was something I was going to be using for the first time, so I spent time studying more about it. I also got the project page ready, the issues I was going to tackle and the milestones that I had to achieve before the evaluation. I also spent a bit of time setting up the home page and a bit of front-end components. Week 1 Blog Week 2 Once we were done with the initial setup, it was time to work on the core of the project. In the second week, I worked on generating the package configuration and the plugin list dummy display page setup. I also ran into issues with the Jenkinsfile so the majority of time was spent fixing it. Finally I managed to get around those problems. You can read more about it in the Week 2 Blog post. Week 2 Blog Week 3 The last week was spent cleaning up most of the code and getting the remaining milestones in. This was probably the hardest part of phase 1 because it involved connecting the front and back end of the project.You can read more about it here. Week 3 Midterm Update The second phase has been going on for the past 3 weeks and we have already accomplished a majority of the deliverables including community configurations, war downloading and filtering of plugins. More details about the mid term report can be found here. Midterm Update Getting the Code The Custom Distribution Service was created from scratch during GSoC and can be found here on Github. Other links GSoC Proposal Design Document Daily Notes Demo Feedback channel Gitter Channel Link.

Configuration-as-code plugin Problem Statement: Convert the existing schema validation workflow from the current scripting language in the Jenkins Configuration as Code Plugin to a Java based rewrite thereby enhancing its readablity and testability supported by a testing framework for the same. Enhance developer experience by developing a VSCode Plugin to facilitate autocompletion and validation which would help the developer write correct yaml files before application to a Jenkins Instance. The Configuration as Code plugin has been designed as an opinionated way to configure Jenkins based on human-readable declarative configuration files. Writing such a file should be feasible without being a Jenkins expert, just translating into code a configuration process one is used to executing in the web UI. The plugin uses a schema to verify the files being applied to the Jenkins instance. With the new JSON Schema being enabled developers can now test their yaml file against it. The schema checks the descriptors i.e. configuration that can be applied to a plugin or Jenkins core, the correct type is used and help text is provided in some cases. VSCode allows us to test out the schema right out of the box with some modifications. This project was built as part of the Community Bridge initiative which is a platform created by the Linux Foundation to empower developers — and the individuals and companies who support them — to advance sustainability, security, and diversity in open source technology. You can take a look at the Jenkins Community Bridge Project Page Steps to Enable the Schema Validation a) The first step includes installing the JCasC Plugin for Visual Studio Code and opening up the extension via the extension list. Shortcut for opening the extension list in VSCode editor using Ctrl + Shift + X. b) In order to enable validation we need to include it in the workspace settings. Navigate to File and then Preference and then Settings. Inside settings search for json and inside settings.json include the following configuration. { "yaml.schemas": { "schema.json": "y[a]?ml" } } You can specify a glob pattern as the value for schema.json which is the file name for the schema. This would apply the schema to all yaml files. eg:.[y[a]?ml] c) The following tasks can be done using VSCode: a) Auto completion (Ctrl + Space): Auto completes on all commands. b) Document Outlining (Ctrl + Shift + O): Provides the document outlining of all completed nodes in the file. d) Create a new file under the work directory called jenkins.yml. For example consider the following contents for the file: jenkins: systemMessage: “Hello World” numExecutors: 2 The above yaml file is valid according to the schema and vscode should provide you with validation and autocompletion for the same. Screenshots We are holding an online meetup on the 26th February regarding this plugin and how you could use it to validate your YAML configuration files. For any suggestions or dicussions regarding the schema feel free to join our gitter channel. Issues can be created on Github.

Community Bridge Introduction Community Bridge is an initiative by the Linux Foundation to accelerate the adoption, innovation and sustainability of open source projects. I came across this initiative in a blog post. I had been contributing to Jenkins at the time and decided to have a chat with Oleg Nenashev and Tracy Miranda regarding the possibility of a project under the Community Bridge initiative. Fortunately for me JCasC ( Jenkins Configuration as Code) had the mentors as well as the project idea in place to start a project. After a few regular meetings we ironed out the details of the programme and on August 7th I began with my journey! JCasC Developer Tools — JSON Schema JSON files when submitted to a server undergo a validation to determine whether the values and the format are correct and that they conform to a well defined schema, this schema is known as a JSON Schema. A YAML file can also be validated using a JSON Schema. The main premise of JCasC is to load YAML files written by developers into the Jenkins instance. An example of a JCasC YAML file is: --- jenkins: systemMessage: “Hello World” numExecutors:2 --- The above YAML configuration will configure Jenkins to display a message Hello world with the number of executors set to two. In order to validate the YAML we have a schema. This schema is written using jelly files (Executable XML files) and currently it is not a valid schema. The first phase of the project is based around rewriting the schema generation to java and developing a better test framework for it, because currently the schema is not testable. Phase 1 — JCasC Dev Tools The first week I got into studying how the schema was generated.With the support of two of my awesome mentors Tim Jacomb and Joseph Peterson I finally got an understanding of the current schema. So JCasC has a set of configurators for describing a YAML file. They are: a) Base Configurators b) Hetero Describable Configurators c) Data Bound Configurators These configurators together successfully describe a YAML file. We proceeded to generate the schema with the help of individual description of each of these configurators. The JSON Schema has a set of components, consider the above yaml file as an example: --- { "jenkins": { "type": "object", "properties": { "systemMessage": { "type": "string" }, "numExecutors": { "type": "integer" } } } } --- So here Jenkins is the base configurator and it has a set of attributes viz systemMessage and numExecutors, so our schema needs to be able to describe a set of attributes for every field in the schema. Some of the fields that our JSON Schema uses to describe the YAML are: 1) type : String, int, Boolean etc. 2) properties : A set of fields describing the part field. 3) id: Unique Identifier for the field Thus the above schema successfully verifies the YAML configuration. JAVA Rewrite We used JSON Objects to build components of the schema.The basic flow that is followed to generate the schema is as follows: a) Iterate through the Base Configurators. b) Iterate over the list of Base Configurator.Attributes and add each attribute to the schema. c) Iterate over the HeteroDescribable Configurators and add each configurator to the schema along with its required properties. The set of PR’s Resolved during Phase 1 are as follows: a) Basic Schema b) Schema Doc Page c) Test Framework d) Nested Schema That is all from me guys, I am currently preparing for phase 2 and working towards fixing any pending issues of Phase 1. Thanks for reading. Phase 2 Goals: We would primarily target VSCode integration in phase 2 with the aim of: a)Validation of JCasC YAML files with the schema b)Autocompletion c)Integration with a live Jenkins instance. Contributions We would love to get feedback from you on the stuff we are working on. Contributions to the project would be highly appreciated. a) Gitter Chat b) Github Repository c) Project Introduction