Recent Blog Posts

Welcome back ! This blog post is briefing my coding phase 1 in Jenkins Machine Learning Plugin for this GSoC 2020. After a fresh introduction of community bonding, On June 1st, coding of GSoC had started officially with phase 1. At this point, every GSoC student should be expected to have a rigid plan with their entire project. With the guidance of mentors I was able to complete a design document and timeline which can be slightly adjustable during the coding. The coding phase was more about coding and discussion. Quick review Pull Requests 21 Jira Issues 11 Major Tasks 3 Completed 3 In progress 0 Week 1 I have to ensure that I have a solid architecture for implementing the core of this plugin such that perhaps I or future community will be able to develop R and Julia kernels for this plugin. Factory method design patterns are suitable when users need different types of products ( Python, R and Julia) without knowing much about the internal infrastructure ( Manager of these interpreters ). All the base classes were implemented this week. Design the Kernel connectors Initiate the interpreter Close the connection Add simple tests Update pom.xml More than these changes, repo was updated with pull request template and licence header. Readme was extended a little at the end of the week. Issues and Challenges Git rebase and squash Tests invokes ipython client in the server failed during the CI build Week 2 With the help of a design document, I had a plan to do the configurations globally and using the Abstract Folder property I could save the configuration and retrieve for the job configuartion. I used to reference some other well developed plugin for the structure of code. That helped me a lot while I was coding. Our first official contributor has popped out his pull request. Form validations and helper html will be a great help in the user point of view as well as developers. A minor bug was fixed with the guidance of mentors by writing tests with ‘Jenkins WebClient`. Until the end of the week, builder class of the plugin has been implemented with lots of research and discussion. Finally, Test connection was added to the global configuration page to start the connection and test it. A single issue that blocked me using py4j authentication about zeppelin-python was reported in Jira. Server Configuration Issues and challenges Backend depends on Apache zeppelin-python API to connect IPython Find relevant extension points to extend the plugin Week 3 Earlier in this week, we were trying to merge our IPython builder PR without any memory leaks or bugs that will cause the system to be devastating while running this plugin. Later, this whole week I was implementing a file parser that could copy the necessary files and had the ability to accomplish the file conversion. Supported file types Python (.py) JSON (Zeppelin notebooks format) IPython builder was able to run Jupyter Notebooks and Zeppelin formatted JSON files at the end of the 3rd week. Minor issues were fixed in the code. We used ANSI color plugin to fix the abnormal view of error messages produced by the ipython kernel. Copying and converting Jupyter Notebook Issues and Challenges Python error messages could not be displayed in rich format If a job is running at user level, but if the python code access file/file path which is not authorized to the user, it returns a permission denied message. While running on agent, notebook has to be written/copied to agent workspace Artifacts should be maintained/reachable from controller after build. Week 4 As all the major tasks has done, the demo preparation and plan for a experimental release was carried during the last week. There were lots of research on how to connect to a existing kernel in remote. Demo and presentation were prepared along the week. Issues and Challenges Releasing the first version was bit late Knowledge transfer How to debug the code through IntelliJ Edit configuration → Add new Configuration → Maven Command line → type hpi:run Click the debug icon on the toolbar or go to Run menu then Debug How to setup to test the plugin Setup JDK 8 and Maven 3.5.* Create a directory $ mkdir machine-learning-plugin Create a virtual environment $ virtualenv venv Activate your virtual environment $ source venv/bin/activate Run $ which python to ensure your python path $ git clone https://github.com/jenkinsci/machine-learning-plugin.git Run $ mvn clean install from the machine-learning-plugin directory Run $ mvn hpi:run to start Jenkins with the plugin Set up the builder with localhost and other parameters Create a job Write python code like print(“plugin works”) Build the job Issues and bugs JENKINS-62528 Issues on Jenkins build in the plugin repository JENKINS-62621 Global configuration for IPython servers JENKINS-62649 Implementation of IPython Builder JENKINS-62711 File parser to copy source files to workspace JENKINS-62733 Python errors are not displayed properly in console log JENKINS-62735 Send/Receive necessary files from controller/agent to agent/controller JENKINS-62593 Improve the documentation JENKINS-62742 Increase Test coverage Resources and Links Github Project page Design document

Externalizing fingerprint storage for Jenkins is a Google Summer of Code 2020 project. We are working on building a pluggable storage engine for fingerprints (see jep:226[]). File fingerprinting is a way to track which version of a file is being used by a job/build, making dependency tracking easy. The fingerprint engine of Jenkins can track usages of artifacts, credentials, files, etc. within the system. Currently, it does this by maintaining a local XML-based database which leads to dependence on the physical disk of the Jenkins controller. Allowing fingerprint storage to be moved to external storages decreases the dependence of Jenkins instances on the physical disk space and also allows for tracking the flow of fingerprints across instances of Jenkins connected to the same external storage. Advantages of using external storage drivers: Remove dependence on Jenkins controller disk storage Can configure pay-as-you-use cloud storages Easy Backup Management Better Reliability and Availability Fingerprints can be tracked across Jenkins instances Along with this API, we are also working on a reference implementation in the form of a plugin, powered by Redis. As phase 1 of this project comes to an end, this blog post serves as a summary of the progress we made to the entire Jenkins community. Current State The new API introduced in Jenkins core is under review. Once merged, it will offer developers to extend it to build external fingerprint storage plugins. The Redis Fingerprint Storage Plugin is alpha release ready. We would immensely appreciate any feedback. External Fingerprint Storage Demo Introducing the new API for plugin developers With PR-4731, we introduce a new fingerprint storage API, allowing configuring custom storage engines. We exposed the following methods in the new FingerprintStorage class: void save() Saves the given Fingerprint in the storage. Fingerprint load(String id) Returns the Fingerprint with the given unique ID. The unique ID for a fingerprint is defined by Fingerprint#getHashString(). void delete(String id) Deletes the Fingerprint with the given unique ID. boolean isReady() Returns true if there is some data in the fingerprint database corresponding to the particular Jenkins instance. Introducing Redis Fingerprint Storage Plugin Redis Fingerprint Storage Plugin uses the new External Fingerprint Storage API to store the fingerprints in a Redis instance. Installation: The alpha release (version 0.1-alpha-1) for the plugin was drafted, and can be installed using the experimental update center. Follow along the following steps after running Jenkins to download and install the plugin: Select Manage Jenkins Select Manage Plugins Go to Advanced tab Configure the Update Site URL as: https://updates.jenkins.io/experimental/update-center.json Click on Submit, and then press the Check Now button. Go to Available tab. Search for Redis Fingerprint Storage Plugin and check the box along it. Click on Install without restart The plugin should now be installed on your system. Usage Once the plugin has been installed, you can configure the Redis server details by following the steps below: Select Manage Jenkins Select Configure System Scroll to the section Redis Fingerprint Storage Configuration and fill in the required details: Host - Enter hostname where Redis is running Port - Specify the port on which Redis is running SSL - Click if SSL is enabled Database - Redis supports integer indexed databases, which can be specified here. Connection Timeout - Set the connection timeout duration in milliseconds. Socked Timeout - Set the socket timeout duration in milliseconds. Credentials - Configure authentication using username and password to the Redis instance. Enabled - Check this to enable the plugin (Note: This is likely to be removed very soon, and will be enabled by default.) Use the Test Redis Connection to verify that the details are correct and Jenkins is able to connect to the Redis instance. Press the Save button. Now, all the fingerprints produced by this Jenkins instance should be saved in the configured Redis server! Future Work Some of the topics we aim to tackle in the next phases include extending the API, fingerprint cleanup, migrations (internal→external, external→internal, external→external), tracing, ORM, implementing the saveable listener, etc. Acknowledgements The Redis Fingerprint Storage plugin is built and maintained by the Google Summer of Code (GSoC) Team for External Fingerprint Storage for Jenkins. Special thanks to Oleg Nenashev, Andrey Falko, Mike Cirioli, Jesse Glick, and the entire Jenkins community for all the contribution to this project. Reaching Out Feel free to reach out to us for any questions, feedback, etc. on the project’s Gitter Channel or the Jenkins Developer Mailing list We use Jenkins Jira to track issues. Feel free to file issues under redis-fingerprint-storage-plugin component. Other Links Phase 1 demo Presentation slides Redis Fingerprint Storage Plugin Issue Tracker for Phase 1 jep:226[] Gitter Channel Project Page

Hello everyone ! This is one of the Jenkins project in GSoC 2020. We are working this new Machine Learning Plugin for this GSoC 2020. This is my story about the community bonding of GSoC 2020. I am happy to share my journey with you. Introducing Myself and my Fantastic 4 Mentors I am Loghi Perinpanayagam from University of Moratuwa. I was selected for GSoC 2020 for Machine Learning Plugin in Jenkins. I am glad to introduce my mentors to this project. I was assigned with four mentors who are really enthusiastic to help me on kicking off this summer of code. Student Loghi Perinpanayagam Mentors Bruno P. Kinoshita Ioannis Moutsatsos Marky Jackson Shivay Lamba How was my preparation last year ? I learned about this open source program in my second year. But atleast I tried last year on a different organization’s project that was related to Data Visualization Recommendation for Data Science. But the problem was I did not contribute as much as this year and was too late in the application process. As usual Machine learning related projects have a lot of competition compared to other projects. I prepared on learning Data visualization in Machine Learning and existing Models for the recommendation system. Finally I wrote a proposal with the SeqToSeq model without much knowledge on neural networks at that time. And I did not communicate much through the dedicated slack channel. That may be one of the reasons for the failure. But the main reason was my latency for GSoC 2019. How did I hurdle GSoC 2020 ? Since the time I realized how open source is needed and helpful for the community, I have been passionate about contributing to open source projects. At the instance, I finished my internship in Bangalore, India in 2019, I immediately focused on participating in GSoC. This is my last year (2020) as a student of my BSc Computer Science life, I wanted to get selected this year as a student. There was a guidance seminar organized by our department, I got to know that Jenkins had opened their project ideas. That was an extremely impressive beginning of my GSoC 2020 journey. I walked through all the draft and accepted projects in the Jenkins.io page. As I am already interested in Machine Learning and I am familiar with Java, I picked the most impressive idea for me that does not have an initial repo. That means I wanted to use my knowledge to think and research a lot with this project. But I had to contribute and want to know about the infrastructure of Jenkins codebase. Because that makes the selection panel easy to pick up the student for the project. Then I repeatedly searched to contribute to Jenkins. I found issues that were easy for me to work from the git plugin and git client plugin. I started to contribute some test issues on git plugin and git client plugin. After I got a clear knowledge on how a plugin works in Jenkins, I started working on the POC with the hint provided in the project idea page. Actually, that was fun to code. Mentors have helped many students during the application process. I was able to do a working POC that had a minimum capability to do the task of the project. Finally mentors opened for proposal submission. I hurried to prepare a draft proposal. After I got reviews from mentors, I started to improve the proposal. At the end of the proposal submission, I was able to deliver a good proposal for this project. As I was curious about this plugin, I dug into more on how to integrate Jupyter notebook with this plugin. I published an medium article as a result of my research during the acceptance waiting period. Results released The result was going to be announced on 4th May, I believed in my project proposal and POC and I got selected for this GSoC 2020. Whoa ! That was a goosebumping moment in my entire life. The feeling was like Something I achieved. As a result of my hard work, I deserved that. For example, I spent 7 days continuously making the POC work without any collision between maven artifacts. Community Bonding After the release of results, I was preparing myself for the community bonding. There are lots of interactions happening between me and mentors than before.I had to update my project page and my profile in Jenkins.io. We had our first meeting with lots of excitement and love on 10th of May. Mentors and I introduced ourselves even though we know each other. We discussed the high level view of GSoC and I asked some questions that I had in my mind. As my plugin was a new repository, most of the discussion was related to the repository and its name. I had to find a name for the new plugin. We had regular conversations about the blogpost and presentations at the end. In the second meeting, We discussed the process for hosting a new plugin in Jenkins, tracking issues with JIRA, blog posts and high level road map for the project. And I suggested some interesting plugin names but they were not matching to the goal of the project, mentors told me to try other names which perfectly describe the project. I was advised to read all the research guidelines and plugin naming conventions. We discussed how code reviews will be done and source code management through the git. After this meeting, our meeting has shifted to the official Jenkins Zoom account. Our third meeting was quite serious about our project planning. I had been preparing my design document for the project with the help of mentors before the meeting day. Hence I got lots of reviews and useful examples for my future work on phase 1. At this point, we decided with the plugin name Machine Learning Plugin which was accepted by all mentors and I created the repo and requested a Jira ticket for the plugin hosting request. We were planning to remind the Jira ticket within the next 3 days. Mentors want me to make sure I updated the Jenkins GSoC page before the community period ends. Lots of discussion carried about the design document that I had been preparing last week before the meeting. Some important points from the meeting notes follows : Define features in the design document Diagrams for the operations How plugin works in distributed environment Code editor library Requirements for the first Plugin release Blog post draft document ToDo works for me for next week Therefore, I had to work hard after this meeting, this made me involved in the project more. I have to put my huge effort to make this opportunity golden. Our team has the willingness to complete this project and will definitely help the Data Science community with this plugin. Kudos to my team for the amazing work so far!!! This was my entire journey until now. Hope you enjoyed it and hope you learned the mistakes I made last year and corrected in this summer. Thanks for reading, and Stay tuned I will be uploading blog posts for those of you interested. Resources and Links Project page Design document

Google Summer of Code (GSoC) is as program where students are paid a stipend by Google to work on a free open source project. Students work on the project full-time for four months (May to August). Mentors are actively involved with students starting at the end of February when students start to work on and submit their applications. (see the timeline) We are looking for project ideas and mentors to participate in GSoC 2020. GSoC project ideas are coding projects that university or college students can accomplish in about four months. The coding projects can be new features, plugins, test frameworks, infrastructure, etc. Anyone can submit a project idea, but of course we like it even better if you offer to mentor your project idea. We accept new project ideas at any time, BUT we need a series of ideas READY before February 5th, 2020 at 7pm UTC, which is the deadline for the Jenkins organization to apply to the GSoC program. So send us your project ideas before the beginning of February so they can get a proper review by the GSoC committee and by the community. How to submit a project idea For 2020, we have simplified the process. Simply create a pull-request with your idea in a.adoc file in the idea folder. It is no longer necessary to submit a Google Doc, but it will still work if you want to do that. See the instructions on submitting ideas which include an.adoc template and some examples. Current list of ideas We currently have a list of project ideas for students to browse, copied from last year. Note that this list is subject to change. What does mentoring involve? Potential mentors are invited to read the information for mentors. Note that being a GSoC mentor does not require expert knowledge of Jenkins. Mentors do not work alone. We make sure that every project has at least two mentors. GSoC org admins will help to find technical advisers, so you can study together with your students. Mentoring takes about 5 to 8 hours of work per week (more at the start, less at the end). Mentors provide guidance, coaching, and sometimes a bit of cheerleading. They review student proposals, pull-requests and the students presentations at the evaluation phase. They fill in the Google provided evaluation report form at the end of coding periods. What do you get in exchange? In return of mentoring, a student works on your project full time for four months. Think about the projects that you’ve always wanted to do but never had the time…​ Having a mentoring opportunity also means that you get to improve your management and people skills. As well, up to two mentors per organization are eligible to participate in the Google Mentor Summit taking place each year. The Jenkins Org Admins try to send different mentors each year. It is also possible to win an additional seat at the summit in the "last minute draw" (Google draws mentors at random to fill the cancellations and empty seats). See this post from one of the 2019 mentors on the kind of experience this was. GSoC is a pretty good return on the investment! For any question, you can find the GSoC Org Admins, mentors and participants on the GSoC SIG Gitter chat.