Good to see you all again ! This is my final blog post about coding phase 3 in Jenkins Machine Learning Plugin for GSoC 2020. Being at the end of GSoC 2020, we had to finish all the pending issues and testing before a stable release in the main repository. Throughout this program, there were lots of learning and hard work will make this plugin valuable to the Data Science and Jenkins community. Summary With combining all of the work in phase 1, 2 and 3, initial version of Machine learning plugin( 1.0 ) was successfully released in Jenkins plugin repository. An interesting feature which allows users to connect to their existing programming language kernels more than connecting to only IPython kernel was introduced in this phase. It can be selected in multiple steps with different kernel. Images and graphs produced by Jupyter notebooks will be saved in user preferred folder in the workspace that can be used for reporting/analytic purposes later. Hoping this blog summarizes the Machine Learning’s features and future contributions. Thank you for your interest and support !!! Main features of Machine Learning plugin Execute Jupyter notebooks directly Run different language scripts using multiple build steps Convert Jupyter Notebooks to Python Configure Jupyter kernels( IPython, IRKernel, IJulia etc) properties Support to execute Notebooks/scripts on Agent Extract graph/map/images from the code Each build step can be associated with a machine learning task Support for Windows and Linux Future improvements Improving performance of the plugin Try to implement jira:JENKINS-63377[] Support parameterized definitions in Notebooks jira:JENKINS-63478[] Increasing testing code coverage Multiple language kernel support If there are existing kernels in the system, user will be able to configure in the global configurations in order to apply in the builder/step configuration. Some popular interactive kernels IPython for python IRKernel for R IJulia for Julia IJavascript for javascript More kernels and installation guides are found here. https://github.com/jupyter/jupyter/wiki/Jupyter-kernels Dump images and graphs Text output will be displayed in the console log. At the same time images/graphs/heat maps and HTMLs will be saved in the workspace. An action is shown in the left panel to display images in realtime. Due to the Content Security Policy of jenkins, some HTMLs which contain harmful javascript may not render in jenkins UI. Fixed bugs There were more bugs identified and fixed with many interactive testings. Setting the working directory of kernels was a big issue while getting datasets/files by script. Zeppelin process launcher was bypassed to fix this issue. Patch version released A major bug which was created while setting the process working directory had patched in the v1.0.1. The latest release is more stable now. Acknowledgement Machine Learning plugin had been developed under GSoC 2020 program. A huge thanks to Bruno P. Kinoshita, Marky Jackson, Shivay Lamba, Ioannis Moutsatsos and Org admins for this wonderful experience. I would be grateful for contributing this plugin continuously and more in Jenkins. Resources Community Bonding blog post Phase 1 blog post Phase 2 blog post Github Project page Plugin page Design document

Welcome back folks! This blog post is about my coding phase 2 in Jenkins Machine Learning Plugin for this GSoC 2020. After successfully passing the evaluation and demo in the phase 1, our team went ahead for facing the challenges in phase 2. Summary This phase of coding was well spent by documentation and by fixing many bugs. As the main feature of connecting to an IPython Kernel is done in phase 1, we were able to focus on fixing minor/major bugs and documenting for the users. According to the JENKINS-62927 issue, a Docker agent was built to facilitate users without concerning plugin dependencies in python. In the act of deprecation of Python 2, we ported our plugin to support Python 3. We have tested our plugin in Conda, venv and Windows environments. Machine learning plugin has successfully passed the end to end test. A feature for a code editor is needed for further discussion/analysis as we have done a simple editor that may be useful in other ways in the future. PR#35 Main features of Machine Learning plugin Run Jupyter notebook, (Zeppelin) JSON and Python files Run Python code directly Convert Jupyter Notebooks to Python and JSON Configure IPython kernel properties Support to execute Notebooks/Python on Agent Support for Windows and Linux Upcoming features Extract graph/map/images from the code Save artifacts according to the step name Generate reports for corresponding build Future improvements Usage of JupyterRestClient Support for multiple language kernels Note : There is no commitment on future improvements during GSoC period Docker agent The following Dockerfile can be used to build the Docker container as an agent for the Machine Learning plugin. This docker agent can be used to run notebooks or python scripts. Dockerfile FROM jenkins/agent:latest MAINTAINER Loghi USER root RUN apt update && apt install --no-install-recommends python3 -y \ python3-pip \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt /requirements.txt RUN pip3 install --upgrade pip setuptools && \ pip3 install --no-cache-dir -r /requirements.txt && \ ln -sf /usr/bin/python3 /usr/bin/python && \ ln -sf /usr/bin/pip3 /usr/bin/pip USER jenkins Ported to Python 3 As discussed in the previous meeting, we concluded that the plugin should support Python 3 as Python 2.7+ has been deprecated since the beginning of 2020. Pull request for docker agent should be also ported to Python 3 support. Jupyter Rest Client API The Jupyter Notebook server API seemed to be promising that it can be also used to run notebooks and codes. There were 3 api implementations that were merged in the master. But we had to focus on what was proposed in the design document and had to finish all must-have issues/works. Jupyter REST client was left for future implementation. It is also a good start to contribute to the plugin from the community. Fixed bugs for running in agent There were a few bugs related to the file path of notebooks while building a job. The major problem was caused by the python dependencies needed to connect to a IPython kernel. All issues/bugs were fixed before the timeline given. R support as a future improvement This is what we tried to give a glimpse of knowledge that this plugin can be extended for multi language support in the future. There was a conclusion that the kernel should be selected dynamically using extension of the script file(like eval_model.rb or train_model.r), instead of scripting the same code for each kernel. Documentation and End to End testing A well explained documentation was published in the repository. A guided tutorial to run a notebook checked out from a git repo in an agent was included in the docs page. Mentors helped to test our plugin in both Linux and Windows. Code editor with rebuild feature Code editor was filtered as a nice to have feature in the design document. After grabbing the idea of Jenkinsfile replay editor, I could do the same for the code. At the same time, when we are getting the source code from git, it is not an elegant way of editing code in the original code. After the discussion, we had to leave the PR open that may have use cases in the future if needed. Jenkins LTS update The plugin has been updated to support Jenkins LTS 2.204.1 as 2.164.3 had some problems with installing pipeline supported API/plugin Installation for experimental version Enable the experimental update center Search for Machine Learning Plugin and check the box along it. Click on Install without restart The plugin should now be installed on your system. Resources Community Bonding blog post Phase 1 blog post Github Project page Design document

Welcome back ! This blog post is briefing my coding phase 1 in Jenkins Machine Learning Plugin for this GSoC 2020. After a fresh introduction of community bonding, On June 1st, coding of GSoC had started officially with phase 1. At this point, every GSoC student should be expected to have a rigid plan with their entire project. With the guidance of mentors I was able to complete a design document and timeline which can be slightly adjustable during the coding. The coding phase was more about coding and discussion. Quick review Pull Requests 21 Jira Issues 11 Major Tasks 3 Completed 3 In progress 0 Week 1 I have to ensure that I have a solid architecture for implementing the core of this plugin such that perhaps I or future community will be able to develop R and Julia kernels for this plugin. Factory method design patterns are suitable when users need different types of products ( Python, R and Julia) without knowing much about the internal infrastructure ( Manager of these interpreters ). All the base classes were implemented this week. Design the Kernel connectors Initiate the interpreter Close the connection Add simple tests Update pom.xml More than these changes, repo was updated with pull request template and licence header. Readme was extended a little at the end of the week. Issues and Challenges Git rebase and squash Tests invokes ipython client in the server failed during the CI build Week 2 With the help of a design document, I had a plan to do the configurations globally and using the Abstract Folder property I could save the configuration and retrieve for the job configuartion. I used to reference some other well developed plugin for the structure of code. That helped me a lot while I was coding. Our first official contributor has popped out his pull request. Form validations and helper html will be a great help in the user point of view as well as developers. A minor bug was fixed with the guidance of mentors by writing tests with ‘Jenkins WebClient`. Until the end of the week, builder class of the plugin has been implemented with lots of research and discussion. Finally, Test connection was added to the global configuration page to start the connection and test it. A single issue that blocked me using py4j authentication about zeppelin-python was reported in Jira. Server Configuration Issues and challenges Backend depends on Apache zeppelin-python API to connect IPython Find relevant extension points to extend the plugin Week 3 Earlier in this week, we were trying to merge our IPython builder PR without any memory leaks or bugs that will cause the system to be devastating while running this plugin. Later, this whole week I was implementing a file parser that could copy the necessary files and had the ability to accomplish the file conversion. Supported file types Python (.py) JSON (Zeppelin notebooks format) IPython builder was able to run Jupyter Notebooks and Zeppelin formatted JSON files at the end of the 3rd week. Minor issues were fixed in the code. We used ANSI color plugin to fix the abnormal view of error messages produced by the ipython kernel. Copying and converting Jupyter Notebook Issues and Challenges Python error messages could not be displayed in rich format If a job is running at user level, but if the python code access file/file path which is not authorized to the user, it returns a permission denied message. While running on agent, notebook has to be written/copied to agent workspace Artifacts should be maintained/reachable from controller after build. Week 4 As all the major tasks has done, the demo preparation and plan for a experimental release was carried during the last week. There were lots of research on how to connect to a existing kernel in remote. Demo and presentation were prepared along the week. Issues and Challenges Releasing the first version was bit late Knowledge transfer How to debug the code through IntelliJ Edit configuration → Add new Configuration → Maven Command line → type hpi:run Click the debug icon on the toolbar or go to Run menu then Debug How to setup to test the plugin Setup JDK 8 and Maven 3.5.* Create a directory $ mkdir machine-learning-plugin Create a virtual environment $ virtualenv venv Activate your virtual environment $ source venv/bin/activate Run $ which python to ensure your python path $ git clone https://github.com/jenkinsci/machine-learning-plugin.git Run $ mvn clean install from the machine-learning-plugin directory Run $ mvn hpi:run to start Jenkins with the plugin Set up the builder with localhost and other parameters Create a job Write python code like print(“plugin works”) Build the job Issues and bugs JENKINS-62528 Issues on Jenkins build in the plugin repository JENKINS-62621 Global configuration for IPython servers JENKINS-62649 Implementation of IPython Builder JENKINS-62711 File parser to copy source files to workspace JENKINS-62733 Python errors are not displayed properly in console log JENKINS-62735 Send/Receive necessary files from controller/agent to agent/controller JENKINS-62593 Improve the documentation JENKINS-62742 Increase Test coverage Resources and Links Github Project page Design document

Hello everyone ! This is one of the Jenkins project in GSoC 2020. We are working this new Machine Learning Plugin for this GSoC 2020. This is my story about the community bonding of GSoC 2020. I am happy to share my journey with you. Introducing Myself and my Fantastic 4 Mentors I am Loghi Perinpanayagam from University of Moratuwa. I was selected for GSoC 2020 for Machine Learning Plugin in Jenkins. I am glad to introduce my mentors to this project. I was assigned with four mentors who are really enthusiastic to help me on kicking off this summer of code. Student Loghi Perinpanayagam Mentors Bruno P. Kinoshita Ioannis Moutsatsos Marky Jackson Shivay Lamba How was my preparation last year ? I learned about this open source program in my second year. But atleast I tried last year on a different organization’s project that was related to Data Visualization Recommendation for Data Science. But the problem was I did not contribute as much as this year and was too late in the application process. As usual Machine learning related projects have a lot of competition compared to other projects. I prepared on learning Data visualization in Machine Learning and existing Models for the recommendation system. Finally I wrote a proposal with the SeqToSeq model without much knowledge on neural networks at that time. And I did not communicate much through the dedicated slack channel. That may be one of the reasons for the failure. But the main reason was my latency for GSoC 2019. How did I hurdle GSoC 2020 ? Since the time I realized how open source is needed and helpful for the community, I have been passionate about contributing to open source projects. At the instance, I finished my internship in Bangalore, India in 2019, I immediately focused on participating in GSoC. This is my last year (2020) as a student of my BSc Computer Science life, I wanted to get selected this year as a student. There was a guidance seminar organized by our department, I got to know that Jenkins had opened their project ideas. That was an extremely impressive beginning of my GSoC 2020 journey. I walked through all the draft and accepted projects in the Jenkins.io page. As I am already interested in Machine Learning and I am familiar with Java, I picked the most impressive idea for me that does not have an initial repo. That means I wanted to use my knowledge to think and research a lot with this project. But I had to contribute and want to know about the infrastructure of Jenkins codebase. Because that makes the selection panel easy to pick up the student for the project. Then I repeatedly searched to contribute to Jenkins. I found issues that were easy for me to work from the git plugin and git client plugin. I started to contribute some test issues on git plugin and git client plugin. After I got a clear knowledge on how a plugin works in Jenkins, I started working on the POC with the hint provided in the project idea page. Actually, that was fun to code. Mentors have helped many students during the application process. I was able to do a working POC that had a minimum capability to do the task of the project. Finally mentors opened for proposal submission. I hurried to prepare a draft proposal. After I got reviews from mentors, I started to improve the proposal. At the end of the proposal submission, I was able to deliver a good proposal for this project. As I was curious about this plugin, I dug into more on how to integrate Jupyter notebook with this plugin. I published an medium article as a result of my research during the acceptance waiting period. Results released The result was going to be announced on 4th May, I believed in my project proposal and POC and I got selected for this GSoC 2020. Whoa ! That was a goosebumping moment in my entire life. The feeling was like Something I achieved. As a result of my hard work, I deserved that. For example, I spent 7 days continuously making the POC work without any collision between maven artifacts. Community Bonding After the release of results, I was preparing myself for the community bonding. There are lots of interactions happening between me and mentors than before.I had to update my project page and my profile in Jenkins.io. We had our first meeting with lots of excitement and love on 10th of May. Mentors and I introduced ourselves even though we know each other. We discussed the high level view of GSoC and I asked some questions that I had in my mind. As my plugin was a new repository, most of the discussion was related to the repository and its name. I had to find a name for the new plugin. We had regular conversations about the blogpost and presentations at the end. In the second meeting, We discussed the process for hosting a new plugin in Jenkins, tracking issues with JIRA, blog posts and high level road map for the project. And I suggested some interesting plugin names but they were not matching to the goal of the project, mentors told me to try other names which perfectly describe the project. I was advised to read all the research guidelines and plugin naming conventions. We discussed how code reviews will be done and source code management through the git. After this meeting, our meeting has shifted to the official Jenkins Zoom account. Our third meeting was quite serious about our project planning. I had been preparing my design document for the project with the help of mentors before the meeting day. Hence I got lots of reviews and useful examples for my future work on phase 1. At this point, we decided with the plugin name Machine Learning Plugin which was accepted by all mentors and I created the repo and requested a Jira ticket for the plugin hosting request. We were planning to remind the Jira ticket within the next 3 days. Mentors want me to make sure I updated the Jenkins GSoC page before the community period ends. Lots of discussion carried about the design document that I had been preparing last week before the meeting. Some important points from the meeting notes follows : Define features in the design document Diagrams for the operations How plugin works in distributed environment Code editor library Requirements for the first Plugin release Blog post draft document ToDo works for me for next week Therefore, I had to work hard after this meeting, this made me involved in the project more. I have to put my huge effort to make this opportunity golden. Our team has the willingness to complete this project and will definitely help the Data Science community with this plugin. Kudos to my team for the amazing work so far!!! This was my entire journey until now. Hope you enjoyed it and hope you learned the mistakes I made last year and corrected in this summer. Thanks for reading, and Stay tuned I will be uploading blog posts for those of you interested. Resources and Links Project page Design document