I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. After a successful Phase 1, finally the 2.0 version of the plugin has been released. The 2.0 version provides seamless integration with Kubernetes environment. 2.0 version features Start a simple Apache Kafka server in Kubernetes. Dynamically provision Remoting Kafka Agent in Kubernetes. Helm chart to bootstrap the whole system in Kubernetes. Start a simple Apache Kafka server in Kubernetes Use of the plugin requires that users have a configured Apache Zookeeper and Apache Kafka server, which could be intimidating for people who just want to try out the plugin. Now, users can start a simple, single-node Apache Kafka server in Kubernetes environment with just one button click. On the Global Configuration page, users can input Kubernetes server information and credentials. When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Apache Zookeeper and Apache Kafka YAML specification files from resources. After downloading images and creating containers, it will automatically update Apache Zookeeper and Apache Kafka URLs into respective fields. Dynamically provision Remoting Kafka Agent in Kubernetes With previous version, users have to manually add/remove nodes so it is hard to scale builds quickly. Kubernetes plugin allows us to dynamically provision agents in Kubernetes but it is designed for JNLP agent. With this new version, Remoting Kafka agent can also be provisioned automatically in Kubernetes environment. Users can find the new feature in Cloud section in /configure. Here users could input Kubernetes connection parameters and desired Remoting Kafka agent properties including labels. When new build with matching labels gets started and there are no free nodes, Cloud will automatically provision Remoting Kafka agent pod in Kubernetes to run the build. Helm Chart Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. You can follow the instruction here to install a demo ready-to-use Helm release. Your kubectl get all should look like this: NAME READY STATUS RESTARTS AGE pod/demo-jenkins-64dbd87987-bmndf 1/1 Running 0 2m21s pod/demo-kafka-0 1/1 Running 0 2m21s pod/demo-zookeeper-0 1/1 Running 0 2m21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/demo-jenkins NodePort 10.108.238.56 8080:30386/TCP 2m21s service/demo-jenkins-agent ClusterIP 10.98.85.184 50000/TCP 2m21s service/demo-kafka ClusterIP 10.109.231.58 9092/TCP 2m21s service/demo-kafka-headless ClusterIP None 9092/TCP 2m21s service/demo-zookeeper ClusterIP 10.103.2.231 2181/TCP 2m21s service/demo-zookeeper-headless ClusterIP None 2181/TCP,3888/TCP,2888/TCP 2m21s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo-jenkins 1/1 1 1 2m21s NAME DESIRED CURRENT READY AGE replicaset.apps/demo-jenkins-64dbd87987 1 1 1 2m21s NAME READY AGE statefulset.apps/demo-kafka 1/1 2m21s statefulset.apps/demo-zookeeper 1/1 2m21s How to Contribute You are welcome to try out the plugin and integrate it into your current setup. If you find out any bug or if you would like to request new feature, you can create ticket at JIRA. If you would like to contribute code directly, you can create pull requests in the GitHub page below. Links Phase 2 Demo Video Phase 2 Presentation Slides Phase 1 Blog Post Phase 1 Demo Video Phase 1 Presentation Slides Remoting over Apache Kafka plugin source code Project Page Gitter Channel

I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. This is the first time I have contributed for Jenkins and I am very excited to announce the features that have been done in Phase 1. Project Introduction Current version of Remoting over Apache Kafka plugin requires users to manually configure the entire system which includes Zookeeper, Kafka and remoting agents. It also doesn’t support dynamic agent provisioning so scalability is harder to achieve. My project aims to solve two problems: Out-of-the-box solution to provision Apache Kafka cluster. Dynamic agent provisioning in a Kubernetes cluster. Current State Kubernetes connector with credentials supported. Apache Kafka provisioning in Kubernetes feature is fully implemented. Helm chart is partially implemented. Apache Kafka provisioning in Kubernetes This feature is part of 2.0 version so it is not yet released officially. You can try out the feature by using the Experimental Update Center to update to 2.0.0-alpha version or building directly from master branch: git clone https://github.com/jenkinsci/remoting-kafka-plugin.git cd remoting-kafka-plugin/plugin mvn hpi:run On the Global Configuration page, users can input Kubernetes server information and credentials. Then they can start Apache Kafka with only one button click. When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Zookeeper and Kafka YAML specification files from resources. Helm Chart Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. As of now, the chart is still a Work in Progress because it is still waiting for Cloud API implementation in Phase 2. However, you can check out the demo chart with a single standalone Remoting Kafka Agent: git clone -b demo-helm-phase-1 https://github.com/longngn/remoting-kafka-plugin.git cd remoting-kafka-plugin K8S_NODE=./helm/jenkins-remoting-kafka/do.sh start The command do.sh start will do the following steps: Install the chart (with Jenkins and Kafka). Launch a Kafka computer on Jenkins controller by applying the following JCasC. jenkins: nodes: - permanent: name: "test" remoteFS: "/home/jenkins" launcher: kafka: {} Launch a single Remoting Kafka Agent pod. You can check the chart state by running kubectl, for example: $ kubectl get all -n demo-helm NAME READY STATUS RESTARTS AGE pod/demo-jenkins-998bcdfd4-tjmjs 2/2 Running 0 6m30s pod/demo-jenkins-remoting-kafka-agent 1/1 Running 0 4m10s pod/demo-kafka-0 1/1 Running 0 6m30s pod/demo-zookeeper-0 1/1 Running 0 6m30s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/demo-0-external NodePort 10.106.254.187 19092:31090/TCP 6m30s service/demo-jenkins NodePort 10.101.84.33 8080:31465/TCP 6m31s service/demo-jenkins-agent ClusterIP 10.97.169.65 50000/TCP 6m31s service/demo-kafka ClusterIP 10.106.248.10 9092/TCP 6m30s service/demo-kafka-headless ClusterIP None 9092/TCP 6m30s service/demo-zookeeper ClusterIP 10.109.222.63 2181/TCP 6m30s service/demo-zookeeper-headless ClusterIP None 2181/TCP,3888/TCP,2888/TCP 6m31s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo-jenkins 1/1 1 1 6m30s NAME DESIRED CURRENT READY AGE replicaset.apps/demo-jenkins-998bcdfd4 1 1 1 6m30s NAME READY AGE statefulset.apps/demo-kafka 1/1 6m30s statefulset.apps/demo-zookeeper 1/1 6m30s Next Phase Plan Implement Cloud API to provision Remoting Kafka Agent. ( JENKINS-57668) Integrate Cloud API implementation with Helm chart. ( JENKINS-58288) Unit tests and integration tests. Release version 2.0 and address feedbacks. ( JENKINS-58289) Links Phase 1 Demo Video Phase 1 Presentation Slides Remoting over Apache Kafka plugin Project Page Gitter Channel