Remoting over Apache Kafka 2.0: Built-in Kubernetes support

Published on 8/19/2019 by Long Nguyen

    I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. After a successful Phase 1, finally the 2.0 version of the plugin has been released. The 2.0 version provides seamless integration with Kubernetes environment.

    2.0 version features

    • Start a simple Apache Kafka server in Kubernetes.

    • Dynamically provision Remoting Kafka Agent in Kubernetes.

    • Helm chart to bootstrap the whole system in Kubernetes.

    Start a simple Apache Kafka server in Kubernetes

    Use of the plugin requires that users have a configured Apache Zookeeper and Apache Kafka server, which could be intimidating for people who just want to try out the plugin. Now, users can start a simple, single-node Apache Kafka server in Kubernetes environment with just one button click.

    Apache Kafka provisioning in Kubernetes UI

    On the Global Configuration page, users can input Kubernetes server information and credentials. When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Apache Zookeeper and Apache Kafka YAML specification files from resources. After downloading images and creating containers, it will automatically update Apache Zookeeper and Apache Kafka URLs into respective fields.

    Dynamically provision Remoting Kafka Agent in Kubernetes

    With previous version, users have to manually add/remove nodes so it is hard to scale builds quickly. Kubernetes plugin allows us to dynamically provision agents in Kubernetes but it is designed for JNLP agent. With this new version, Remoting Kafka agent can also be provisioned automatically in Kubernetes environment.

    Remoting Kafka Cloud UI

    Users can find the new feature in Cloud section in /configure. Here users could input Kubernetes connection parameters and desired Remoting Kafka agent properties including labels. When new build with matching labels gets started and there are no free nodes, Cloud will automatically provision Remoting Kafka agent pod in Kubernetes to run the build.

    Remoting Kafka Agent get provisioned

    Helm Chart

    Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. You can follow the instruction here to install a demo ready-to-use Helm release. Your kubectl get all should look like this:

    NAME                                READY   STATUS    RESTARTS   AGE
pod/demo-jenkins-64dbd87987-bmndf   1/1     Running   0          2m21s
pod/demo-kafka-0                    1/1     Running   0          2m21s
pod/demo-zookeeper-0                1/1     Running   0          2m21s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/demo-jenkins              NodePort    10.108.238.56   <none>        8080:30386/TCP               2m21s
service/demo-jenkins-agent        ClusterIP   10.98.85.184    <none>        50000/TCP                    2m21s
service/demo-kafka                ClusterIP   10.109.231.58   <none>        9092/TCP                     2m21s
service/demo-kafka-headless       ClusterIP   None            <none>        9092/TCP                     2m21s
service/demo-zookeeper            ClusterIP   10.103.2.231    <none>        2181/TCP                     2m21s
service/demo-zookeeper-headless   ClusterIP   None            <none>        2181/TCP,3888/TCP,2888/TCP   2m21s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-jenkins   1/1     1            1           2m21s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-jenkins-64dbd87987   1         1         1       2m21s

NAME                              READY   AGE
statefulset.apps/demo-kafka       1/1     2m21s
statefulset.apps/demo-zookeeper   1/1     2m21s

    How to Contribute

    You are welcome to try out the plugin and integrate it into your current setup. If you find out any bug or if you would like to request new feature, you can create ticket at JIRA. If you would like to contribute code directly, you can create pull requests in the GitHub page below.

    About the Author
    Long Nguyen
    		Long Nguyen

    Long is a Software Engineering student at FPT University, Vietnam. He started to contribute for Jenkins from Google Summer of Code 2019 for the project Remoting over Apache Kafka with Kubernetes features