Recent Blog Posts

Abstract This project implemented two new credential bindings to perform authenticated operations using command line git in Jenkins pipeline and freestyle jobs. The two credential bindings are gitSshPrivateKey and gitUsernamePassword. Implementation Type Feature Location The gitUsernamePassword binding is implemented in Jenkins git plugin v4.8.0. The gitSshPrivateKey binding is implemented in a pull request to the Jenkins git plugin Dependencies Credentials Binding Plugin - It is used to bind Git specific environment variables with shell scripts/commands which perform git authentication on behalf of the user, without their interaction with the command-line. Bouncy Castle API Plugin - Provides an API to do common tasks like PEM/PKCS#8 Encoding/Decoding and ensuring its stability among Bouncy Castle API versions. SSH Server Plugin - Provides an API to perform tasks like OpenSSH private key encoding and decoding. Phase 1: Git Username Password Binding (gitUsernamePassword) Deliverables Support git authentication over the HTTP protocol Use the GIT_ASKPASS environment variable to provide user credentials to command line git Support different OS environments : CentOS 7, CentOS 8, Debian 9, Debian 10, FreeBSD 12, OpenBSD 6.9, openSUSE 15.2, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 21.04, and Windows 10. Processors : amd64, arm32, arm64, and s390x. Authentication support for command line git only, not JGit or JGit Apache. Check for specific git versions Setting git specific environment variables based on OS type Automated test coverage more than 90% Resources Pull Requests Add Git Credentials binding for Username and Password Check the least command line git version required Git username password binding doc update in git-plugin gitUsernamePassword binding explanation Webinar slides Git username password binding released blog post Phase 1 demo and presentation: Phase 2: Git SSH Private Key Binding (gitSshPrivateKey) Deliverables To support git authentication over the SSH protocol Supports: Private Key Formats OpenSSH PEM PKCS#8 Encryption algorithms RSA DSA ECDSA ED25519 OS environments : CentOS 7, CentOS 8, Debian 9, Debian 10, FreeBSD 12, OpenBSD 6.9, openSUSE 15.3, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 21.04, and Windows 10. Processors : amd64, arm32, arm64, and s390x. Authentication support for command line git only, not JGit or JGit Apache. Use git specific environment variables depending upon the minimum git version GIT_SSH_COMMAND - If the version is greater than 2.3, provides ssh command including the necessary options. SSH_ASKPASS - If the version is less than 2.3, an executable script is attached to the variable. Setting variables based on the OS type Resources Pull Requests Add Git Credentials binding for SSH Private Key Last GSOC-2021 noted commit Scope change of getSSHExecutable method gitSshPrivateKey binding explanation Webinar Slides Final phase demo and presentation Achievements The git credential bindings which are available through the git plugin automate the git authentication process for a user effortlessly The gitUsernamePassword and gitSshPrivateKey binding provides git authentication support for Pipeline and Freestyle Project users in various OS environments on different processors The gitUsernamePassword binding has been released and is readily available from git plugin v4.8.0 and above The gitSshPrivateKey binding provides support for OpenSSH format which is default for OpenSSH v7.8 and above Future Work SSH private key binding pull request merge and release Unexpected complications from Jenkins class loader required extra effort and investigation, including an experiment shading a dependency into the git plugin We intentionally chose to avoid the complication and risk of shading the dependency If the SSH library use requires shading, then we may need to use maven modules in the git plugin

Google Summer of Code 2021 is implementing git credentials binding for sh, bat, and powershell . Git credentials binding is one of the most requested features for Jenkins Pipeline (see jira:JENKINS-28335[]). The project involves extending the Credentials Binding Plugin to create custom bindings for two types of credentials essential to establish a remote connection with a git repository Username/Password SSH Private Key Why use git credentials binding? Many operations in a Jenkins Pipeline or Freestyle job can benefit from authenticated access to git repositories. Authenticated access to a git repository allows a Jenkins job to apply a tag and push the tag merge a commit and push the merge update submodules from private repositories retrieve large files with git LFS The git credentials username / password binding included in git plugin 4.8.0 allows Pipeline and Freestyle jobs to use command line git from sh, bat, and powershell for authenticated access to git repositories. How to use git credentials binding? The binding is accessible using the withCredentials Pipeline step. It requires two parameters: credentialsId Reference id provided by creating a Username/Password type credential in the Jenkins configuration. To understand how to configure credentials in a Jenkins environment: Using Credentials gitToolName Name of the git installation in the machine running the Jenkins instance (Check Global Tool Configuration section in Jenkins UI) Note: In case a user is not aware of the git tool installation of the particular machine, the default git installation will be chosen. Examples The withCredentials wrapper allows declarative and scripted Pipeline jobs to perform authenticated command line git operations with sh , bat , and powershell tasks. Shell example withCredentials([gitUsernamePassword(credentialsId: 'my-credentials-id', gitToolName: 'git-tool')]) { sh 'git fetch --all' } Batch example withCredentials([gitUsernamePassword(credentialsId: 'my-credentials-id', gitToolName: 'git-tool')]) { bat 'git submodule update --init --recursive' } Powershell example withCredentials([gitUsernamePassword(credentialsId: 'my-credentials-id', gitToolName: 'git-tool')]) { powershell 'git push' } The Pipeline Syntax Snippet Generator is a good way to explore the syntax of the withCredentials step and the git username / password credentials binding. Limitations The git credentials username / password binding has been tested on command line git versions 1.8.3 through 2.32.0. It has been tested on CentOS 7, CentOS 8, Debian 9, Debian 10, FreeBSD 12, OpenBSD 6.9, openSUSE 15.2, Ubuntu 18.04, Ubuntu 20.04, Ubuntu 21.04, and Windows 10. Processor testing has included amd64, arm32, arm64, and s390x. The binding does not support private key credentials. The binding is not supported on command line git versions prior to 1.8.3. What’s next? Private key credentials support is coming soon.

Since the beginning of the project, the core value which drove its progress was "To enhance the user experience for running Jenkins jobs by reducing the overall execution time". To achieve this goal, we laid out a path: Compare the two existing git implementations i.e CliGitAPIImpl and JGitAPIImpl using performance benchmarking Use the results to create a feature which would improve the overall performance of git plugin Also, fix existing user reported performance issues Let’s take a journey to understand how we’ve built the new features. If you’d like to skip the journey part, you can directly go to the [major performance improvements] section and the [minor performance section] to see what we’ve done! Journey to release The project started with deciding to choose a git operation and then trying to compare the performance of that operation by using command line git and then with JGit. Stage 1: Benchmark results with git fetch The performance of git fetch (average execution time/op) is strongly correlated to the size of a repository There exists an inflection point on the scale of repository size after which the nature of JGit performance changes (it starts to degrade) After running multiple benchmarks, it is safe to say that for a large sized repository command line git would be a better choice of implementation. We can use this insight to implement a feature which avoids JGit with large repositories. Stage 2: Comparing platforms The project was also concerned that there might be important differences between operating systems. For example, what if command line Git for Windows performed very differently than command line Git on Linux or FreeBSD? Benchmarks were run to compare fetch performance on several platforms. Running git fetch operation for a 400 MiB sized repository on: AMD64 Microsoft Winders AMD64 FreeBSD IBM PowerPC 64 LE Ubuntu 18 IBM System 390 Ubuntu 18 The result of running this experiment is given below: The difference in performance between git and JGit remains constant across all platforms. Benchmark results on one platform are applicable to all platforms. Stage 3: Performance of git fetch and repository structure The area of the circle enclosing each parameter signifies the strength of the positive correlation between the performance of a git fetch operation and that parameter. From the diagram: Size of the aggregated objects is the dominant player in determining the execution time for a git fetch Number of branches and Number of tags play a similar role but are strongly overshadowed by size of repository Number of commits has a negligible effect on the performance of running git fetch After running these experiments from Stage-1 to Stage-3, we developed a solution called the GitToolChooser which is explained in the next stage Stage 4: Faster checkout with Git tool chooser This feature takes the responsibility of choosing the optimal implementation from the user and hands it to the plugin. It takes the decision of recommending an implementation on the basis of the size of the repository. Here is how it works. The image above depicts the performance enhancements we have performed over the course of the GSoC project. These improvements have enabled the checkout step to be finished within half of what it used to take earlier in some cases. Let’s talk about performance improvements in two parts. Major performance improvements Building Tensorflow (~800 MiB) using a Jenkins pipeline, there is over 50% reduction in overall time spent in completing a job! The result is consistent multiple platforms. The reason for such a decrease is the fact that JGit degrades in performance when we are talking about large sized repositories. Since the GitToolChooser is aware of this fact, it chooses to recommend command line git instead which saves the user some time. Minor performance improvements Note: Enable JGit before using the new performance features to let GitToolChooser work with more options → Here’s how Building the git plugin (~ 20 MiB) using a Jenkins pipeline, there is a drop of a second across all platforms when performance enhancement is enabled. Also, eliminating a redundant fetch reduces unnecessary load on git servers. The reason for this change is the fact that JGit performs better than command line git for small sized repositories (<50MiB) as an already warmed up JVM favors the native Java implementation. Releases Git Plugin 4.4.0 Add GitToolChooser Remove redundant fetch Git Client Plugin 3.4.0 Add support to communicate compatibility of JGit with certain additional SCM behaviors The road ahead Support from other branch source plugins Plugins like the GitHub Branch Source Plugin or GitLab Branch Source Plugin need to extend an extension point provided by the git plugin to facilitate the exchange of information related to size of a remote repository hosted by the particular git provider JENKINS-63519 : GitToolChooser predicts the wrong implementation Addition of this feature to GitSCMSource Detection of lock related delays accessing the cache directories present on the controller This issue was reported by the plugin maintainer Mark Waite, there is a need to reproduce the issue first and then find a possible solution. Reaching out Feel free to reach out to us for any questions or feedback on the project’s Gitter Channel or the Jenkins Developer Mailing list. Report an issue at Jenkins Jira. Useful Links Phase 1 Blog: https://www.jenkins.io/blog/2020/07/09/git-performance-improvement-phase1/ Phase 2 Blog: https://www.jenkins.io/blog/2020/07/29/git-performance-improvement-phase2/ Project Page: https://www.jenkins.io/projects/gsoc/2020/projects/git-plugin-performance/ Demonstration

The second phase of the Git Plugin Performance Improvement project has been great in terms of the progress we have achieved in implementing performance improvement insights derived from the phase one JMH micro-benchmark experiments. What we’ve learned so far in this project is that a git fetch is highly correlated to the size of the remote repository. In order to make fetch improvements in this plugin, our task was to find the difference in performance for the two available git implementations in the Git Plugin, git and JGit. Our major finding was that git performs much better than JGit when it comes to a large sized repository (>100 MiB). Interestingly, JGit performs better than git when size of the repository is less than 100 MiB. In this phase, we were successful in coding this derived knowledge from the benchmarks into a new functionality called the GitToolChooser. GitToolChooser This class aims to add the functionality of recommending a git implementation on the basis of the size of a repository which has a strong correlation to the performance of git fetch (from performance Benchmarks). It utilizes two heuristics to calculate the size: Using cached .git dir from multibranch projects to estimate the size of a repository Providing an extension point which, upon implementation, can use REST APIs exposed by git service providers like Github, GitLab, etc to fetch the size of the remote repository. Will it optimize your Jenkins instance? That requires one of the following: you have a multibranch project in your Jenkins instance, the plugin can use that to recommend the optimal git implementation you have a branch Source Plugin installed in the Jenkins instance, the particular branch source plugin will recommend a git implementation using REST APIs provided by GitHub or GitLab respectively. The architecture and code for this class is at: PR-931 Note : This functionality is an upcoming feature in the subsequent Git Plugin release. JMH benchmarks in multiple environments The benchmarks were being executed on Linux and macOS machines frequently but there was a need to check if the results gained from those benchmarks would hold true across more platforms to ensure that the solution (GitToolChooser) is generally platform-agnostic. To test this hypothesis, we performed an experiment: Running git fetch operation for a 400 MiB sized repository on: Windows FreeBSD 12 ppc64le s390x The result of running this experiment is given below: Observations: ppc64le and s390x are able to run the operation in almost half the time it takes for the Windows or FreeBSD 12 machine. This behavior may be attributed to the increased computational power of those machines. The difference in performance between git and JGit remains constant across all platforms which is a positive sign for the GitToolChooser as its recommendation would be consistent across multiple devices and operating systems. Release Plan 🚀 JENKINS-49757 - Avoid double fetch from Git checkout step This issue was fixed in phase one, avoids the second fetch in redundant cases. It will be shipped with some benchmarks on the change in performance due to the removal of the second fetch. PR-574 PR-904 GitToolChooser PR-931 This pull request is under review, will be shipped in one of the subsequent Git Plugin releases. Current Challenges with GitToolChooser Implement the extension point to support GitHub Branch Source Plugin, Gitlab Branch Source Plugin and Gitea Plugin. The current version of JGit doesn’t support LFS checkout and sparse checkout, need to make sure that the recommendation doesn’t break existing use cases. Future Work In phase three, we wish to: Release a new version of the Git and Git Client Plugin with the features developed during the project Continue to explore more areas for performance improvement Add a new git operation: git clone (Stretch Goal) Reaching Out Feel free to reach out to us for any questions or feedback on the project’s Gitter Channel or the Jenkins Developer Mailing list. Project Page Phase 1 Blog Post

Git Plugin Performance Improvement is a Google Summer of Code 2020 project. It aims to improve the performance of the git plugin, which provides fundamental git functionalities. Internally, the plugin provides these functionalities using two implementations: command line git and JGit (pure java implementation). CLI git is the default implementation for the plugin, a user can switch to JGit if needed The project is divided into two parallel stages: Stage 1 : Create benchmarks which evaluate the execution time of a git operation provided by CLI git and JGit using JMH, a micro benchmarking test harness. Stage 2 : Implement the insights gained from the analysis into the plugin to improve the overall performance of the plugin. The project also aims to fix any existing performance bottlenecks within the plugin as well. Benchmarks The benchmarks are written using JMH. It was introduced in a GSoC 2019 project to Jenkins. JMH is provided within the plugin through the Jenkins Unit Test Harness POM dependency. The JMH benchmarks are created and run within the git client plugin During phase-1, we have created benchmarks for two operations: "git fetch" and "git ls-remote" Results and Analysis The benchmark analysis for git fetch: Git fetch results The performance of git fetch (average execution time/op) is strongly correlated to the size of a repository There exists an inflection point on the scale of repository size after which the nature of JGit performance changes (it starts to degrade) After running multiple benchmarks, it is safe to say that for a large sized repository CLI-git would be a better choice of implementation. We can use this insight to implement a feature which avoids JGit when it comes to large repositories. Please refer to PR-521 for an elaborate explanation on these results Note: Repository size means du -h .git Fixing redundant fetch issue The git plugin performs two fetch operations instead of one while performing a fresh checkout of a remote git repository. To fix this issue, we had to safely remove the second fetch keeping multiple use-cases in mind. The fix itself was not difficult to code, but to do that safely without breaking any existing use-case was a challenging task. Further Plan After consolidating a benchmarking strategy during Phase 1, the next steps will be: Provide functionality to the git plugin, which enables it to estimate the size of the repository without cloning it. Broaden the scope of benchmarking strategy Consider parameters like number of branches, references and commit history to find a relation with the performance of a git operation The git plugin depends on other plugins like Credentials which might require benchmarking the plugin itself and the effects of these external dependencies on the plugin’s performance Focus on other use-cases of the plugin For phase-1, I focused on the checkout step and the operations involved with it For the next phase, the focus will shift to other areas like Multibranch pipelines or Organisation Folders How can you help? If you have reached this far of the blog, you might be interested in the project. To help, you can Review the benchmarks in the benchmarks module Analyse the benchmarks results available on ci.jenkins.io [soon] Come visit our Gitter channel: https://gitter.im/jenkinsci/git-plugin

One common pattern for automated releases I have seen and used relies on Git tags as the catalyst for a release process. The immutable nature of releases and the immutable nature of tags can definitely go hand in hand, but up until few months ago Jenkins Pipeline was not able to trigger effectively off of Git tags. In this post I want to briefly share how to use tags to drive behaviors in Jenkins Pipeline. Consider the following contrived Jenkinsfile, which contains the three basic stages of Build, Test, and Deploy: pipeline { agent any stages { stage('Build') { steps { sh 'make package' } } stage('Test') { steps { sh 'make check' } } stage('Deploy') { when { tag "release-*" } steps { echo 'Deploying only because this commit is tagged...' sh 'make deploy' } } } } Of particular note is the when condition on the "Deploy" stage which is applying the tag criteria. This means the stage would only execute when the Pipeline has been triggered from a tag in Git matching the release-* Ant-style wildcard. In practice, this means that all pull requests, and branch-based Pipeline Runs result in the stage being skipped: When I push a release-1.0 tag, the Pipeline will then be triggerd and run the "Deploy" stage: Out of the box, Pipelines won’t trigger off of the presence of tags, which means that a Multibranch Pipeline must have a configuration update to know that it must Discover Tags. Configuring From the configuration screen of a Multibranch Pipeline (or GitHub Organization Folder), Discovering tags can be enabled by adding the appropriate "Behavior" to the Branch Source configuration: With these changes, the Jenkinsfile in the tagged versions of my source repository can now drive distinct deployment behavior which is not otherwise enabled in the Pipeline.