Recent Blog Posts

The Jenkins project relies heavily on its infrastructure. We use websites like www.jenkins.io and plugins.jenkins.io, ticketing systems like issues.jenkins.io, CI/CD infrastructure like ci.jenkins.io, and many other services. Just to provide some context about the Jenkins infrastructure scale, here are some stats from April 2020: Over 600 000 people visited www.jenkins.io Over 250 000 Jenkins servers regularly checked the Jenkins package server and the Jenkins update server Over 43 000 continuous integration jobs ran on ci.jenkins.io Over 950 plugins ran their continuous integration pipelines on ci.jenkins.io Country by country visitors to jenkins.io The Jenkins project, as an open source project, is built and maintained by its awesome community. Like in any organization, there are specific people who make sure that those services are always up and running. Everyone is welcome to participate. Infrastructure is no exception, we are always looking for new contributors to the infrastructure! While we can’t share publicly everything like secrets and certificates, we still try to be as transparent as possible so that everybody can understand and improve our infrastructure without having privileged access. What better way than using Git to manage infrastructure work? Who said GitOps? Since the creation of the Jenkins-infra organization on GitHub in March 2008, more than 650 people have contributed to over 80 git repositories. Those contributions make the Jenkins community what it is today. If you can’t find something there, it probably means that some help is welcomed. More recently, with help from Gavin Mogan, Tim Jacomb, and Alex Earl, big achievements have been possible on many fronts like automating Jenkins releases, refreshing plugins.jenkins.io, adding new agents to ci.jenkins.io, and maintaining our Kubernetes cluster. We thank them for their help and for the infrastructure progress they have enabled. Infrastructure at Scale Running infrastructure at the scale the Jenkins project does is expensive and sometimes quite challenging. We are fortunate enough to be supported by many leading companies that provide us their expertise, their products, and their support. Recently, Amazon Web Services donated $60 000 to run Jenkins infrastructure on the AWS cloud. We’re so grateful for their donation and for the flexibility it provides. We’re running Linux agents with AMD64 and ARM64 architectures on AWS. We’re using AWS cloud for our Windows agents. The generous infrastructure donation from Amazon Web Services has increased our continuous integration capacity and broadened our platform coverage. Our Sponsors Major sponsors of the Jenkins infrastructure include CloudBees, Oregon State University Open Source Lab, Continuous Delivery Foundation, Red Hat, Amazon Web Services, and GitHub. Additional sponsors of Jenkins infrastructure services and software include Atlassian, Datadog, Fastly, IBM. JFrog, Pagerduty, Rackspace, Sentry, Serverion, SpinUp, Tsinghua University, and XMission. Each of these organizations support the Jenkins project in their own way. We thank them for their contributions, their support and for their willingness to help the Jenkins community. https://www.jenkins.io/projects/infrastructure/

API keys and secrets are difficult to handle safely, and probably something you avoid thinking about. In this post I’ll show how the new AWS Secrets Manager Credentials Provider plugin allows you to marshal your secrets into one place, and use them securely from Jenkins. When CI/CD pipelines moved to the public cloud, credential management did not evolve with them. If you’re in this situation, you may have seen a number of tactical workarounds to keep Jenkins builds talking to the services they depend on. The workarounds range from bad (hardcoding plaintext secrets into Git) to merely painful (wrangling Hiera EYAML), but their common feature is that they tend to make copies of secrets beyond the reach of automation. This increases their attack surface, makes routine key rotation impractical, and makes remediation difficult after a breach. The good news is that there is a better way! AWS Secrets Manager is a comprehensive solution for secure secret storage. You define a secret just once for your whole AWS account, then you give your consumers permission to use the secrets. Secrets Manager lets you manage a secret entry (name and metadata) separately from its value, and it integrates with other AWS services that you already use: Secret entry management: Manual (Web console, AWS CLI) or with an infrastructure management tool ( Terraform, CloudFormation etc.) Secret value management: Manual (Web console, AWS CLI) or automatic (secret rotation Lambda function). Access control: AWS IAM policies (for both applications and human operators). Secret encryption: Amazon KMS automatically encrypts the secret value. Use either the account’s default KMS key, or a customer-managed KMS key. Auditing: AWS CloudTrail and CloudWatch Events. A couple of teams in my company started to use Secrets Manager from Jenkins jobs by calling the AWS CLI, but this remained a niche approach as it was quite unwieldy. There was clearly an appetite to integrate key developer apps with a centralised secrets store, but production-ready integrations were needed for wider adoption. So this year I created the AWS Secrets Manager Credentials Provider plugin for Jenkins, with help from friends in the Jenkins community, to do exactly that. This is how you set it up…​ Install the plugin from the Jenkins update center. Give Jenkins read-only access to Secrets Manager with an IAM policy. (Optional) Configure the plugin, either through the Global Configuration screen or Jenkins Configuration As Code. This is how you use it…​ Create your build secrets in AWS Secrets Manager. (You can start by uploading secrets via the AWS CLI. More sophisticated methods of secret creation are also available.) View the credentials in the Jenkins UI, to check that Jenkins can see them. Bind the credentials by ID in your Jenkins job. The provider supports the following standard Jenkins credential types: Secret Text Username With Password SSH User Private Key PKCS#12 Certificate And it has powerful advantages over quick-fix tactical solutions: Your Jenkins jobs consume the credentials with no knowledge of Secrets Manager, so they stay vendor-independent. The provider caches relevant Secrets Manager API calls, for a quicker and more reliable experience. The provider integrates with the ecosystem of existing Jenkins credential consumers, such as the Git and SSH Agent plugins. The provider records credential usage in the central Jenkins credentials tracking log. Jenkins can use multiple credentials providers concurrently, so you can incrementally migrate credentials to Secrets Manager while consuming other credentials from your existing providers. After the plugin’s first public release, developers at other companies adopted it too. It has had contributions so far from people at Elsevier, GoDaddy, and Northeastern University, as well as the fantastic Jenkins core team. We even got fan mail for our work! In enterprise security, "The important things are always simple. The simple things are always hard. The easy way is always mined." ( @thegrugq) It’s easy to buy a shiny ‘next generation' security appliance and drop it into your network. But it’s hard to embed the security fundamentals (like secrets management, OS patching, secure development) across your organisation. This Jenkins plugin is part of the effort [ 1 ] to take one of the persistent hard problems in security, and make it easier for everyone. 1. If you’re on Azure or you run most of your workload on Kubernetes, check out the Azure Credentials Plugin and the Kubernetes Credentials Provider Plugin.

This is a guest post by Aleksei Besogonov, Senior Software Developer at Amazon Web Services. Earlier this year, we published a case study on how Lyft has used Amazon EC2 Spot instances to save 75% on their continuous delivery infrastructure costs by simply changing four lines of code. Several other EC2 customers like Mozilla have also reduced costs of their continuous integration, deployment and testing pipelines by up to 90% on Spot instances. You can view the current savings on Spot instances over EC2 On-demand instances using the Spot Bid Advisor : AWS Spot instances are spare EC2 instances that you can bid on. While your Spot instances may be terminated when EC2’s spare capacity declines, you can automatically replenish these instances and maintain your target capacity using EC2 Spot fleets. As each instance type and Availability Zone provides an alternative capacity pool, you can select multiple such pools to launch the lowest priced instances currently available by launching a Spot fleet on the Amazon EC2 Spot Requests console or using the AWS CLI/SDK tools. In this walkthrough, we’ll show you how to configure Jenkins to automatically scale a fleet of Spot instances up or down depending on the number jobs to be completed. Request an Amazon EC2 Spot fleet To get started, login to Amazon EC2 console, and click on Spot Requests in the left hand navigation pane. Alternatively, you can directly login to Amazon EC2 Spot Requests console. Then click on the Request Spot Instances button at the top of the dashboard. In the Spot instance launch wizard, select the Request & Maintain option to request a Spot fleet that automatically provisions the most cost-effective EC2 Spot instances, and replenishes them if interrupted. Enter an initial target capacity, choose an AMI, and select multiple instance types to automatically provision the lowest priced instances available. On the next page, ensure that you have selected a key pair, complete the launch wizard, and note the Spot fleet request ID. Amazon EC2 Spot fleet automates finding the lowest priced instances for you, and enables your Jenkins cluster to maintain the required capacity; so, you don’t need any bidding algorithms to provision the optimal Spot instances over time. Configure Jenkins Install the Plugin From the Jenkins dashboard, select Manage Jenkins, and then click Manage Plugins. On the Available tab, search for and select the EC2 Fleet Jenkins Plugin. Then click the Install button. After the plugin installation is completed, select Manage Jenkins from the Jenkins dashboard, and click Configure System. In the Cloud section, select Amazon Spot Fleet to add a new Cloud. Configure AWS Credentials Next, we will configure the AWS and agent node credentials. Click the Add button next to AWS Credentials, select Jenkins, and enter your AWS Access Key, secret, and ID. Next, click the Add button in the Spot fleet launcher to configure your agents with an SSH key. Select Jenkins, and enter the username and private key (from the key pair you configured in your Spot fleet request) as shown below. Confirm that the AWS and SSH credentials you just added are selected. Then choose the region, and the Spot fleet request ID from the drop-down. You can also enter the maximum idle time before your cluster automatically scales down, and the maximum cluster size that it can scale up to. Submit Jobs and View Status After you have finished the previous step, you can view the EC2 Fleet Status in the left hand navigation pane on the Jenkins dashboard. Now, as you submit more jobs, Jenkins will automatically scale your Spot fleet to add more nodes. You can view these new nodes executing jobs under the Build Executor Status. After the jobs are done, if the nodes remain free for the specified idle time (configured in the previous step), then Jenkins releases the nodes, automatically scaling down your Spot fleet nodes. Build faster and cheaper If you have a story to share about your team or product, or have a question to ask, do leave a comment for us; we’d love to connect with you!