Browser-testing with Sauce OnDemand and Pipeline
| This is a guest post by Liam Newman, Technical Evangelist at CloudBees. |
Testing web applications across multiple browsers on different platforms can be challenging even for smaller applications. With Jenkins and the Sauce OnDemand Plugin, you can wrangle that complexity by defining your Pipeline as Code.
Pipeline ♥ UI Testing, Too
I recently started looking for a way to do browser UI testing for an open-source JavaScript project to which I contribute. The project is targeted primarily at Node.js but we’re committed to maintaining browser-client compatibility as well. That means we should run tests on a matrix of browsers. Sauce Labs has an "open-sauce" program that provides free test instances to open-source projects. I decided to try using the Sauce OnDemand Plugin and Nightwatch.js to run Selenium tests on a sample project first, before trying a full-blown suite of tests.
Starting from Framework
I started off by following Sauce Labs' instructions on "Setting up Sauce Labs with Jenkins" as far as I could. I installed the JUnit and Sauce OnDemand plugins, created an account with Sauce Labs, and added my Sauce Labs credentials to Jenkins. From there I started to get a little lost. I’m new to Selenium and I had trouble understanding how to translate the instructions to my situation. I needed a working example that I could play with.
Happily, there’s a whole range of sample projects in
"saucelabs-sample-test-frameworks"
on GitHub, which show how to integrate Sauce Labs with various test frameworks, including Nightwatch.js.
I forked the Nightwatch.js sample to
bitwiseman/JS-Nightwatch.js
and set to writing my Jenkinsfile.
Between the sample and the Sauce Labs instructions,
I was able to write a pipeline that ran five tests on one browser via
Sauce Connect:
node {
stage "Build"
checkout scm
sh 'npm install' (1)
stage "Test"
sauce('f0a6b8ad-ce30-4cba-bf9a-95afbc470a8a') { (2)
sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) { (3)
sh './node_modules/.bin/nightwatch -e chrome --test tests/guineaPig.js || true' (4)
junit 'reports/**' (5)
step([$class: 'SauceOnDemandTestPublisher']) (6)
}
}
}| 1 | Install dependencies |
| 2 | Use my previously added sauce credentials |
| 3 | Start up the Sauce Connect tunnel to Sauce Labs |
| 4 | Run Nightwatch.js |
| 5 | Use JUnit to track results and show a trend graph |
| 6 | Link result details from Sauce Labs |
|
This pipeline expects to be run from a |
I ran this job a few times to get the JUnit report to show a trend graph.
This sample app generates the SauceOnDemandSessionID for each test, enabling the Jenkins Sauce OnDemand Plugin’s result publisher to link results to details Sauce Labs captured during the run.
Adding Platforms
Next I wanted to add a few more platforms to my matrix. This would require changing both the test framework configuration and the pipeline. I’d need to add new named combinations of platform, browser, and browser version (called "environments") to the Nightwatch.js configuration file, and modify the pipeline to run tests in those new environments.
This is a perfect example of the power of pipeline as code. If I were working with a separately configured pipeline, I’d have to make the change to the test framework, then change the pipeline manually. With my pipeline checked in as code, I could change both in one commit, preventing errors resulting from pipeline configurations going out of sync from the rest of the project.
I added three new environments to nightwatch.json:
"test_settings" : {
"default": { /*----8<----8<----8<----*/ },
"chrome": { /*----8<----8<----8<----*/ },
"firefox": {
"desiredCapabilities": {
"platform": "linux",
"browserName": "firefox",
"version": "latest"
}
},
"ie": {
"desiredCapabilities": {
"platform": "Windows 10",
"browserName": "internet explorer",
"version": "latest"
}
},
"edge": {
"desiredCapabilities": {
"platform": "Windows 10",
"browserName": "MicrosoftEdge",
"version": "latest"
}
}
}And I modified my Jenkinsfile to call them:
//----8<----8<----8<----8<----8<----8<----
sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) {
def configs = [ (1)
'chrome',
'firefox',
'ie',
'edge'
].join(',')
// Run selenium tests using Nightwatch.js
sh "./node_modules/.bin/nightwatch -e ${configs} --test tests/guineaPig.js" (2)
} //----8<----8<----8<----8<----8<----8<----| 1 | Using an array to improve readability and make it easy to add more platforms later. |
| 2 | Changed from single-quoted string to double-quoted to support variable substitution. |
|
Test frameworks have bugs too. Nightwatch.js (v0.9.8) generates incomplete JUnit files, reporting results without enough information in them to distinguish between platforms. I implemented a fix for it and submitted a PR to Nightwatch.js. This blog shows output with that fix applied locally. |
As expected, Jenkins picked up the new pipeline and ran Nightwatch.js on four platforms. Sauce Labs of course recorded the results and correctly linked them into this build. Nightwatch.js was already configured to use multiple worker threads to run tests against those platforms in parallel, and my Sauce Labs account supported running them all at the same time, letting me cover four configurations in less that twice the time, and that added time was most due to individual new environments taking longer to complete. When I move to the actual project, this will let me run broad acceptance passes quickly.
Conclusion: To Awesome and Beyond
Considering the complexity of the system, I was impressed with how easy it was to integrate Jenkins with Sauce OnDemand to start testing on multiple browsers. The plugin worked flawlessly with Jenkins Pipeline. I went ahead and ran some additional tests to show that failure reporting also behaved as expected.
//----8<----8<----8<----8<----8<----8<----
sh "./node_modules/.bin/nightwatch -e ${configs}" (1)
//----8<----8<----8<----8<----8<----8<----| 1 | Removed --test filter to run all tests |
Epilogue: Pipeline vs. Freestyle
Just for comparison here’s the final state of this job in Freestyle UI versus fully-commented pipeline code:
|
This includes the AnsiColor Plugin to support Nightwatch.js' default ANSI color output. |
Pipeline
node {
stage "Build"
checkout scm
// Install dependencies
sh 'npm install'
stage "Test"
// Add sauce credentials
sauce('f0a6b8ad-ce30-4cba-bf9a-95afbc470a8a') {
// Start sauce connect
sauceconnect(options: '', useGeneratedTunnelIdentifier: false, verboseLogging: false) {
// List of browser configs we'll be testing against.
def platform_configs = [
'chrome',
'firefox',
'ie',
'edge'
].join(',')
// Nightwatch.js supports color ouput, so wrap this step for ansi color
wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm']) {
// Run selenium tests using Nightwatch.js
// Ignore error codes. The junit publisher will cover setting build status.
sh "./node_modules/.bin/nightwatch -e ${platform_configs} || true"
}
junit 'reports/**'
step([$class: 'SauceOnDemandTestPublisher'])
}
}
}|
This pipeline expects to be run from a |
Not only is the pipeline as code more compact, it also allows for comments to further clarify what is being done. And as I noted earlier, changes to this pipeline code are committed the same as changes to the rest of the project, keeping everything synchronized, reviewable, and testable at any commit. In fact, you can view the full set of commits for this blog post in the blog/sauce-pipeline branch of the bitwiseman/JS-Nightwatch.js repository.
 | Liam Newman Liam started his software career as a tester, which might explain why he’s such a fan of CI/CD and Pipeline as Code. He has spent the majority of his software engineering career implementing Continuous Integration systems at companies big and small. He is a Jenkins project contributor and an expert in Jenkins Pipeline, both Scripted and Declarative. Liam currently works as a Jenkins Evangelist at CloudBees. When not at work, he enjoys testing gravity by doing Aikido. |
