| commit | author | age | ||
| 064b38 | 1 | # The Non-Functionals Strike back |
| 5277ca | 2 | > In this exercise we explore the non-functional side of testing. |
| 5afd94 | 3 | |
| a6fbc9 | 4 | <!--  --> |
| ec5c7d | 5 | |
| D | 6 | ## Exercise Intro |
| 1173e5 | 7 | Non-functional testing provides valuable insights into code quality and application performance. Often overlooked but usually one of the most essential types of testing, non-functional testing types can include, but are not limited to |
| ec5c7d | 8 | - Performance Testing |
| D | 9 | - Security testing |
| 10 | - Static Code analysis | |
| 11 | - Vulnerability scanning | |
| 12 | ||
| 1173e5 | 13 | There are many tools out there for supporting these testing types but often they are left to the end of a delivery. Many traditional projects will leave Performance testing or security sign off to a few weeks before Go Live. This raises the question of what do we do if things do not pass these tests? Do we hold off the Go Live or accept the risk? In most cases we can learn earlier if things will be show stoppers and more importantly we can automate them. |
| ec5c7d | 14 | |
| e43fd2 | 15 | For example; imagine a developer called `Sam` has checked in some poorly performing function into an API that spikes its response time. Let's call this `Sam Code`. This `Sam Code` may not be caught by our Unit Tests but could have very big impact on application usability. Before building code on top this and it becoming more of an issue at the end of a project; we could code and capture metrics around API response time and track these over time and hopefully spot regressions earlier. |
| ec5c7d | 16 | |
| 2576ab | 17 | Another one of the age old questions is "How do we know we're testing enough?". Well the simple answer is you can never do enough testing! But how do we know we are testing the right things? Code coverage metrics can be run on our application while running the tests. They can identify what files are being tested and a line by line of how many times a test executes against a block of code. Reporting these metrics in our pipeline gives a greater handle on the quality of our testing. |
| ec5c7d | 18 | |
| 2576ab | 19 | Static code analysis can provide great insights into the quality of the code we've written. By analysing code without executing it bugs and common gotchas can be identified. Code complexity can be assessed using tooling such as SonarQube. Linting of code is a useful thing for non compiled languages such as JavaScript and can be executed in a build to provide some feedback. |
| ec5c7d | 20 | |
| D | 21 | Integrating these tools into the developer workflow can greatly improve quality and readability of code. Putting them in the pipeline ensures they are executed - removing the _"It works on my machine"_ view some may take. |
| 22 | ||
| 23 | #### Why run non functional tests? | |
| 24 | - Early identify common code gotchas and pitfalls | |
| 25 | - Tighten our feedback loop by executing non functional tests earlier in a project lifecycle | |
| 26 | - Improve Code readability with agreeing a coding standard and then codifying it. | |
| 27 | - Improve trust in the code quality and bring together security teams with dev teams earlier | |
| 28 | - They can be automated; so why not run them as part of a build! | |
| 29 | ||
| 43f2f2 | 30 | _____ |
| D | 31 | |
| 32 | ## Learning Outcomes | |
| 33 | As a learner you will be able to | |
| 007b55 | 34 | - Create additional Jenkins stages to scan for security vulnerabilities in the Apps |
| 064b38 | 35 | - Assess test quality by producing coverage reports as part of a build |
| 007b55 | 36 | - Improve code readability with linting |
| D | 37 | - Do some light performance testing to monitor throughput of APIs |
| 43f2f2 | 38 | |
| D | 39 | ## Tools and Frameworks |
| b5d705 | 40 | > Below is a collection of the new frameworks and tools that will be used in this exercise |
| 43f2f2 | 41 | |
| 007b55 | 42 | 1. [eslint](https://eslint.org/) - ESLint is an open source JavaScript linting utility originally created by Nicholas C. Zakas in June 2013. Code linting is a type of static analysis that is frequently used to find problematic patterns or code that doesn’t adhere to certain style guidelines. There are code linters for most programming languages, and compilers sometimes incorporate linting into the compilation process. |
| a6fbc9 | 43 | <!-- 1. [Zed Attack Proxy](https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project) - The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers*. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications. Its also a great tool for experienced pentesters to use for manual security testing. --> |
| RS | 44 | 2. [Arachni Crawler](http://www.arachni-scanner.com/) - Arachni is a feature-full, modular, high-performance Ruby framework aimed towards helping penetration testers and administrators evaluate the security of modern web applications. It is free, with its source code public and available for review. It is versatile enough to cover a great deal of use cases, ranging from a simple command line scanner utility, to a global high performance grid of scanners, to a Ruby library allowing for scripted audits, to a multi-user multi-scan web collaboration platform. In addition, its simple REST API makes integration a cinch. |
| 45 | <!-- 1. [stryker](http://stryker-mutator.io/) - Mutation testing! What is it? Bugs, or mutants, are automatically inserted into your production code. Your tests are run for each mutant. If your tests fail then the mutant is killed. If your tests passed, the mutant survived. The higher the percentage of mutants killed, the more effective your tests are. It's really that simple. --> | |
| 43f2f2 | 46 | |
| D | 47 | ## Big Picture |
| a6fbc9 | 48 | > In the previous exercise; we introduced pipeline-as-code and new Jenkins Slave nodes. This exercise focuses on extending the pipeline with non-functional testing and some automated security testing. |
| 664135 | 49 | |
| 14a966 | 50 |  |
| 43f2f2 | 51 | |
| D | 52 | _____ |
| 53 | ||
| 54 | ## 10,000 Ft View | |
| a6fbc9 | 55 | > This lesson will use the Arachni scanner to improve the pipeline. Linting will be included in the build and code coverage too. |
| 43f2f2 | 56 | |
| a6fbc9 | 57 | 1. Add a new stage after the e2e tests on the front end to run the Arachni scanner against the deployed apps. |
| 43f2f2 | 58 | |
| 2576ab | 59 | 2. Add Code Coverage reporting to the build for gaining greater insight into test improvements. |
| 007b55 | 60 | |
| a6fbc9 | 61 | 3. Add `npm run lint` to the Frontend and report the result in Jenkins. |
| 007b55 | 62 | |
| a6fbc9 | 63 | 4. Create a new Jenkins job to run some light performance testing against the API layer using the perf tests tasks. |
| 43f2f2 | 64 | |
| D | 65 | ## Step by Step Instructions |
| a6fbc9 | 66 | <!-- > This is a well structured guide with references to exact filenames and indications as to what should be done. --> |
| 43f2f2 | 67 | |
| 1173e5 | 68 | ### Part 1 - Add Security scanning to the pipeline |
| a6fbc9 | 69 | > _In this exercise, the first of our non-functional testing is explored in the form of some security scanning. We will add the scan to our Jenkinsfile and have them run as a new stage_ |
| 43f2f2 | 70 | |
| a6fbc9 | 71 | 1. Open the `todolist-fe` application's `Jenkinsfile` in your favourite editor. The file is stored in the root of the project. Ensure that you using the `master` branch of `todolist-fe` |
| RS | 72 | ```bash |
| 73 | cd todolist-fe | |
| 74 | git checkout master | |
| 75 | ``` | |
| dc377a | 76 | |
| a6fbc9 | 77 | 2. The file is laid out with a collection of stages that correspond to each part of our build as seen below. We will create a new stage to for the Arachni scan. |
| cd34aa | 78 |  |
| dc377a | 79 | |
| a6fbc9 | 80 | 3. Create a new stage called `Arachni Scan` underneath the `stage("e2e test") { }` section as shown below. The contents of the `e2e test` have been removed for simplicity. |
| dc377a | 81 | ```groovy |
| D | 82 | stage("e2e test") { |
| 83 | // ... stuff in here .... | |
| 84 | } | |
| a6fbc9 | 85 | stage('Arachni Scan') { |
| RS | 86 | agent { |
| 87 | node { | |
| 88 | label "jenkins-slave-arachni" | |
| dc377a | 89 | } |
| D | 90 | } |
| a6fbc9 | 91 | when { |
| RS | 92 | expression { GIT_BRANCH ==~ /(.*master|.*develop)/ } |
| 93 | } | |
| 94 | steps { | |
| 95 | sh ''' | |
| 96 | /arachni/bin/arachni http://${E2E_TEST_ROUTE} --report-save-path=arachni-report.afr | |
| 97 | /arachni/bin/arachni_reporter arachni-report.afr --reporter=xunit:outfile=report.xml --reporter=html:outfile=web-report.zip | |
| 98 | unzip web-report.zip -d arachni-web-report | |
| 99 | ''' | |
| 100 | } | |
| 101 | post { | |
| 102 | always { | |
| 103 | junit 'report.xml' | |
| 104 | publishHTML target: [ | |
| 105 | allowMissing: false, | |
| 106 | alwaysLinkToLastBuild: false, | |
| 107 | keepAll: true, | |
| 108 | reportDir: 'arachni-web-report', | |
| 109 | reportFiles: 'index.html', | |
| 110 | reportName: 'Arachni Web Crawl' | |
| 111 | ] | |
| 112 | } | |
| 113 | } | |
| 114 | } | |
| 43f2f2 | 115 | ``` |
| dc377a | 116 | |
| a6fbc9 | 117 | <!-- 2. Let's start filling out the configuration for the OWASP Zap scan first. We will set the label to our slave created in previous exercise and a `when` condition of the master or develop branch. |
| dc377a | 118 | ```groovy |
| D | 119 | stage('OWASP Scan') { |
| 120 | agent { | |
| 121 | node { | |
| 122 | label "jenkins-slave-zap" | |
| 123 | } | |
| 124 | } | |
| 125 | when { | |
| 126 | expression { GIT_BRANCH ==~ /(.*master|.*develop)/ } | |
| 127 | } | |
| 128 | } | |
| 43f2f2 | 129 | ``` |
| dc377a | 130 | |
| 3dce53 | 131 | 2. Add a `step` with a `sh` command to run the tool by passing in the URL of the app we're going to test. |
| dc377a | 132 | ```groovy |
| D | 133 | stage('OWASP Scan') { |
| 3dce53 | 134 | agent { |
| D | 135 | node { |
| 136 | label "jenkins-slave-zap" | |
| 137 | } | |
| dc377a | 138 | } |
| 3dce53 | 139 | when { |
| D | 140 | expression { GIT_BRANCH ==~ /(.*master|.*develop)/ } |
| 141 | } | |
| 142 | steps { | |
| 143 | sh ''' | |
| 144 | /zap/zap-baseline.py -r index.html -t ${E2E_TEST_ROUTE} || return_code=$? | |
| 145 | echo "exit value was - " $return_code | |
| 146 | ''' | |
| 147 | } | |
| dc377a | 148 | } |
| D | 149 | ``` |
| 150 | ||
| e43fd2 | 151 | 2. Finally add the reporting for Jenkins in `post` hook of our Declarative Pipeline. This is to report the findings of the scan in Jenkins as an HTML report. |
| dc377a | 152 | ```groovy |
| D | 153 | stage('OWASP Scan') { |
| 154 | agent { | |
| 155 | node { | |
| 156 | label "jenkins-slave-zap" | |
| 157 | } | |
| 158 | } | |
| 159 | when { | |
| 160 | expression { GIT_BRANCH ==~ /(.*master|.*develop)/ } | |
| 161 | } | |
| 162 | steps { | |
| 163 | sh ''' | |
| 3dce53 | 164 | /zap/zap-baseline.py -r index.html -t http://${E2E_TEST_ROUTE} || return_code=$? |
| D | 165 | echo "exit value was - " $return_code |
| dc377a | 166 | ''' |
| D | 167 | } |
| 168 | post { | |
| 169 | always { | |
| 170 | // publish html | |
| 171 | publishHTML target: [ | |
| 172 | allowMissing: false, | |
| 173 | alwaysLinkToLastBuild: false, | |
| 174 | keepAll: true, | |
| 175 | reportDir: '/zap/wrk', | |
| 176 | reportFiles: 'index.html', | |
| 177 | reportName: 'Zap Branniscan' | |
| 178 | ] | |
| 179 | } | |
| 180 | } | |
| 181 | } | |
| a6fbc9 | 182 | ``` --> |
| dc377a | 183 | |
| a6fbc9 | 184 | <!-- 2. Let's add our Arachni Scan to the second part of the parallel block. The main difference between these sections is Jenkins will report an XML report too for failing the build accordingly. Below is the snippet for the Arachni scanning. --> |
| RS | 185 | <!-- ```groovy |
| dc377a | 186 | stage('Arachni Scan') { |
| D | 187 | agent { |
| 188 | node { | |
| 189 | label "jenkins-slave-arachni" | |
| 190 | } | |
| 191 | } | |
| 192 | when { | |
| 193 | expression { GIT_BRANCH ==~ /(.*master|.*develop)/ } | |
| 194 | } | |
| 195 | steps { | |
| 196 | sh ''' | |
| da9923 | 197 | /arachni/bin/arachni http://${E2E_TEST_ROUTE} --report-save-path=arachni-report.afr |
| dc377a | 198 | /arachni/bin/arachni_reporter arachni-report.afr --reporter=xunit:outfile=report.xml --reporter=html:outfile=web-report.zip |
| D | 199 | unzip web-report.zip -d arachni-web-report |
| 200 | ''' | |
| 201 | } | |
| 202 | post { | |
| 203 | always { | |
| 204 | junit 'report.xml' | |
| 205 | publishHTML target: [ | |
| 206 | allowMissing: false, | |
| 207 | alwaysLinkToLastBuild: false, | |
| 208 | keepAll: true, | |
| 209 | reportDir: 'arachni-web-report', | |
| 210 | reportFiles: 'index.html', | |
| 211 | reportName: 'Arachni Web Crawl' | |
| 212 | ] | |
| 213 | } | |
| 214 | } | |
| 215 | } | |
| a6fbc9 | 216 | ``` --> |
| dc377a | 217 | |
| a6fbc9 | 218 | 4. With this config in place, commit your code (from your terminal). Wait for a few minutes until a new build in Jenkins is trigerred: |
| 7c2c91 | 219 | ```bash |
| fad576 | 220 | git add . |
| D | 221 | ``` |
| 222 | ```bash | |
| 223 | git commit -m "ADD - security scanning tools to pipeline" | |
| 224 | ``` | |
| 225 | ```bash | |
| 226 | git push | |
| 7c2c91 | 227 | ``` |
| D | 228 | |
| a6fbc9 | 229 | 5. Check out the Blue Ocean Jenkins view for how the new stage is viewed! |
| RS | 230 |  |
| 3dce53 | 231 | |
| a6fbc9 | 232 | 6. Once the Jobs have completed; navigate to the Jobs status and see the scores. You can find the graphs and test reports on overview of the Job. Explore the results! |
| cd34aa | 233 |  |
| D | 234 |  |
| 235 | ||
| 236 | <p class="tip"> | |
| a6fbc9 | 237 | NOTE - your build may have failed, or marked as unstable because of the a security failure but the reports should still be generated, it is OK to proceed with the next exercise! |
| cd34aa | 238 | </p> |
| 43f2f2 | 239 | |
| a6fbc9 | 240 | <!-- 2. TODO - add solution for failing Security scans! --> |
| 2c15b7 | 241 | |
| 007b55 | 242 | ### Part 2 - Add Code Coverage & Linting to the pipeline |
| dabc6f | 243 | > _Let's continue to enhance our pipeline with some non-functional testing. Static code analysis and testing coverage reports can provide a useful indicator on code quality and testing distribution_ |
| 43f2f2 | 244 | |
| a6fbc9 | 245 | 1. Coverage reports are already being generated as part of the tests. We can have Jenkins produce a HTML report showing in detail where our testing is lacking. Open the `todolist-fe` in your favourite editor. |
| dabc6f | 246 | |
| a6fbc9 | 247 | 2. Open the `Jenkinsfile` in the root of the project; move to the `stage("node-build"){ ... }` section. In the `post` section add a block for producing a `HTML` report as part of our builds. This is all that is needed for Jenkins to report the coverage stats. |
| dabc6f | 248 | ```groovy |
| D | 249 | // Post can be used both on individual stages and for the entire build. |
| 250 | post { | |
| 251 | always { | |
| 252 | archive "**" | |
| 253 | junit 'test-report.xml' | |
| 254 | // publish html | |
| 255 | publishHTML target: [ | |
| 256 | allowMissing: false, | |
| 257 | alwaysLinkToLastBuild: false, | |
| 258 | keepAll: true, | |
| 259 | reportDir: 'reports/coverage', | |
| 260 | reportFiles: 'index.html', | |
| 261 | reportName: 'Code Coverage' | |
| 262 | ] | |
| 263 | } | |
| 264 | ``` | |
| 265 | ||
| 1173e5 | 266 | 3. To get the linting working; we will add a new step to our `stage("node-build"){ }` section to lint the JavaScript code. Continuing in the `Jenkinsfile`, After the `npm install`; add a command to run the linting. |
| dabc6f | 267 | ```groovy |
| D | 268 | echo '### Install deps ###' |
| 269 | sh 'npm install' | |
| 270 | echo '### Running linting ###' | |
| 3dce53 | 271 | sh 'npm run lint' |
| dabc6f | 272 | ``` |
| D | 273 | |
| a6fbc9 | 274 | 4. Save the `Jenkinsfile` and commit it to trigger a build with some more enhancements. |
| dabc6f | 275 | ```bash |
| fad576 | 276 | git add . |
| D | 277 | ``` |
| 278 | ```bash | |
| 279 | git commit -m "ADD - linting and coverage to the pipeline" | |
| 280 | ``` | |
| 281 | ```bash | |
| 282 | git push | |
| dabc6f | 283 | ``` |
| D | 284 | |
| a6fbc9 | 285 | 5. A new Jenkins build is trigerred and the pipeline fails. Look in the Jenkins log for what the issue might be: |
| 3dce53 | 286 |  |
| D | 287 | |
| a6fbc9 | 288 | 6. Fix the error identified by the linter by commenting out the offending line. Commit and push your changes to trigger a new build. |
| RS | 289 | |
| 290 | 7. Once the build passes again, view the coverage graph; go to the job's build page and open the `Code Coverage` report from the nav bar on the side. Sometimes this won't display on the `yourjenkins.com/job/todolist-fe/job/branch/` sidebar, click on an individual build in the build history and it should appear on the side navbar. | |
| 3dce53 | 291 |  |
| D | 292 | |
| a6fbc9 | 293 | 8. Open the report to drill down into detail of where testing coverage could be improved! |
| 5277ca | 294 |  |
| D | 295 | <p class="tip"> |
| 1173e5 | 296 | NOTE - a good practice for teams is to try and increase the code coverage metrics over the life of a project. Teams will often start low and use practices such as retrospective to increase the quality at specific times. |
| 5277ca | 297 | </p> |
| D | 298 | |
| a6fbc9 | 299 | <!-- 3. (Optional Step) - Install the Checkstyle plugin; and add `checkstyle pattern: 'eslint-report.xml'` below the `publishHTML` block to add reporting to Jenkins! --> |
| dabc6f | 300 | |
| 5277ca | 301 | ### Part 3 - Nightly light performance testing |
| a6fbc9 | 302 | > _In this part of the exercise, we will execute light performance tasks in our API to collect data about throughput time in hopes if the API ever has some `Sam` quality code checked in, we will spot it_ |
| dabc6f | 303 | |
| 1173e5 | 304 | An arbitrary value for the APIs to respond in has been chosen. It is set in the `todolist-api/tasks/perf-test.js` file. In this exercise we will get Jenkins to execute the tests and fail based on the score set there! |
| 5277ca | 305 | |
| a6fbc9 | 306 | 1. Create a new Item on Jenkins, `nightly-perf-test` and make it a freestyle job. |
| 2c15b7 | 307 |  |
| D | 308 | |
| a6fbc9 | 309 | 2. Set the `label` on `Restrict where this project can be run` to `jenkins-slave-npm` one used by the build jobs previously. |
| 2c15b7 | 310 |  |
| 5277ca | 311 | |
| a6fbc9 | 312 | 3. In the SCM section; set the project to use the `todolist-api` git project. Set the credentials accordingly. |
| 2c15b7 | 313 |  |
| D | 314 | |
| 315 | 4. Set the build to execute each night; for example 0300 in the morning. Hit `Build periodically` on the Build Triggers section and set it to `H 3 * * *`. | |
| 316 |  | |
| 317 | ||
| a6fbc9 | 318 | 5. Set the `Color ANSI Console Output` on the Build Environment section. |
| 5277ca | 319 | |
| a6fbc9 | 320 | 6. Click `add build step` and select `execute shell` and add the following to it, replacing `<YOUR_NAME>` as expected. We will just test the `create` and `show` API for the moment. We are grabbing the response code of the perf-test to keep Jenkins running both shells steps and then exiting with whichever fails: |
| 5277ca | 321 | ```bash |
| 530a25 | 322 | export E2E_TEST_ROUTE=todolist-api-<YOUR_NAME>-dev.<APPS_URL> |
| 5277ca | 323 | npm install |
| D | 324 | set +e |
| 325 | npm run perf-test:create | |
| 326 | rc1=$? | |
| 327 | npm run perf-test:show | |
| 328 | rc2=$? | |
| 329 | set -e | |
| 330 | exit $(($rc1 | $rc2)) | |
| 331 | ``` | |
| 332 | ||
| a6fbc9 | 333 | 7. On the Post Build actions section we will plot the data from the perf tests in Jenkins. Add a `Post-build Action > Plot Build Data`. |
| 5277ca | 334 | |
| a6fbc9 | 335 | 8. On the new dialog, name the Plot group e.g. `bench-tests` and add `create-api` as the Plot title. Set the `Number of Builds to Include` to a large number like `100`. Set the Data Series file to be `reports/server/perf/create-perf-score.csv` and mark the `Load data from CSV field` checkbox. Apply those changes |
| 5277ca | 336 |  |
| D | 337 | |
| a6fbc9 | 338 | 9. Hit `Add Plot` to add another. Set Plot group to `bench-tests` again but this time setting the Plot title to `show-api`. Set the Data Series file to be `reports/server/perf/show-perf-score.csv` and mark the `Load data from CSV` radio button. Save those changes and run the job (Job could take a while to execute!). |
| 5277ca | 339 | |
| a6fbc9 | 340 | 10. Run it a few times to start to generate the data points on the plot. The `bench-tests` plot is available on the job's homepage |
| 5277ca | 341 |  |
| 43f2f2 | 342 | |
| D | 343 | _____ |
| 344 | ||
| a6fbc9 | 345 | <!-- ## Extension Tasks |
| 43f2f2 | 346 | > _Ideas for go-getters. Advanced topic for doers to get on with if they finish early. These will usually not have a solution and are provided for additional scope._ |
| D | 347 | |
| dabc6f | 348 | - Enhance the `todolist-api` with the security scanning tools as you've done for the `todolist-api` |
| D | 349 | - Enhance the `todolist-api` with the coverage reporting as you've done for `todolist-api` |
| 007b55 | 350 | - Add Black Duck or other package scanning tooling for our NodeJS app |
| D | 351 | - Add Container Vulnerability scanning tooling to the pipeline |
| 5277ca | 352 | - Add `Stryker` to create mutants and do additional non functional testing of the App |
| dabc6f | 353 | - Add the Checkstyle plugin to Jenkins for reporting scores |
| 43f2f2 | 354 | |
| D | 355 | ## Additional Reading |
| 356 | > List of links or other reading that might be of use / reference for the exercise | |
| 357 | ||
| 01c4da | 358 | ## Slide Links |
| RH | 359 | |
| 360 | - [Intro](https://docs.google.com/presentation/d/1YQ0hUV3o7DW8O40SiI-BQZXCOSVeQGjo2iTxCL2GZfk/) | |
| 361 | - [Wrap-up](https://docs.google.com/presentation/d/102hRHDlC9PUIsMs3m1fZy8QUaB5UKzBlhBPdehRWw38/) | |
| a6fbc9 | 362 | - [All Material](https://drive.google.com/drive/folders/1seT0V3ABHNonvtFvORNt836NgSeYPuWW) --> |
