RedHatTraining/rht-labs-docs.git

Merge branch 'master' of github.com:rht-labs/enablement-docs

donal

2018-07-03 85ecf2df38725b1cb19a1131f91622a2649df4aa

commit \| author \| age
b586be	1	# An Enslaved Hope
5a16fd	2
5e7a31	3	> In this exercise we'll break free from the chains of point'n'click Jenkins by introducing pipeline as code in the form of `Jenkinsfile`. Following this we will introduce some new Jenkins slaves that will be used in later exercises.
b586be	4
d03287	5	![jenkins-fail-meme](../images/exercise4/jenkins-fail-meme.jpeg)
D	6
b586be	7	There are a number of ways pipeline as code can be achieved in Jenkins.
5e7a31	8	* The Job DSL Plugin - this is a slightly older but very functional DSL mechanism to create reusable pipelines. Create a `groovy` file to run Jenkins Domain Specific Language to create jobs, functions and other items. In Jenkins; you then can execute this file which will build all of the config.xml files needed for each Job.
a9efee	9	* The Scripted Pipeline - The scripted pipeline introduced the Jenkinsfile and the ability for developers to write their Jenkins setup as groovy code. A repo with a Jenkinsfile in its root can be pointed to by Jenkins and it will automatically build out each of the stages described within. The scripted pipeline is ultimately Groovy at its core.
d43408	10	* The Declarative Pipeline - This approach looks to simplify and opinionate what you can do and when you can do it in a pipeline. It does this by giving you top level `block` which define sections, directives and steps. The declarative syntax is not run as groovy but you can execute groovy inside script blocks. The advantage of it over scripted is validation of the config and lighter approach with requirement to understand all of the `groovy` syntax
b586be	11
5a16fd	12	_____
D	13
	14	## Learning Outcomes
	15	As a learner you will be able to
5e7a31	16	- Use a Jenkinsfile to create a declarative pipeline to build, bake and deploy the Todolist App
CM	17	- Identify the differences between scripted, declarative and DSL pipelines
867471	18	- Create Jenkins slave nodes for use in builds in future exercises
5a16fd	19
D	20	## Tools and Frameworks
	21	> Name of tool - short description and link to docs or website
	22
b586be	23	1. [Pipeline](https://jenkins.io/doc/book/pipeline/) - Overview of the Jenkinsfile approach
D	24	1. [Pipeline Syntax](https://jenkins.io/doc/book/pipeline/syntax/) - Documentation for the declarative pipeline
	25	1. [Groovy](http://groovy-lang.org/) - Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at improving developer productivity thanks to a concise, familiar and easy to learn syntax. It integrates smoothly with any Java program, and immediately delivers to your application powerful features, including scripting capabilities, Domain-Specific Language authoring, runtime and compile-time meta-programming and functional programming.
3fdb03	26	1. [Zed Attack Proxy](https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project) - The OWASP Zed Attack Proxy (ZAP) is one of the world’s most popular free security tools and is actively maintained by hundreds of international volunteers*. It can help you automatically find security vulnerabilities in your web applications while you are developing and testing your applications. Its also a great tool for experienced pentesters to use for manual security testing.
D	27	1. [Arachni Crawler](http://www.arachni-scanner.com/) - Arachni is a feature-full, modular, high-performance Ruby framework aimed towards helping penetration testers and administrators evaluate the security of modern web applications. It is free, with its source code public and available for review. It is versatile enough to cover a great deal of use cases, ranging from a simple command line scanner utility, to a global high performance grid of scanners, to a Ruby library allowing for scripted audits, to a multi-user multi-scan web collaboration platform. In addition, its simple REST API makes integration a cinch.
5a16fd	28
D	29	## Big Picture
	30	This exercise begins cluster containing blah blah
	31
	32	_____
	33
	34	## 10,000 Ft View
867471	35	> The goal of this exercise is to move to using the Jenkinsfile in the todolist-api and todolist-fe projects. Additionally we will create new slaves for use in the next exercise.
5a16fd	36
853f0c	37	2. On Jenkins; create a multibranch pipeline project to scan the GitLab endpoint for each app. Use the Jenkinsfile provided to run the stages. Replace `<YOUR_NAME>` with the appropriate variable.
5a16fd	38
b586be	39	2. Create two new Jenkins slaves for the `OWASP ZAP` scanner and the `Arachni` WebCrawler
5a16fd	40
D	41	## Step by Step Instructions
5e7a31	42	> This is a fairly structured guide with references to exact filenames and sections of text to be added.
5a16fd	43
b586be	44	### Part 1 - The Jenkinsfile
867471	45	> _In this exercise we'll replace the Pipeline we created in Exercise 2 with a Jenkinsfile approach_
5a16fd	46
b586be	47	2. On your terminal navigate to your `todolist-api` project and checkout the pipeline feature branch that's been already created for you.
D	48	```bash
58480f	49	git checkout feature/jenkinsfile
5a16fd	50	```
D	51
5e7a31	52	2. Open up your `todolist-api` application in your favourite editor and move to the `Jenkinsfile` in the root of the project. The high-level structure of the file is shown collapsed below.
b586be	53	![pipeline-overview](../images/exercise4/pipeline-overview.png)
D	54	Some of the key things to note:
5e7a31	55	* `pipeline {}` is how all declarative Jenkins pipelines begin.
498e55	56	* `environment {}` defines environment variables to be used across all build stages
A	57	* `options {}` contains specific Job specs you want to run globally across the jobs e.g. setting the terminal colour
	58	* `stage {}` all jobs must have one stage. This is the logical part of the build that will be executed e.g. `bake-image`
b586be	59	* `steps {}` each `stage` has one or more steps involved. These could be execute shell or git checkout etc.
5e7a31	60	* `agent {}` specifies the node the build should be run on e.g. `jenkins-slave-npm`
853f0c	61	* `post {}` hook is used to specify the post-build-actions. Jenkins declarative pipeline syntax provides very useful callbacks for `success`, `failure` and `always` which are useful for controlling the job flow
W	62	* `when {}` is used for flow control. It can be used at the stage level and be used to stop pipeline entering that stage. e.g. when branch is master; deploy to `test` environment.
b586be	63
867471	64	2. The Jenkinsfile is mostly complete to do all the testing etc that was done in previous exercises. Some minor changes will be needed to orchestrate namespaces. Find and replace all instances of `<YOUR_NAME>` in the Jenkinsfile. Update the `<GIT_USERNAME>` to the one you login to the cluster with; this variable is used in the namespace of your git projects when checking out code etc. Ensure the `GITLAB_DOMAIN` matches your git host.
b586be	65	```groovy
D	66	environment {
	67	// GLobal Vars
d43408	68	PIPELINES_NAMESPACE = "<YOUR_NAME>-ci-cd"
b586be	69	APP_NAME = "todolist-api"
D	70
	71	JENKINS_TAG = "${JOB_NAME}.${BUILD_NUMBER}".replace("/", "-")
	72	JOB_NAME = "${JOB_NAME}".replace("/", "-")
	73
	74	GIT_SSL_NO_VERIFY = true
	75	GIT_CREDENTIALS = credentials('jenkins-git-creds')
5d8b9d	76	GITLAB_DOMAIN = "gitlab.apps.<SOME_DOMAIN>.com"
5965fd	77	GITLAB_PROJECT = "<GIT_USERNAME>"
b586be	78	}
D	79	```
	80
	81	2. With these changes in place, push your changes to the `feature/jenkinsfile` branch.
	82	```bash
58480f	83	git add Jenkinsfile
D	84	```
	85	```bash
	86	git commit -m "ADD - namespace and git repo to pipeline"
	87	```
	88	```bash
	89	git push
b586be	90	```
D	91
	92	2. When the changes have been successfully pushed; Open Jenkins.
	93
	94	2. Create a `New Item` on Jenkins. Give it the name `todolist-api` and select `Multibranch Pipeline` from the bottom of the list as the job type.
	95	![multibranch-select](../images/exercise4/multibranch-select.png)
	96
	97	2. On the job's configure page; set the Branch Sources to `git`
	98	![multibranch-select-git](../images/exercise4/multibranch-select-git.png)
	99
45eb81	100	2. Fill in the Git settings with your `todolist-api` GitLab url and set the credentials as you've done before. `https://gitlab.apps.lader.rht-labs.com/<YOUR_NAME>/todolist-api.git`
b586be	101	![multibranch-git](../images/exercise4/multibranch-git.png)
D	102
5e7a31	103	2. Set the `Scan Multibranch Pipeline Triggers` to be periodic and the interval to 1 minute. This will poll the GitLab instance for new branches or change sets to build.
b586be	104	![multibranch-scan-time](../images/exercise4/multibranch-scan-time.png)
D	105
853f0c	106	2. Save the Job configuration to run the intial scan. The log will show scans for `master` and `develop` branches, which have no `Jenkinsfile` so are skipped. The resulting view will show the `feature/jenkinsfile` job corresponding the only branch that currently has one. The build should run automatically.
b586be	107	![todolist-api-multi](../images/exercise4/todolist-api-multi.png)
D	108
abb5b1	109	2. The pipeline file is setup to only run `bake` & `deploy` stages when on `master` or `develop` branch. This is to provide us with very fast feedback for team members working on feature or bug fix branches. Each time someone commits or creates a new branch a basic build with testing occurs to give very rapid feedback to the team. Let's now update our `master` and `develop` branches to include the Jenkinsfile and delete the feature branch.
b586be	110	```bash
58480f	111	git checkout develop
D	112	```
	113	```bash
	114	git merge feature/jenkinsfile
abb5b1	115	# you may get merge conflicts at this point
58480f	116	```
D	117	```bash
	118	git add .
	119	```
	120	```bash
	121	git commit -m "Jenkinsfile updates"
	122	```
	123	```bash
	124	git checkout master
	125	```
	126	```bash
	127	git merge develop
	128	```
	129	```bash
	130	git push -u origin --all
	131	```
	132	```bash
b586be	133	# this is to delete the branch from the remote
58480f	134	git push origin :feature/jenkinsfile
b586be	135	```
D	136
	137	2. Back on Jenkins we should see our `todolist-api` pipelines have changed with the `develop` and `master` now appearing. The feature was deleted so this job should have gone away.
	138	![todolist-api-multi-dev-test](../images/exercise4/todolist-api-multi-dev-test.png)
	139
	140	2. With the builds running for `develop` and `master` we can explore the Blue Ocean View for Jenkins. On the Job overview page, hit the Open Blue Ocean ![open-blue-ocean](../images/exercise4/open-blue-ocean.png)
	141	button on the side to see what modern Jenkins looks like.
	142	![blue-ocean-todolist-api](../images/exercise4/blue-ocean-todolist-api.png)
	143
	144	2. We can move on to the `todolist-fe` job. The process is the same as before, checkout the feature branch
	145	```bash
58480f	146	cd todolist-fe
D	147	```
	148	```bash
	149	git checkout feature/jenkinsfile
b586be	150	```
D	151
5e7a31	152	2. Open up your `todolist-fe` application in your favourite editor and move to the `Jenkinsfile` in the root of the project. Update all `<YOUR_NAME>` and `<GIT_USERNAME>` as you did before, including in the prepare environment steps. Check the `GITLAB_DOMAIN` is set too.
abb5b1	153	![jenkinsfile-prep](../images/exercise4/jenkinsfile-prep.png)
b586be	154
5e7a31	155	2. Commit your changes to your feature branch as you did previously.
b586be	156	```bash
58480f	157	git add Jenkinsfile
D	158	```
	159	```bash
	160	git commit -m "ADD - namespace and git repo to pipeline"
	161	```
	162	```bash
	163	git push
b586be	164	```
D	165
	166	2. This time update your master and develop branches before creating config in Jenkins
	167	```
58480f	168	git checkout develop
D	169	```
	170	```bash
	171	git merge feature/jenkinsfile
abb5b1	172	# you may get merge conflicts at this point
58480f	173	```
D	174	```bash
	175	git add .
	176	```
	177	```bash
	178	git commit -m "Jenkinsfile updates"
	179	```
	180	```bash
	181	git checkout master
	182	```
	183	```bash
	184	git merge develop
	185	```
	186	```bash
abb5b1	187	# this is to delete the branch from the remote
58480f	188	git push origin :feature/jenkinsfile
D	189	```
	190	```bash
	191	git push -u origin --all
b586be	192	```
D	193
	194	2. On Jenkins; create a new `Multibranch Pipeline` job called `todolist-fe`.
	195
5e7a31	196	2. Add the `todolist-fe` git repository and set the credentials for git accordingly.
b586be	197
D	198	2. Set the trigger to scan every minute as done previously. Save the configuration and we should see the collection of Jobs as shown below.
	199	![todolist-fe-multi](../images/exercise4/todolist-fe-multi.png)
	200
6e5c07	201	2. Run the jobs and validate the app is working as expected in the `test` environment!
abb5b1	202	![todolist-test](../images/exercise4/todolist-test.png)
40c2e4	203
D	204	### Part 2 - OCP Pipeline
	205	> _This exercise adds a new BuildConfig to our cluster for the todolist-apps to run their pipelines in OpenShift using the OpenShift Jenkins Sync Plugin. We will use the OpenShift Applier to create the content in the cluster_
	206
17cc59	207	2. Open the `todolist-fe` app in your favourite editor. Move to the `.openshift-applier` directory. Explore the `template/ocp-pipeline`. This template creates a BuildConfig for OpenShift with a Jenkinsfile from a given repo. In this case; it will be the `Jenkinsfile` at the root of our application.
40c2e4	208
D	209	2. Open the `params/ocp-pipeline` file and update `PIPELINE_SOURCE_REPOSITORY_URL` with the git url of your project (Don't forget to add the `.git` at the end). For example:
	210	```
5d8b9d	211	PIPELINE_SOURCE_REPOSITORY_URL=https://gitlab.apps.<SOME_DOMAIN>.com/<GIT_USERNAME>/todolist-fe.git
40c2e4	212	PIPELINE_SOURCE_REPOSITORY_REF=develop
D	213	NAME=todolist-fe
	214	```
	215
17cc59	216	2. Create a new object in `inventory/group_vars/all.yml` to drive the `ocp-pipeline` template with the parameters file you've just created. It can be put under the existing `todolist-fe-build` object.
40c2e4	217	```yaml
D	218	- name: todolist-ocp-pipeline
	219	template: "{{ playbook_dir }}/templates/ocp-pipeline.yml"
	220	params: "{{ playbook_dir }}/params/ocp-pipeline"
	221	namespace: "{{ ci_cd_namespace }}"
	222	tags:
	223	- pipeline
	224	```
	225	![ocp-pipeline-applier](../images/exercise4/ocp-pipeline-applier.png)
	226
5e7a31	227	2. Use the OpenShift Applier to create the cluster content
40c2e4	228	```bash
58480f	229	cd .openshift-applier
D	230	```
	231	```bash
	232	ansible-playbook apply.yml -i inventory/ \
40c2e4	233	-e "filter_tags=pipeline"
D	234	```
	235
752f2a	236	2. With these changes in place, commit your changes to GitLab
D	237	```bash
58480f	238	git add .
D	239	```
	240	```bash
	241	git commit -m "ADD - ocp pipeline in git repo"
	242	```
	243	```bash
	244	git push
752f2a	245	```
D	246
40c2e4	247	2. Login to your OpenShift Cluster and go to the `<YOUR_NAME>-ci-cd` namespace. On the side menu; hit Builds > Pipeline to see your newly created pipeline running in OCP Land.
D	248	![ocp-pipeline-view](../images/exercise4/ocp-pipeline-view.png)
	249
	250	2. Running the pipeline from here will run it in Jenkins. You can see the job sync between OpenShift and Jenkins if you login to Jenkins. You should see a folder with `<YOUR_NAME>-ci-cd` and your pipeline jobs inside of it.
	251	![ocp-pipeline-jenkins](../images/exercise4/ocp-pipeline-jenkins.png)
	252
	253	2. With the configuration in place for the `todolist-fe`; repeat the process for the `todolist-api`. Update the `todolist-api/.openshift-applier/inventory/group_vars/all.yml` with a new object to drive the params and template
	254	```yaml
752f2a	255	- name: todolist-ocp-pipeline
40c2e4	256	template: "{{ playbook_dir }}/templates/ocp-pipeline.yml"
D	257	params: "{{ playbook_dir }}/params/ocp-pipeline"
	258	namespace: "{{ ci_cd_namespace }}"
	259	tags:
	260	- pipeline
	261	```
	262
	263	2. Update the `todolist-api/.openshift-applier/params/ocp-pipeline`
	264	```
5d8b9d	265	PIPELINE_SOURCE_REPOSITORY_URL=https://gitlab.apps.<SOME_DOMAIN>.com/<GIT_USERNAME>/todolist-api.git
40c2e4	266	PIPELINE_SOURCE_REPOSITORY_REF=develop
D	267	NAME=todolist-api
	268	```
	269
5e7a31	270	2. Use the OpenShift Applier to create the cluster content
40c2e4	271	```bash
58480f	272	cd todolist-api/.openshift-applier
D	273	```
	274	```bash
	275	ansible-playbook apply.yml -i inventory/ \
40c2e4	276	-e "filter_tags=pipeline"
D	277	```
	278
	279	2. Login to your OpenShift Cluster and go to the `<YOUR_NAME>-ci-cd` namespace. On the side menu; hit Builds > Pipeline to see your newly created pipeline running in OCP Land.
	280	![ocp-pipeline-view2](../images/exercise4/ocp-pipeline-view2.png)
	281
752f2a	282	2. Commit your changes to GitLab
D	283	```bash
58480f	284	git add .
D	285	```
	286	```bash
	287	git commit -m "ADD - ocp pipeline in git repo"
	288	```
	289	```bash
	290	git push
752f2a	291	```
40c2e4	292
D	293	### Part 3 - Security Scanning Slaves
b586be	294	> _This exercise focuses on updating the `enablement-ci-cd` repo with some new jenkins-slave pods for use in future exercise_
5a16fd	295
752f2a	296	#### 3a - OWASP ZAP
6e5c07	297	> _OWASP ZAP (Zed Attack Proxy) is a free open source security tool used for finding security vulnerabilities in web applications._
A	298
caec0d	299	3. On your terminal; move to the `enablement-ci-cd` repo. We need to checkout a template for OpenShift to build our Jenkins Slave images and some parameters for the `zap` slave.
6e5c07	300	```bash
caec0d	301	git checkout exercise4/zap-and-arachni params/jenkins-slave-zap templates/jenkins-slave-generic-template.yml
6e5c07	302	```
A	303
3437ff	304	3. This should have created the following files which we will fill out. We will use a `ZAP` image hosted on the `rht-labs/ci-cd` repo so there will be no `Dockerfile` needed:
09d869	305	- `params/jenkins-slave-zap`
6e5c07	306
ccc1ac	307	3. Create an object in `inventory/host_vars/ci-cd-tooling.yml` called `jenkins-slave-zap` and add the following content:
64ae3c	308	```yaml
ccc1ac	309	- name: "jenkins-slave-zap"
3fdb03	310	namespace: "{{ ci_cd_namespace }}"
D	311	template: "{{ playbook_dir }}/templates/jenkins-slave-generic-template.yml"
ccc1ac	312	params: "{{ playbook_dir }}/params/jenkins-slave-zap"
3fdb03	313	tags:
D	314	- zap
6e5c07	315	```
a82977	316	![zap-object](../images/exercise4/zap-object.png)
6e5c07	317
A	318	3. Run the ansible playbook filtering with tag `zap` so only the zap build pods are run.
	319	```bash
58480f	320	ansible-playbook apply.yml -e target=tools \
3fdb03	321	-i inventory/ \
D	322	-e "filter_tags=zap"
6e5c07	323	```
A	324
5e7a31	325	3. Head to https://console.lader.rht-labs.com on OpenShift and move to your ci-cd project > builds. You should see `jenkins-slave-zap` has been built.
09d869	326	![zap-build](../images/exercise4/zap-build.png)
6e5c07	327
752f2a	328	#### 3b - Arachni Scan
6e5c07	329	> _Arachni is a feature-full, modular, high-performance Ruby framework aimed towards helping penetration testers and administrators evaluate the security of web applications._
09d869	330
5e7a31	331	3. On your terminal; checkout the params and Docker file. The Dockerfile for the `Arachni` scanner is included here and we will point the build to it.
09d869	332	```bash
5e7a31	333	git checkout exercise4/zap-and-arachni params/jenkins-slave-arachni docker/jenkins-slave-arachni
09d869	334	```
6e5c07	335
ccc1ac	336	3. Create an object in `inventory/host_vars/ci-cd-tooling.yml` called `jenkins-slave-arachni` with the following content:
64ae3c	337	```yaml
ccc1ac	338	- name: "jenkins-slave-arachni"
3fdb03	339	namespace: "{{ ci_cd_namespace }}"
D	340	template: "{{ playbook_dir }}/templates/jenkins-slave-generic-template.yml"
ccc1ac	341	params: "{{ playbook_dir }}/params/jenkins-slave-arachni"
3fdb03	342	tags:
D	343	- arachni
6e5c07	344	```
A	345
5e7a31	346	3. Update the `jenkins-slave-arachni` files `SOURCE_REPOSITORY_URL` to point to your GitLab's hosted version of the `enablement-ci-cd` repo.
09d869	347	```
55ba17	348	SOURCE_REPOSITORY_URL=https://gitlab.apps.lader.rht-labs.com/<GIT_USERNAME>/enablement-ci-cd.git
09d869	349	SOURCE_CONTEXT_DIR=docker/jenkins-slave-arachni
D	350	BUILDER_IMAGE_NAME=registry.access.redhat.com/openshift3/jenkins-slave-base-rhel7:latest
	351	NAME=jenkins-slave-arachni
	352	SOURCE_REPOSITORY_REF=master
	353	```
	354
	355	3. With these changes in place, push your changes to the `master` branch.
	356	```bash
58480f	357	git add .
D	358	```
	359	```bash
	360	git commit -m "ADD - Arachni scanning image"
	361	```
	362	```bash
	363	git push
09d869	364	```
D	365
5e7a31	366	3. Run the Ansible playbook filtering with tag `arachni` so only the arachni build pods are run.
6e5c07	367	```bash
58480f	368	ansible-playbook apply.yml -e target=tools \
3fdb03	369	-i inventory/ \
D	370	-e "filter_tags=arachni"
6e5c07	371	```
A	372
5e7a31	373	3. Head to https://console.lader.rht-labs.com on OpenShift and move to your ci-cd project > builds. You should see `jenkins-slave-arachni`.
45eb81	374	![builds-zap-arachni](../images/exercise4/builds-zap-arachni.png)
5a16fd	375
D	376	_____
	377
	378	## Extension Tasks
	379	> _Ideas for go-getters. Advanced topic for doers to get on with if they finish early. These will usually not have a solution and are provided for additional scope._
b586be	380
D	381	Jenkins S2I
	382	- Add the multi-branch configuration to the S2I to have Jenkins come alive with the `todolist-api` and `-fe` configuration cooked into it for future uses.
	383
9af076	384	Jenkins Pipeline Extension
853f0c	385	- Add an extension to the pipeline that promotes code to the UAT environment once the master job has been successful.
5e7a31	386	- Use a WAIT to allow for manual input to approve the promotion
9af076	387
D	388	Jenkins e2e extension (blue/green)
498e55	389	- Add a step in the pipeline to only deploy to the `test` environment if the e2e tests have run successfully against which ever environment (blue or green) is not deployed.
5a16fd	390
D	391	## Additional Reading
	392	> List of links or other reading that might be of use / reference for the exercise
	393
4f0295	394	## Slide Links
RH	395
	396	- [Intro](https://docs.google.com/presentation/d/1B3Fv4g66zZ8ZkqBq9TYmImJhUDvMecXCt4q3DXGWhjc/)
	397	- [Wrap-up](https://docs.google.com/presentation/d/1EOk6y798Xh1hsaQlxRuqyr23FIIf7sNY4any_yXIL7A/)
5e7a31	398	- [All Material](https://drive.google.com/drive/folders/1oCjpl33Db7aPocmpu3NNF0B9czRvFq3m)