#+TODO: TODO(t) NEXT(n) WAITING(w) SOMEDAY(s) DELEGATED(g) PROJ(p) PLANNED(l) | DONE(d) FORWARDED(f) CANCELLED(c) #+startup: beamer #+LaTeX_CLASS: beamer #+LaTeX_CLASS_OPTIONS: [a4paper] #+LaTeX_CLASS_OPTIONS: [captions=tableheading] #+LATEX_HEADER: \usetheme{Warsaw} \usepackage{courier} #+LATEX_HEADER: \usepackage{textpos} #+LATEX_HEADER: \RequirePackage{fancyvrb} #+LATEX_HEADER: \DefineVerbatimEnvironment{verbatim}{Verbatim}{fontsize=\tiny} # +LATEX_HEADER: \setbeamercolor{title}{fg=green} # +LATEX_HEADER: \setbeamercolor{structure}{fg=black} # +LATEX_HEADER: \setbeamercolor{section in head/foot}{fg=green} # +LATEX_HEADER: \setbeamercolor{subsection in head/foot}{fg=green} # +LATEX_HEADER: \setbeamercolor{item}{fg=green} #+LATEX_HEADER: \setbeamerfont{frametitle}{family=\ttfamily} # logo # +LATEX_HEADER: \addtobeamertemplate{frametitle}{}{ \begin{textblock*}{100mm}(0.85\textwidth,-0.8cm) \includegraphics[height=0.7cm,width=2cm]{niit-logo.png} \end{textblock*}} #+OPTIONS: toc:nil ^:nil #+LANGUAGE: en #+TITLE: Diving into OpenShift * Helpful tools on the way - *jq(1)* is a JSON query tool for the command line, a must have - *yq* is a python script that does YAML queries - yes, there's also *xq* for XML but that's for the JAVA folks ;) - *jo* to produce JSON output on the shell How to get? - jq: it's very likely shipped as a package with your distribution - yq: install via =pip install --user yq= - xq: install via =pip install --user xq= - jo: get the sources from https://github.com/jpmens/jo * jq The most basic usage is filter down a JSON for a specific attributes value. #+begin_src js { "kind": "ServiceAccount", "apiVersion": "v1", "metadata": { "name": "foobar", "creationTimestamp": null } } #+end_src To get the name of the ServiceAccount you would run =jq .metadata.name= If you have a list, and want to see all items, use [], for example: =jq .items[]= If you want to get a specific list entry by index, run =jq .items[5]= to get the sixth entry * yq *yq* works very similar in the basic usage as *jq*, however per default it outputs - confusing enough - JSON (because it's a wrapper for jq). If you use the =-y= option, it will output YAML. I find this very handy for reading data from =ConfigMaps=: #+begin_example $ oc get cm foobar -o yaml | yq -y .data key: value text: hello world #+end_example * jo *jo* is a very special tool that can be very interesting if you have shell commands producing output that should be converted into JSON. More examples can be found here: https://jpmens.net/2016/03/05/a-shell-command-to-create-json-jo/ #+begin_example $ ps -fo uid,pid,ppid,stime,tty,time,args | nawk '$NF~/ksh/ { > printf("uid=%s pid=%s ppid=%s stime=%s tty=%s cputime=%s comm=%s\n" > , $1, $2, $3, $4, $5, $6, $7)}' | while read line; do jo ${line} > done | jo -pa [ { "uid": 4100, "pid": 4880, "ppid": 4001, "stime": "Jan_01", "tty": "pts/2", "cputime": "00:00", "comm": "-ksh" }, { "uid": 4100, "pid": 20599, "ppid": 4880, "stime": "23:46:19", "tty": "pts/2", "cputime": "00:00", "comm": "-ksh" } ] #+end_example * oc get -o jsonpath If you use *oc* or *kubectl* to retreive JSON data, you can also filter down with JSONPATH instead of using *jq*. The basic syntax is simple, imagine you want the hostname of a route, which is in the =host= key under the =spec=: #+begin_example $ oc get -o jsonpath='{"http://"}{.spec.host}{"\n"}' route hi http://hi-olbohlen-demo.apps-crc.testing #+end_example And because we are clever, we will create a shell alias from it: #+begin_example $ alias gr='oc get -o jsonpath="{\"http://\"}{.spec.host}{\"\n\"}" route ' [crc@lol cookie-operator]$ gr hi http://hi-olbohlen-demo.apps-crc.testing $ curl $(gr hi) hello php [...] #+end_example * Declarative versus Imperative It's important to understand that Kubernetes works declarative. What does that mean? Traditionally we work imperative with computers, we tell them what and how to execute. The problem here is, that the result is undefined, there are so many side-cases to catch. If we work declarative, we declare our will to the computer and rely on algorithms that shall fulfill our will. Of course this still can fail, but ideally the code results in "failed" or "completed". * Controller Scheme #+begin_src ditaa :file ocp-workshop1.png :cmdline -E -s 0.8 +---------------------------+ +-----------------------+ +-------------------------+ |{s} | | Kube API Server | | Kube Controller Manager | | etcd | | | | | | | <---> |etcd connector | <---> | watches API for updates | |/kubernetes.io/pods/ns/pod | | | | if update: run code | | | | REST Connector | | | +---------------------------+ +-----------------------+ +-------------------------+ ^ | HTTP(S)/REST | (JSON) v +-----------------------+ | REST Client | | | | oc/kubectl | | (converts JSON to | | output format) | | | +-----------------------+ #+end_src * Syntactic sugar OpenShift's *oc* client, but also *kubectl* have various sub-commands to make the start with Kubernetes easier. Especially in the beginning the idea to declare everything in JSON or YAML doesn't sound convincing. But if you look under the hood, commands like *oc new-app* or *kubectl create* will send declarations. You can validate that by doing a dry-run: #+begin_example $ oc create --dry-run=client -o yaml sa foobar apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: null name: foobar #+end_example * Dry-Runs There are two ways of dry-runs: =client= and =server= If we do a client dry-run, nothing gets sent to the kubernetes API, so we basically only test if we can convert the local YAML into JSON. Illegal values, missing required keys or immutable values cannot be detected. if you however use a =server= dry-run, your declaration gets sent to the API, but it will not get persisted into the ETCD. #+begin_example $ oc create --dry-run=server -o yaml sa foobar apiVersion: v1 kind: ServiceAccount metadata: creationTimestamp: "2023-01-02T20:33:23Z" name: foobar namespace: kitchen uid: 3bfc1a4c-9a87-4bfa-9dd7-c0941a98ec1f $ oc get sa foobar Error from server (NotFound): serviceaccounts "foobar" not found #+end_example As you can see, you get an =uid= and a =creationTimestamp= back. However, it will not actually be created. * Client Side debugging Turn on the debugging in *oc* / *kubectl* with =--loglevel=x=, the higher the value, the more verbose. If you want to see a lot of details, =--loglevel=9= is very helpful. For example if you want to validate how server side dry-runs work, run: #+begin_example $ oc --loglevel=9 create --dry-run=server -o yaml sa foobar I0102 21:39:10.590303 1993958 loader.go:372] Config loaded from file: /home/crc/.kube/config [...] I0102 21:39:10.818643 1993958 request.go:1073] Request Body: {"kind":"ServiceAccount", "apiVersion":"v1","metadata":{"name":"foobar","creationTimestamp":null}} I0102 21:39:10.818703 1993958 round_trippers.go:466] curl -v -XPOST -H "Content-Type: application/json" -H "Accept: application/json, */*" -H "User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4" -H "Authorization: Bearer " 'https://api.crc.testing:6443/api/v1/namespaces/kitchen/ serviceaccounts?dryRun=All&fieldManager=kubectl-create&fieldValidation=Ignore' I0102 21:39:10.822375 1993958 round_trippers.go:553] POST https://api.crc.testing:6443/api/ v1/namespaces/kitchen/serviceaccounts?dryRun=All&fieldManager=kubectl-create& fieldValidation=Ignore 201 Created in 3 milliseconds [...] #+end_example Very handy is that with =--loglevel=9=, *oc* will also print a *curl* command line that could be used to simulate that call. * Use curl to modify K8s resources Using the log output from the previous slide, we can now try to create a =ServiceAccount= via *curl*. We need of course the JSON declaration...thankfull we can generate that with a client dry-run: #+begin_example $ oc create --dry-run=client -o json sa foobar >/tmp/sa-foobar.json $ curl -k -s -XPOST -d "@/tmp/sa-foobar.json" -H "Content-Type: application/json" \ > -H "Accept: application/json, */*" -H "User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4" \ > -H "Authorization: Bearer $(oc whoami -t)" \ > 'https://api.crc.testing:6443/api/v1/namespaces/api-access/serviceaccounts?fieldManager=\ > kubectl-create&fieldValidation=Ignore' $ oc get sa foobar NAME SECRETS AGE foobar 1 11s #+end_example And how to delete the =ServiceAccount=? Simple: #+begin_example $ curl -k -XDELETE -H "Content-Type: application/json" -H "Accept: application/json, */*" \ > -H "User-Agent: oc/4.11.0 (linux/amd64) kubernetes/1928ac4" \ > -H "Authorization: Bearer $(oc whoami -t)" \ > 'https://api.crc.testing:6443/api/v1/namespaces/api-access/serviceaccounts/foobar' {"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"foobar", "namespace":"api-access","uid":"94a1c0b4-2177-41b7-bbe2-0ff5e3932785","resourceVersion":"1258915", "creationTimestamp":"2023-01-02T20:57:35Z"},"secrets":[{"name":"foobar-dockercfg-brshk"}], "imagePullSecrets":[{"name":"foobar-dockercfg-brshk"}]} $ oc get sa foobar Error from server (NotFound): serviceaccounts "foobar" not found #+end_example * Weird effects with the K8s API - the API will ignore unkown keys, so you could put a "=Olaf: was here=" key-value pair anywhere you want, the API will happily ignore it! - that means, if you *oc edit* a resource and damage the name of a key or dictionary which is not mandatory, it will throw away those definitions (=OAuth/cluster.spec.IdentityProviders=) - You can create dependent resources on non-existing objects, i.e. you can give non-existing users privileges: ETCD does not support referential integrity, so no forein key constraints - your declaration will be processed after your client terminated the REST connection, so always check the results! * Short Refresh on RBAC Kubernetes supports Role Based Access Controls (RBAC) We have five different resources here: - =Users= - =Groups= - =ServiceAccounts= - =Roles= / =ClusterRoles= - =RoleBindings= / =ClusterRoleBindings= * Users and Groups =Users= are a resource that have an access token, to talk to the external API. =Groups= bundle =Users=, as you would expect. There are also system Users (they contain colons..) which are *hard coded* and there is no way to list them via the API. The same is true for system Groups! System Users cannot authenticate via the external API. * ServiceAccounts If =Pods= would run under the User who created the =Pod=, others probably couldn't operate on them. For that reason we need =ServiceAccounts=, they bundle permissions (=Roles=, =SCCs=), specific secrets (for pulling Images, etc) and they have an API Access Token that we can use within the =Pod= to talk to the Kubernetes API. * Roles / ClusterRoles Where is the difference between =Roles= and =ClusterRoles=? It just says if the resource itself is scoped in a =Namespace= or if it's available for the whole Cluster. If you want to define a Role for all Namespaces in your Cluster, you create a =ClusterRole=. If you need a local Role just for a specific Namespace, you create a =Role=. * How does a ClusterRole look like? A =Role= contains a list of =Rules=, specifying what is allowed if you own that =Role=. The most simple one is the =cluster-admin= =ClusterRole=. #+begin_example $ oc get clusterrole cluster-admin -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: "2022-12-06T10:10:16Z" labels: kubernetes.io/bootstrapping: rbac-defaults name: cluster-admin resourceVersion: "75" uid: d6fd1146-5bc7-4815-8170-3905ae1e2856 rules: - apiGroups: - '*' resources: - '*' verbs: - '*' - nonResourceURLs: - '*' verbs: - '*' #+end_example So you can issue any command verb on any resource in any apiGroup. (iddqd) * And what is a RoleBinding? #+begin_src ditaa :file rbac.png :cmdline -E -s 0.8 +------+ +------+ | User | | Role | | |--------+ +--------------------+ +-----| | +------+ +------->| RoleBinding | <---+ +------+ | | +----------------+ +--->| | | ServiceAccount |--+ +--------------------+ | | ^ +----------------+ | | +-------+ | | Group |----------------------+ | | +-------+ #+end_src #+begin_example $ oc get rolebinding admin -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-01-02T20:55:05Z" name: admin namespace: api-access resourceVersion: "1258364" uid: a59f1f83-86b1-47eb-97ea-985f13d03102 roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: admin subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kubeadmin #+end_example * And a ClusterRoleBinding? Well, a =ClusterRoleBinding= binds on a Cluster-scope, so the permissions take effect cluster-wide. Note here that there is no =metadata.namespace:= attribute #+begin_example $ oc get clusterrolebinding cluster-admin -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: "2022-12-06T10:10:17Z" labels: kubernetes.io/bootstrapping: rbac-defaults name: cluster-admin resourceVersion: "159" uid: b971b2ac-e3a8-4db9-ba91-f017ec0a7a5f roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:masters #+end_example * I need a cookie receipt database! ...and because it makes total sense, we are going to abuse the K8s API for it. - thankfully we can extend K8s with Custom Resource Definitions (CRDs) - but how does it work? - =CustomResourceDefinitions= are themselves a =Resource=, based on a =ResourceDefinition= #+begin_example $ oc api-resources | egrep "(NAME|CustomResourceDefinition)" NAME SHORTNAMES APIVERSION NAMESPACED KIND customresourcedefinitions crd,crds apiextensions.k8s.io/v1 false CustomResourceDefinition $ oc explain crds KIND: CustomResourceDefinition VERSION: apiextensions.k8s.io/v1 DESCRIPTION: CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format <.spec.name>.<.spec.group>. [...] #+end_example * Preparing the cookiereceipts CRD Let's create the CRD from https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/cookie-crd.yaml #+begin_example $ oc new-project kitchen Now using project "kitchen" on server "https://api.crc.testing:6443". $ oc create -f cookie-crd.yaml Error from server (Forbidden): error when creating "cookie-crd.yaml": customresourcedefinitions.apiextensions.k8s.io is forbidden: User "developer" cannot create resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope $ oc login -u kubeadmin $ oc create -f cookie-crd.yaml customresourcedefinition.apiextensions.k8s.io/cookiereceipts.de.eenfach.olbohlen created $ oc login -u developer #+end_example Now the cluster knows about the CRD and we could store receipts! * Storing some sample receipts We try to store sample cookie receipts...but: #+begin_example $ oc create -f sample-cookie.yaml Error from server (Forbidden): error when creating "sample-cookie.yaml": cookiereceipts.de.eenfach.olbohlen is forbidden: User "developer" cannot create resource "cookiereceipts" in API group "de.eenfach.olbohlen" in the namespace "kitchen" Error from server (Forbidden): error when creating "sample-cookie.yaml": cookiereceipts.de.eenfach.olbohlen is forbidden: User "developer" cannot create resource "cookiereceipts" in API group "de.eenfach.olbohlen" in the namespace "kitchen" #+end_example We need to set up some RBAC resources first: - a =ClusterRole= that allows viewing receipts - a =ClusterRole= that allows editing receipts - and a =ClusterRoleBinding= that allows that for authenticated users * Creating RBAC resources Apply the RBAC definitions from: https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/cookie-rbac.yaml #+begin_example $ oc login -u kubeadmin $ oc create -f cookie-rbac.yaml clusterrole.rbac.authorization.k8s.io/cookiereceipt-edit created clusterrole.rbac.authorization.k8s.io/cookiereceipt-view created clusterrolebinding.rbac.authorization.k8s.io/cookiereceipt-edit created #+end_example The =ClusterRoleBinding= "cookiereceipt-edit" allows =system:authenticated:oauth= group members to edit =cookiereceipts=. =system:authenticated:oauth= contains all users that logged in via the OAuth service (via an =IdentityProvider=). * Storing some sample receipts (hopefully this time!!) Now we should be able to create the sample receipts: #+begin_example $ oc login -u developer $ oc create -f sample-cookie.yaml cookiereceipt.de.eenfach.olbohlen/vintage-chocolate-chip created cookiereceipt.de.eenfach.olbohlen/double-dipped-shortbread created $ oc get cookiereceipt NAME AGE double-dipped-shortbread 17s vintage-chocolate-chip 17s #+end_example There is no functionality here - we just stored the receipts in the etcd via the K8s API. * Can we validate that in the ETCD? Yes, we can use *etcdctl* to look into the db #+begin_example $ oc login -u kubeadmin $ oc rsh -n openshift-etcd -c etcdctl \ > etcd-crc-pbwlw-master-0 etcdctl get / --prefix --keys-only | grep -i cookie /kubernetes.io/apiextensions.k8s.io/customresourcedefinitions/cookiereceipts.de.eenfach.olbohlen /kubernetes.io/apiserver.openshift.io/apirequestcounts/cookiereceipts.v1.de.eenfach.olbohlen /kubernetes.io/clusterrolebindings/cookiereceipt-edit /kubernetes.io/clusterroles/cookiereceipt-edit /kubernetes.io/clusterroles/cookiereceipt-view /kubernetes.io/de.eenfach.olbohlen/cookiereceipts/kitchen/double-dipped-shortbread /kubernetes.io/de.eenfach.olbohlen/cookiereceipts/kitchen/vintage-chocolate-chip /kubernetes.io/secrets/openshift-machine-config-operator/cookie-secret #+end_example The =etcd pod= in the =openshift-etcd= project has a sidecar container called =etcdctl= which contains the *etcdctl* utility and the correct environment. The data hierarchy starts with =/= and with =--keys-only= we don't get the values. * Let's look into a receipt Check if we can *etcdctl get* on a known receipt: #+begin_example $ oc rsh -n openshift-etcd -c etcdctl etcd-crc-pbwlw-master-0 etcdctl \ > get /kubernetes.io/de.eenfach.olbohlen/cookiereceipts/kitchen/vintage-chocolate-chip /kubernetes.io/de.eenfach.olbohlen/cookiereceipts/kitchen/vintage-chocolate-chip {"apiVersion":"de.eenfach.olbohlen/v1","kind":"CookieReceipt","metadata":{ "creationTimestamp":"2023-01-02T15:18:49Z","generation":1,"managedFields": [{"apiVersion":"de.eenfach.olbohlen/v1","fieldsType":"FieldsV1", [...] #+end_example With *etcdctl put*, we could also update the entry and *etcdctl watch* allows us to see updates to reosurces. #+begin_example $ oc rsh -n openshift-etcd -c etcdctl etcd-crc-pbwlw-master-0 etcdctl watch \ > --prefix /kubernetes.io/de.eenfach.olbohlen/cookiereceipts PUT /kubernetes.io/de.eenfach.olbohlen/cookiereceipts/kitchen/double-dipped-shortbread {"apiVersion":"de.eenfach.olbohlen/v1","kind":"CookieReceipt","metadata":{"annotations": {"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\": \"de.eenfach.olbohlen/v1\",\"kind\":\"CookieReceipt\",\"metadata\":{ [...] #+end_example (run an *oc* apply/replace/create/delete in another terminal) * Now can we do anything with our receipts? Of course we can *oc get -o yaml* for example on them and filter: #+begin_example $ oc get cookiereceipt vintage-chocolate-chip -o yaml | yq -y .spec.ingredients[0] amount: 150 name: salted butter remarks: softened unit: grams #+end_example This is handy, as we can extract exactly the data which we need at a time. But...it's a lot of manual work... * I'm an operator with my pocket calculator Operators were introduced as "Kubernetes Native Applications" and that actually means nothing. Operators are in the end just =Pods=. These Pods run one or more containers, but one container should run a =Controller= that can interprete your =CustomResources=. So let's write a CookieReceipt Operator. In shell-script... :) Of course this Operator is not compatible with the =OperatorLifecycyleManager= (=OLM=), so we have to install it manually. * What do we need? We need: - a ContainerImage - and therefore probably a *Containerfile* - Controller code Then we are going to build the Operator ContainerImage and push it to a Registry. * Let's review the Containerfile The Containerfile is here: https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/Containerfile the base image is a "kshbase" image, which itself is based upon ubi9 containing also a ksh93 and an oc client. * Have a look at the Controller The controller is written in KornShell 93 (ksh93), which is mostly bash compatible :) The code is here: https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/receipt-processor.ksh * Now let's also have a look at the deployment Note: this deployment does not use an =ImageStream=, so it would work also on native k8s https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/cookie-operator-deployment.yaml This deployment requires a =ServiceAccount= called "cookieprocessor", this =ServiceAccount= provides a Token to authenticate against the API (which we use in the controller script). * The ServiceAccount We need a =ServiceAccount=, but that alone will not help. The =ServiceAccount= is NOT member of =system:authenticated:oauth=, so it can't read =cookiereceipts= based on the =ClusterRoleBinding= we created earlier. For that reason we also create a =RoleBinding= (namespaced!) that allows reading receipts: https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/cookieprocessor-sa.yaml * Building the stuff together #+begin_example $ oc create -f cookieprocessor-sa.yaml serviceaccount/cookieprocessor created rolebinding.rbac.authorization.k8s.io/cookiereceipt-view created #+end_example The registry docker.eenfach.de requires login credentials, so we need to set up a secret and link it. First login to the registry with *podman login*, then pick the resulting auth.json: #+begin_example $ podman login -u olbohlen docker.eenfach.de Password: Login Succeeded! $ oc create secret generic docker-eenfach-de \ > --from-file=.dockerconfigjson=${XDG_RUNTIME_DIR}/containers/auth.json \ > --type kubernetes.io/dockerconfigjson secret/docker-eenfach-de created $ oc secrets link cookieprocessor docker-eenfach-de --for pull #+end_example * Deploying the Operator Now that we have everything in place, we will just deploy the Operator Pod: #+begin_example $ oc create -f cookie-operator-deployment.yaml deployment.apps/receipt-processor created $ oc get pod NAME READY STATUS RESTARTS AGE receipt-processor-7f9969697b-qt9lv 1/1 Running 0 17s $ oc logs -f receipt-processor-7f9969697b-qt9lv New receipt found: double-dipped-shortbread -------------------------------------------------------------------------- Pre: we heat up the oven to 180 degrees Celsius Fetching ingredients from receipt: ---------------------------------- Fetching 200grams of salted butter (softened) [...] #+end_example The Operator will process both sample receipts. * Test the Operator We should test if the Operator notices new receipts, so let's create a third receipt from https://www.eenfach.de/gitblit/blob/~olbohlen!cookie-operator.git/master/oaty-hazelnut-cookies.yaml #+begin_example $ oc create -f oaty-hazelnut-cookies.yaml cookiereceipt.de.eenfach.olbohlen/oaty-hazelnut created #+end_example After a few seconds, we should see in the Operator log: #+begin_example New receipt found: oaty-hazelnut -------------------------------------------------------------------------- Pre: we heat up the oven to 180 degrees Celsius [...] #+end_example * Test for updated receipts But what if we update a resource? Keep the *oc logs -f* on the Operator Pod open, and in another terminal let's patch a receipt. #+begin_example $ oc patch cookiereceipts double-dipped-shortbread --type merge \ > -p '{"spec":{"temperature":172}}' cookiereceipt.de.eenfach.olbohlen/double-dipped-shortbread patched #+end_example And again in the log you should see #+begin_example New receipt found: double-dipped-shortbread -------------------------------------------------------------------------- Pre: we heat up the oven to 172 degrees Celsius Fetching ingredients from receipt: ---------------------------------- [...] #+end_example * Thank You Thank you for your attention.\\ \\ Do you have any questions?\\ \\ Feel free to ask now or contact me later: [[mailto:olaf.bohlen@niit.com][olaf.bohlen@niit.com]]