commit | author | age
|
222cf2
|
1 |
|
9c6c5a
|
2 |
# userland-tools |
AŠ |
3 |
|
222cf2
|
4 |
oi-userland comes with some tools used all over the build system and |
9c6c5a
|
5 |
these tools reside in the [tools/](https://github.com/OpenIndiana/oi-userland/tree/oi/hipster/tools) directory. |
AŠ |
6 |
|
222cf2
|
7 |
## userland-fetch |
D |
8 |
_userland-fetch_ is a tool to download and verify remote archives and files. |
|
9 |
The build system uses it to download all archive packages. It takes care of download security testing, |
|
10 |
gpg verification, checksum verification, partial downloads/continuation, and storing hash files. |
|
11 |
|
|
12 |
|
|
13 |
### Notes |
|
14 |
|
|
15 |
* Use `--hash <algorithm>:<checksum>` to skip `HASH_DIR` checking for a download. |
|
16 |
* Use `--hash none` to skip all hash checking for a download. |
|
17 |
* Use `--sigurl <URL>` to set the exact GPG signature to check. |
|
18 |
This signature will be saved locally for future use. |
|
19 |
* Use `--sigurl none` to skip all signature checking for a download. |
|
20 |
* Use `--keep` to stop _userland-fetch_ from deleting any files. |
|
21 |
* Use `-nN` to force checking of both signature and hash. |
|
22 |
* Use `--search` to add a URL to the list to query for files, signatures and hashes. |
|
23 |
Can be used more than once. |
|
24 |
|
|
25 |
##### protocols |
|
26 |
_userland-fetch_ decides whether a download protocol is secure by checking the `SECURE_PROTOCOLS` envvar. |
|
27 |
This can be set to a space-delimited list of schemes. (URLs are in the form `<scheme>://<resource>`) |
|
28 |
**Override with caution!** |
|
29 |
|
|
30 |
##### verification |
|
31 |
_userland-fetch_ uses the following rules to determine the validity of a download: |
|
32 |
* confirm the file has a good signature, using each extension in `SIGNATURE_EXTENSIONS`, OR |
|
33 |
* confirm the file has a correct hash entry in a file in `HASH_DIR`, OR |
|
34 |
* check if `URL` scheme is in `SECURE_PROTOCOLS` and `ALLOW_UNVERIFIED_DOWNLOADS=yes`. |
|
35 |
|
|
36 |
Downloaded hash files are also checked for signatures and protocol security. |
|
37 |
|
|
38 |
##### signature extensions |
|
39 |
_userland-fetch_ searches for signature files with the extensions listed in the `SIGNATURE_EXTENSIONS` envvar. |
|
40 |
If you find a strange extension for a remote signature, you can add it to `SIGNATURE_EXTENSIONS`, but keep |
|
41 |
in mind that the file must be a detached GPG signature. |
|
42 |
|
|
43 |
##### hashfiles and hash directory |
|
44 |
_userland-fetch_ checks the envvar `HASH_DIR` to find the path to search for and store hashfiles. It will search |
|
45 |
each file in `HASH_DIR` for a hash entry containing the basename of `--file <path>`, or the basename of `--url <URL>`, stopping on the first match. Subsequent matches are ignored. Files are checked in `sorted()` order. |
|
46 |
|
|
47 |
_userland-fetch_ will strip directory information from the hashfiles when scanning them. The hashfiles themselves |
|
48 |
are not altered. (I.E. `<hash> a/b/file.txt` becomes `<hash> file.txt` internally, so it will match) |
|
49 |
|
|
50 |
In the event of an erroneous match, you can add a hashfile to the hashes directory manually, or |
|
51 |
have it download with the build in a Makefile target: |
|
52 |
``` |
|
53 |
$(HASH_DIR)/<filename>: |
|
54 |
$(FETCH) -N --url <URL> --file $@ |
|
55 |
download:: $(HASH_DIR)/<filename> |
|
56 |
``` |
|
57 |
|
|
58 |
### Defaults |
|
59 |
``` |
|
60 |
ALLOW_UNVERIFIED_DOWNLOADS=no |
|
61 |
SECURE_PROTOCOLS=https |
|
62 |
HASH_DIR=<file_arg directory>/hashes; (if --file <filename>) |
|
63 |
HASH_DIR=<current directory>/hashes ; (if no --file argument) |
|
64 |
SIGNATURE_EXTENSIONS=sig asc ; (space delimited, no '.' prefix) |
|
65 |
DEFAULT_HASH_FILES=SHA256SUMS sha256sums.txt sha256sum.txt (space delimited) |
|
66 |
DEFAULT_HASH_ALGO=sha256 (this is sent directly to python's hashlib.new() - only set one algorithm) |
|
67 |
``` |
|
68 |
|
|
69 |
### Use cases |
|
70 |
|
|
71 |
`userland-fetch --url <URL>` |
|
72 |
* Download the file at URL, placing it in the current directory. |
|
73 |
* Skip downloading if the output file already exists. |
|
74 |
* Verify the download or file using the method stated above. |
|
75 |
|
|
76 |
`userland-fetch --url <URL> --file <path>` |
|
77 |
* Same as above, but output the download to `path`. |
|
78 |
|
|
79 |
`userland-fetch -gn --url <URL>` |
|
80 |
* Download the file at `URL`, placing it in the current directory. |
|
81 |
* Look for these hash files in `URL`'s remote directory: |
|
82 |
* * Filenames in the `DEFAULT_HASH_FILES` envvar |
|
83 |
* * A few styles of `<filename>.DEFAULT_HASH_ALGO` |
|
84 |
* If a local copy of a hashfile exists, use it instead of downloading the remote one |
|
85 |
* Search hashfiles for an entry matching the filename; stop on first match. |
|
86 |
* Verify the download, ensuring that a correct hash is present. |
|
87 |
|
|
88 |
`userland-fetch -c --url <URL>` |
|
89 |
* Do -g, but replace any files in HASH_DIR with their remote versions. |
|
90 |
|
|
91 |
`userland-fetch -GN --url <URL>` |
|
92 |
* Download the file at URL, placing it in the current directory. |
|
93 |
* Look for these signature files in `URL`'s remote directory: |
|
94 |
* * `<filename>.ext` for each extension in `SIGNATURE_EXTENSIONS` |
|
95 |
* If a local copy of a signature exists, __overwrite it__ |
|
96 |
* Verify the file, ensuring that it has a good signature. |
|
97 |
|
|
98 |
`userland-fetch --file <path>` |
|
99 |
* Verify the file using the method stated above; protocol checking is skipped. |
|
100 |
|
|
101 |
`userland-fetch -n --file <path>` |
|
102 |
* Ensure a correct hash exists for `path` in HASH_DIR. |
|
103 |
|
|
104 |
`userland-fetch -c --file <path>` |
|
105 |
* Same as `userland-fetch -n --file <path>`. |
|
106 |
|
9c6c5a
|
107 |
## userland-zone |
AŠ |
108 |
|
|
109 |
_userland-zone_ is a tool to manage a lifecycle of build zones in oi-userland. |
|
110 |
The intended and main use case is the use in our continuous integration system and provides a clean build environment. |
|
111 |
It works in a way that it creates a template zone and all build zones are cloned from it. |
|
112 |
To make it easier for new joiners, _userland-zone_ assumes certain things and set some defaults: |
|
113 |
|
|
114 |
* /zones is a ZFS dataset |
|
115 |
|
|
116 |
Recommended way how to create _zones_ dataset: |
|
117 |
|
|
118 |
```shell script |
|
119 |
zfs create -o mountpoint=/zones rpool/zones |
|
120 |
``` |
|
121 |
|
|
122 |
* **/ws/archives** is present in the global zone and hosts downloaded userland source archives |
|
123 |
* **/ws/code** is present in the global zone and has a working copy of oi-userland repository |
|
124 |
* template zone is called _prbuilder-template_ and is never running |
|
125 |
* build zones are called _prbuilder-ID_ where ID is an identifier passed as an argument |
|
126 |
|
|
127 |
When working with _userland-zone_, use the following workflow: |
|
128 |
|
|
129 |
* **userland-zone create-template** - this creates a template zone, which is used as a golden image for other build zones. |
|
130 |
|
|
131 |
* **userland-zone spawn-zone --id 123** - this creates a build zone, _prbuilder-123_. Once the zone has been built, |
|
132 |
**/ws/archives** and **/ws/code** from the global zone will be mounted under the same location inside the build zone. |
|
133 |
|
|
134 |
* The build zone provides no networking, so source tarball will have to be downloaded in the global zone |
|
135 |
in via **gmake download**. |
|
136 |
|
|
137 |
* Once, the source tarball has been downloaded, the build inside the zone can happen via **zlogin prbuilder-123**. |
|
138 |
Inside the zone, execute **cd /ws/code/components/CATEGORY/COMPONENT** and run **gmake publish**. |
|
139 |
The build will proceed in the clean environment of the build zone as expected and the built package will be |
|
140 |
published to the local repository. |
|
141 |
|
|
142 |
* When the build finished or the build zone is not needed, it can be safely destroyed |
|
143 |
via **userland-zone destroy-zone --id 123**. |
|
144 |
|
|
145 |
* Before every build, it is recommend to update the template zone via **userland-zone update-template**. |
|
146 |
This is especially important in cases when compilers or libraries get updated and developers should always use the latest |
|
147 |
bits to build oi-userland components. |
|
148 |
|
|
149 |
* If you want to get rid of the template zone, delete it via **userland-zone delete-template**. |
|
150 |
|
|
151 |
|