| commit | author | age | ||
| 222cf2 | 1 | |
| 9c6c5a | 2 | # userland-tools |
| AŠ | 3 | |
| 222cf2 | 4 | oi-userland comes with some tools used all over the build system and |
| 9c6c5a | 5 | these tools reside in the [tools/](https://github.com/OpenIndiana/oi-userland/tree/oi/hipster/tools) directory. |
| AŠ | 6 | |
| 222cf2 | 7 | ## userland-fetch |
| D | 8 | _userland-fetch_ is a tool to download and verify remote archives and files. |
| 9 | The build system uses it to download all archive packages. It takes care of download security testing, | |
| 10 | gpg verification, checksum verification, partial downloads/continuation, and storing hash files. | |
| 11 | ||
| 12 | ||
| 13 | ### Notes | |
| 14 | ||
| 15 | * Use `--hash <algorithm>:<checksum>` to skip `HASH_DIR` checking for a download. | |
| 16 | * Use `--hash none` to skip all hash checking for a download. | |
| 17 | * Use `--sigurl <URL>` to set the exact GPG signature to check. | |
| 18 | This signature will be saved locally for future use. | |
| 19 | * Use `--sigurl none` to skip all signature checking for a download. | |
| 20 | * Use `--keep` to stop _userland-fetch_ from deleting any files. | |
| 21 | * Use `-nN` to force checking of both signature and hash. | |
| 22 | * Use `--search` to add a URL to the list to query for files, signatures and hashes. | |
| 23 | Can be used more than once. | |
| 24 | ||
| 25 | ##### protocols | |
| 26 | _userland-fetch_ decides whether a download protocol is secure by checking the `SECURE_PROTOCOLS` envvar. | |
| 27 | This can be set to a space-delimited list of schemes. (URLs are in the form `<scheme>://<resource>`) | |
| 28 | **Override with caution!** | |
| 29 | ||
| 30 | ##### verification | |
| 31 | _userland-fetch_ uses the following rules to determine the validity of a download: | |
| 32 | * confirm the file has a good signature, using each extension in `SIGNATURE_EXTENSIONS`, OR | |
| 33 | * confirm the file has a correct hash entry in a file in `HASH_DIR`, OR | |
| 34 | * check if `URL` scheme is in `SECURE_PROTOCOLS` and `ALLOW_UNVERIFIED_DOWNLOADS=yes`. | |
| 35 | ||
| 36 | Downloaded hash files are also checked for signatures and protocol security. | |
| 37 | ||
| 38 | ##### signature extensions | |
| 39 | _userland-fetch_ searches for signature files with the extensions listed in the `SIGNATURE_EXTENSIONS` envvar. | |
| 40 | If you find a strange extension for a remote signature, you can add it to `SIGNATURE_EXTENSIONS`, but keep | |
| 41 | in mind that the file must be a detached GPG signature. | |
| 42 | ||
| 43 | ##### hashfiles and hash directory | |
| 44 | _userland-fetch_ checks the envvar `HASH_DIR` to find the path to search for and store hashfiles. It will search | |
| 45 | each file in `HASH_DIR` for a hash entry containing the basename of `--file <path>`, or the basename of `--url <URL>`, stopping on the first match. Subsequent matches are ignored. Files are checked in `sorted()` order. | |
| 46 | ||
| 47 | _userland-fetch_ will strip directory information from the hashfiles when scanning them. The hashfiles themselves | |
| 48 | are not altered. (I.E. `<hash> a/b/file.txt` becomes `<hash> file.txt` internally, so it will match) | |
| 49 | ||
| 50 | In the event of an erroneous match, you can add a hashfile to the hashes directory manually, or | |
| 51 | have it download with the build in a Makefile target: | |
| 52 | ``` | |
| 53 | $(HASH_DIR)/<filename>: | |
| 54 | $(FETCH) -N --url <URL> --file $@ | |
| 55 | download:: $(HASH_DIR)/<filename> | |
| 56 | ``` | |
| 57 | ||
| 58 | ### Defaults | |
| 59 | ``` | |
| 60 | ALLOW_UNVERIFIED_DOWNLOADS=no | |
| 61 | SECURE_PROTOCOLS=https | |
| 62 | HASH_DIR=<file_arg directory>/hashes; (if --file <filename>) | |
| 63 | HASH_DIR=<current directory>/hashes ; (if no --file argument) | |
| 64 | SIGNATURE_EXTENSIONS=sig asc ; (space delimited, no '.' prefix) | |
| 65 | DEFAULT_HASH_FILES=SHA256SUMS sha256sums.txt sha256sum.txt (space delimited) | |
| 66 | DEFAULT_HASH_ALGO=sha256 (this is sent directly to python's hashlib.new() - only set one algorithm) | |
| 67 | ``` | |
| 68 | ||
| 69 | ### Use cases | |
| 70 | ||
| 71 | `userland-fetch --url <URL>` | |
| 72 | * Download the file at URL, placing it in the current directory. | |
| 73 | * Skip downloading if the output file already exists. | |
| 74 | * Verify the download or file using the method stated above. | |
| 75 | ||
| 76 | `userland-fetch --url <URL> --file <path>` | |
| 77 | * Same as above, but output the download to `path`. | |
| 78 | ||
| 79 | `userland-fetch -gn --url <URL>` | |
| 80 | * Download the file at `URL`, placing it in the current directory. | |
| 81 | * Look for these hash files in `URL`'s remote directory: | |
| 82 | * * Filenames in the `DEFAULT_HASH_FILES` envvar | |
| 83 | * * A few styles of `<filename>.DEFAULT_HASH_ALGO` | |
| 84 | * If a local copy of a hashfile exists, use it instead of downloading the remote one | |
| 85 | * Search hashfiles for an entry matching the filename; stop on first match. | |
| 86 | * Verify the download, ensuring that a correct hash is present. | |
| 87 | ||
| 88 | `userland-fetch -c --url <URL>` | |
| 89 | * Do -g, but replace any files in HASH_DIR with their remote versions. | |
| 90 | ||
| 91 | `userland-fetch -GN --url <URL>` | |
| 92 | * Download the file at URL, placing it in the current directory. | |
| 93 | * Look for these signature files in `URL`'s remote directory: | |
| 94 | * * `<filename>.ext` for each extension in `SIGNATURE_EXTENSIONS` | |
| 95 | * If a local copy of a signature exists, __overwrite it__ | |
| 96 | * Verify the file, ensuring that it has a good signature. | |
| 97 | ||
| 98 | `userland-fetch --file <path>` | |
| 99 | * Verify the file using the method stated above; protocol checking is skipped. | |
| 100 | ||
| 101 | `userland-fetch -n --file <path>` | |
| 102 | * Ensure a correct hash exists for `path` in HASH_DIR. | |
| 103 | ||
| 104 | `userland-fetch -c --file <path>` | |
| 105 | * Same as `userland-fetch -n --file <path>`. | |
| 106 | ||
| 9c6c5a | 107 | ## userland-zone |
| AŠ | 108 | |
| 109 | _userland-zone_ is a tool to manage a lifecycle of build zones in oi-userland. | |
| 110 | The intended and main use case is the use in our continuous integration system and provides a clean build environment. | |
| 111 | It works in a way that it creates a template zone and all build zones are cloned from it. | |
| 112 | To make it easier for new joiners, _userland-zone_ assumes certain things and set some defaults: | |
| 113 | ||
| 114 | * /zones is a ZFS dataset | |
| 115 | ||
| 116 | Recommended way how to create _zones_ dataset: | |
| 117 | ||
| 118 | ```shell script | |
| 119 | zfs create -o mountpoint=/zones rpool/zones | |
| 120 | ``` | |
| 121 | ||
| 122 | * **/ws/archives** is present in the global zone and hosts downloaded userland source archives | |
| 123 | * **/ws/code** is present in the global zone and has a working copy of oi-userland repository | |
| 124 | * template zone is called _prbuilder-template_ and is never running | |
| 125 | * build zones are called _prbuilder-ID_ where ID is an identifier passed as an argument | |
| 126 | ||
| 127 | When working with _userland-zone_, use the following workflow: | |
| 128 | ||
| 129 | * **userland-zone create-template** - this creates a template zone, which is used as a golden image for other build zones. | |
| 130 | ||
| 131 | * **userland-zone spawn-zone --id 123** - this creates a build zone, _prbuilder-123_. Once the zone has been built, | |
| 132 | **/ws/archives** and **/ws/code** from the global zone will be mounted under the same location inside the build zone. | |
| 133 | ||
| 134 | * The build zone provides no networking, so source tarball will have to be downloaded in the global zone | |
| 135 | in via **gmake download**. | |
| 136 | ||
| 137 | * Once, the source tarball has been downloaded, the build inside the zone can happen via **zlogin prbuilder-123**. | |
| 138 | Inside the zone, execute **cd /ws/code/components/CATEGORY/COMPONENT** and run **gmake publish**. | |
| 139 | The build will proceed in the clean environment of the build zone as expected and the built package will be | |
| 140 | published to the local repository. | |
| 141 | ||
| 142 | * When the build finished or the build zone is not needed, it can be safely destroyed | |
| 143 | via **userland-zone destroy-zone --id 123**. | |
| 144 | ||
| 145 | * Before every build, it is recommend to update the template zone via **userland-zone update-template**. | |
| 146 | This is especially important in cases when compilers or libraries get updated and developers should always use the latest | |
| 147 | bits to build oi-userland components. | |
| 148 | ||
| 149 | * If you want to get rid of the template zone, delete it via **userland-zone delete-template**. | |
| 150 | ||
| 151 | ||
