commit | author | age
|
bc9b99
|
1 |
Summary. |
AP |
2 |
-------- |
|
3 |
|
|
4 |
Testing of a userland component that provides tests is performed by hooking |
|
5 |
those tests up to the 'test' target and running 'gmake test'. This should |
|
6 |
generally pass, as failing tests may indicate things you have to fix, or |
|
7 |
upstream tests that aren't applicable or need modifications on OpenIndiana. |
|
8 |
|
|
9 |
'gmake test' is often run when a component is upgraded or otherwise |
|
10 |
intentionally changed, but sometimes it would be useful to rerun the tests |
|
11 |
after something else has changed (such as the system being upgraded, or a |
|
12 |
change in compilers; see the 'System Test' section below) and see if that |
|
13 |
has affected the tests. |
|
14 |
|
|
15 |
We do this by having a 'master test file' that contains the expected results, |
|
16 |
and having a compare target that runs the tests and compares them with the |
|
17 |
results. |
|
18 |
|
|
19 |
Note: because the initial test run and the current test run may have a |
|
20 |
different environment (different users, different locales, different machines, |
|
21 |
different compilers...) the results need to have all such output dependencies |
|
22 |
removed or abstracted. |
|
23 |
|
|
24 |
|
|
25 |
Setting up a master test file for a Userland component. |
|
26 |
------------------------------------------------------- |
|
27 |
|
|
28 |
When setting up a test-and-compare run for a new component, you will need to |
|
29 |
have master test file(s) that are identical for both the x86 and SPARC |
|
30 |
platforms. It is suggested that they should initially be created by doing |
|
31 |
something like the following in your x86 Userland workspace, then copied to |
|
32 |
the same locations in your SPARC workspace and retested there. |
|
33 |
|
|
34 |
When you run "gmake test", a check is made to see if there is a master |
21b154
|
35 |
file of test results. If there is, then a test-and-compare run is performed. |
bc9b99
|
36 |
If there isn't, then just a "normal" run of the test suite is performed. |
AP |
37 |
|
|
38 |
The name of the master test file (or files to be exact), will depend upon |
|
39 |
whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a |
|
40 |
Python or Perl component. |
|
41 |
|
|
42 |
The default master file name is defined in |
|
43 |
make-rules/shared-macros.mk and is: |
|
44 |
|
21b154
|
45 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master |
bc9b99
|
46 |
|
AP |
47 |
where COMPONENT_TEST_RESULTS_DIR is: |
|
48 |
|
|
49 |
COMPONENT_TEST_RESULTS_DIR = $(COMPONENT_DIR)/test |
|
50 |
|
21b154
|
51 |
so that means one single common master test results file will be used. |
MT |
52 |
|
|
53 |
If the component needs different test result file based on bitness (32-bit vs. |
|
54 |
64-bit) then you should simply set USE_COMMON_TEST_MASTER to "no" in the |
|
55 |
component's Makefile before shared-macros.mk is included. Then the master test |
|
56 |
result file names will be: |
bc9b99
|
57 |
|
AP |
58 |
32-bit: components/<component-name>/test/results-32.master |
|
59 |
|
|
60 |
64-bit: components/<component-name>/test/results-64.master |
|
61 |
|
|
62 |
both: components/<component-name>/test/results-32.master |
|
63 |
components/<component-name>/test/results-64.master |
|
64 |
|
21b154
|
65 |
For both Python and Perl the common master test result file |
MT |
66 |
$(COMPONENT_TEST_RESULTS_DIR)/results-all.master is used by default. |
|
67 |
|
|
68 |
If USE_COMMON_TEST_MASTER is set to "no" then for Python COMPONENT_TEST_MASTER |
|
69 |
is overridden in make-rules/setup.py.mk to be: |
bc9b99
|
70 |
|
AP |
71 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master |
|
72 |
|
|
73 |
so that means it's looking for one or more of: |
|
74 |
|
|
75 |
2.7: components/python/<component-name>/test/results-2.7-32.master |
|
76 |
2.7: components/python/<component-name>/test/results-2.7-64.master |
0be775
|
77 |
3.7: components/python/<component-name>/test/results-3.7-64.master |
MT |
78 |
3.9: components/python/<component-name>/test/results-3.9-64.master |
bc9b99
|
79 |
|
AP |
80 |
depending upon which versions of Python this component supports. |
|
81 |
|
21b154
|
82 |
Perl is similar, when USE_COMMON_TEST_MASTER is set to "no" the |
MT |
83 |
COMPONENT_TEST_MASTER is being overridden in make-rules/makemaker.mk to be: |
bc9b99
|
84 |
|
21b154
|
85 |
COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION).master |
bc9b99
|
86 |
|
AP |
87 |
so that means it's looking for one or more of: |
|
88 |
|
21b154
|
89 |
5.34: components/perl/<component>/test/results-5.34.master |
MT |
90 |
5.36: components/perl/<component>/test/results-5.36.master |
bc9b99
|
91 |
|
AP |
92 |
depending upon which versions of Perl this component supports. |
|
93 |
|
21b154
|
94 |
In order to do a test-and-compare run rather than just run the component |
bc9b99
|
95 |
test suite, initially just create an empty master test file (or files). |
AP |
96 |
|
|
97 |
For example, for elinks, which just has 64-bit tests, do: |
|
98 |
|
|
99 |
$ cd components/elinks |
|
100 |
$ mkdir test |
|
101 |
$ touch test/results-64.master |
|
102 |
$ gmake test |
|
103 |
|
|
104 |
At this point, you have a set of test results in |
|
105 |
components/elinks/build/test/$(MACH64)/test-64-results. |
|
106 |
|
|
107 |
Even better, there are a set of "global" regular expressions that are |
|
108 |
run on those test results to try to normalize them. The output from that |
|
109 |
is placed in components/elinks/build/test/$(MACH64)/results-64.snapshot |
|
110 |
|
|
111 |
You can now use the contents of that file as a first cut at the master results. |
|
112 |
|
|
113 |
$ cp build/test/$(MACH64)/results-64.snapshot test/results-64.master |
|
114 |
|
|
115 |
Now run the tests again. Note that you have to get back to a clean start |
|
116 |
just in case the test process tries to compile any code. |
|
117 |
|
|
118 |
$ gmake clean |
|
119 |
$ gmake test |
|
120 |
|
|
121 |
At this point, it will again compare the test results against the master(s), |
|
122 |
and if there are still differences, they will be placed in |
|
123 |
components/elinks/build/test/$(MACH64)/test-64-diffs |
|
124 |
|
|
125 |
Typically these differences will be for things like usernames, temporary |
|
126 |
filenames, timings etc. If you have some differences, then you are going |
|
127 |
to have to write special regexp expressions in your component Makefile to |
|
128 |
"normalize" them, and adjust the master test results file so that it |
|
129 |
matches what the normalized version looks like. |
|
130 |
|
|
131 |
Transform examples. |
|
132 |
------------------- |
|
133 |
|
|
134 |
1/ Handling temporary filenames. |
|
135 |
|
|
136 |
See the transform in the asciidoc Makefile: |
|
137 |
|
|
138 |
COMPONENT_TEST_TRANSFORMS += \ |
|
139 |
'-e "s|/tmp......|/tmpxxxxxx|g" ' |
|
140 |
|
|
141 |
for how to "normalize" any temporary filenames that appear in the test results. |
|
142 |
|
|
143 |
2/ Build related output. |
|
144 |
|
|
145 |
Items (such as paths to compilers) should be removed from master test files. |
|
146 |
For example, in testing the xz component, we get the following different |
|
147 |
output for compiling a .c file for 32-bit and 64-bit: |
|
148 |
|
|
149 |
x86 32-bit: |
|
150 |
|
|
151 |
/usr/gcc/6/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests -I.. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/common -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/liblzma/api -I../lib -D_REENTRANT -pthreads -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -m32 -O3 -MT test_check.o -MD -MP -MF .deps/test_check.Tpo -c -o test_check.o /export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests/test_check.c |
|
152 |
|
|
153 |
x86 64-bit: |
|
154 |
|
|
155 |
/usr/gcc/6/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests -I.. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/common -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/liblzma/api -I../lib -D_REENTRANT -pthreads -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -m64 -O3 -MT test_check.o -MD -MP -MF .deps/test_check.Tpo -c -o test_check.o /export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests/test_check.c |
|
156 |
|
|
157 |
These can all be normalized (so that a single master test file can be used), |
|
158 |
by adding the following line to COMPONENT_TEST_TRANSFORMS in the xz component |
|
159 |
Makefile: |
|
160 |
|
|
161 |
'-e "s|\(^$(CC)\).*|\\$$(CC)|" ' |
|
162 |
|
|
163 |
The equivalent compile line in the master test file is now set to just: |
|
164 |
|
|
165 |
$(CC) |
|
166 |
|
|
167 |
Note that we are using $(CC) here which will pick up the current C compiler. |
|
168 |
Should the C compiler or build options change in the future, then there will |
|
169 |
be no need to adjust the master test results file to match. It's all done |
|
170 |
automatically. |
|
171 |
|
|
172 |
It is recognized that an alternate approach to solving this problem is to |
|
173 |
create a transform to just remove lines like this from the master test files, |
|
174 |
but there is currently no automatic way to pick up the default compiler. |
|
175 |
|
|
176 |
---- |
|
177 |
|
|
178 |
There will be other examples as more components are adjusted to test |
|
179 |
against master results files. |
|
180 |
|
|
181 |
If a lot of people start generating the very same ones, then we can |
|
182 |
consider adding them to the "global" set of regexps in |
|
183 |
make-rules/shared-macros.mk which currently looks like: |
|
184 |
|
|
185 |
COMPONENT_TEST_TRANSFORMS = \ |
|
186 |
'-e "s|$(@D)|\\$$(@D)|g" ' \ |
|
187 |
'-e "s|$(PERL)|\\$$(PERL)|g" ' \ |
|
188 |
'-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" ' |
|
189 |
|
|
190 |
Note that in some cases, gsed is not powerful enough. E.g., if you need to |
|
191 |
do more than massage or remove certain lines, but only compare certain sections |
|
192 |
of the file, then you may need to set COMPONENT_TEST_TRANSFORMER to something |
|
193 |
other than $(GSED) such as $(NAWK). In this case, you will want to set |
|
194 |
COMPONENT_TEST_TRANSFORMS using "=" instead of "+=", as the default sed |
|
195 |
transforms will likely need to be tossed aside in favor of whatever arguments |
|
196 |
your favorite transformer might need. |
|
197 |
|
|
198 |
When your master test file(s) are in good shape, then you should "git add" |
|
199 |
them to your workspace. |