3 <meta http-equiv="content-type" content="text/html; charset=utf-8">
4 <title>TestKit Reference Manual</title>
6 <body vlink="purple" link="blue" lang="EN-US">
7 <h1 style=" text-align:center">$BRANDING TestKit Reference Manual</h1>
8 <h3 style=" text-align:center">Author: Chris Koeritz</h3>
9 <address style=" text-align:center"> Version 1.0 ― Updated August 14 2020</address>
10 <h1>The $BRANDING TestKit</h1>
11 <p>The TestKit is a collection of scripts that leverages the ShUnit unit
12 testing environment. The TestKit provides a pattern for creating
13 test suites using a simple configuration file approach. Full
14 reporting on test runs is provided in a convenient tabular format.</p>
15 <p>Generally, writing a test script using the TestKit is a matter of
16 minutes. A blank test is provided as a template, and that can be
17 expanded with whatever test steps are needed.</p>
18 <p>TestKit (and ShUnit) are implemented in the GNU Bash script language, but
19 a TestKit test script can invoke external applications, written in
20 whatever programming language or scripting tool is desired, using the
21 standard POSIX interfaces.<br>
23 <h2> Getting the TestKit</h2>
24 <p>Follow these steps to download and install a new "vanilla" version of the
27 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
28 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
29 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
30 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">sudo
31 mkdir /opt/feistymeow.org<br>
32 sudo chown -R $USER /opt/feistymeow.org<br>
33 cd /opt/feistymeow.org<br>
34 git clone git://feistymeow.org/feisty_meow<br>
35 ls feisty_meow/testkit # the testkit location; can be copied
36 elsewhere for use.</span><span style="font-family: monospace;"><br>
39 <p>The above steps were used to kick-start the local version of the TestKit.</p>
40 <p>It is possible to check out the TestKit within one's own code base (by
41 adding the Feisty Meow® Codebase that was retrieved above). Then one
42 can retrieve an updated Feisty Meow® TestKit by running "git pull" on the
43 "feisty_meow" folder. This will get the latest version of TestKit
44 without disturbing whatever project's revision control repository contains
45 the TestKit for testing.</p>
46 <h3>Preparing the TestKit on Linux</h3>
47 <p>Linux is the easiest environment for running the TestKit, given that the
48 tests were built using the bash shell within a Linux environment. If
49 some of the packages used in the tests are missing (such as expect and gnu
50 awk), these may need to be installed from the appropriate repository for
51 your Linux distribution. Most distributions include these packages
52 automatically however.</p>
53 <h3> Preparing the TestKit on Mac OS X</h3>
54 <p>The test suite runs well on modern Macs with Intel CPUs. Due to
55 some differences in the versions of a few applications on the Mac, some
56 GNU tools may need to be installed to run the TestKit. These are
57 available via the Brew installer tool. <br>
60 <h3> Preparing the TestKit on MS-Windows</h3>
61 <p>The Cygwin Unix emulation system is required to run the TestKit on
62 Windows. This package is available at: <a href="http://cygwin.com/install.html">http://cygwin.com/install.html</a></p>
63 <p>The default packages selected by Cygwin are the starting point of the
64 install. In addition to those packages, the following packages are
65 also required (see list below). Rather than using the cygwin setup
66 program for this task, the next section describes how to install Cygwin
67 with the apt-cyg tool. Apt-cyg is the preferred method, since it
68 involves less interaction with the somewhat clunky Cygwin installer.
69 If necessary, it is possible to install all the packages without apt-cyg
70 just by using the Cygwin setup program. To find each of these
71 packages more easily, try switching the “View” button on the Cygwin setup
72 program to “Full” to get an alphabetized list.</p>
73 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
74 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
75 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
76 margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">bc
107 <h3>Apt-cyg Installation Process</h3>
108 <p>The apt-cyg program brings the convenience of the Debian and Ubuntu
109 installer application (apt-get) to Cygwin. This program does require
110 a couple of additional setup steps. This material is drawn from the
111 apt-cyg home page: <a href="https://github.com/transcode-open/apt-cyg">https://github.com/transcode-open/apt-cyg</a></p>
112 <p>1. Install the basic Cygwin packages with setup.exe (rather than
113 the long list above), but add these two packages which are not selected by
119 <p>2. Download and install the apt-cyg program from within a
120 Cygwin bash prompt:</p>
121 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
122 background:#DDD9C3;margin-left:.5in;margin-right:.1in">
123 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
124 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">lynx
125 -source rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg<br>
126 install apt-cyg /bin</span></p>
128 <p>3. Install the packages required for
130 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
131 background:#DDD9C3;margin-left:.5in;margin-right:.1in">
132 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
133 margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">apt-cyg
134 install bc crypt cygutils emacs email expect gcc-g++ git gitk gvim \<br>
135 inetutils less make mutt ncftp openssh perl procps python
137 shutdown time unzip util-linux vim xinit xterm zip</span></p>
139 <p>4. The installation will run for a
140 while but then should conclude with all required packages installed.</p>
141 <h2> Setting up a Test Suite</h2>
142 <p>Running tests in TestKit uses a configuration file called
143 “testkit.config” to define the environment and, optionally, which test
144 scripts to run. This file is the main switchboard that defines where
145 the tests will find the resources they require.</p>
146 <p>The configuration file can be specified via the environment variable
147 “TESTKIT_CFG_FILE”. This variable can be set to any location,
148 enabling the configuration file to reside in a directory other than the
149 toolkit folder. If the variable is not defined, then the testing
150 config file defaults to “$TESTKIT_ROOT/testkit.config”.</p>
151 The TESTKIT_ROOT variable is frequently referred to in command
152 examples. It is set up automatically by the prepare_tools script (see
154 <h2>Running a Test Suite</h2>
155 <p>Once the TestKit configuration file has been established, running a whole
156 test suite can be accomplished with this command:<br>
158 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
159 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
160 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
161 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;"> bash
162 <i>{TESTKIT_FOLDER}</i>/test_driver.sh </span></p>
164 <p>Where the <i>{TESTKIT_FOLDER}</i> should be replaced with whatever path
165 the TestKit is stored in.</p>
166 <p>Alternatively, if the TESTKIT_ROOT folder is already established, the
167 tests can be run with:</p>
168 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
169 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
170 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
171 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;"> bash
172 "$TESTKIT_ROOT/test_driver.sh"</span></p>
175 <h3> What to Expect From the Test Run</h3>
176 <p>The test_driver.sh script will output a few informative lines of text
177 before printing a table of the tests that it intends to run.</p>
178 <p>After the test plan is shown, all of the tests listed will be executed in
179 the order they are listed in, and they will each produce output.
180 Each individual test (usually a separate bash script) produces a summary
181 at the end of its run with a count of tests and a report of the tests
182 success or failure.</p>
183 <p>At the end of all the tests in the suite, the table of tests is printed
184 again with the results for each test. For example, this is a test
185 run that had no errors in any test (that's good, since it is our super
186 simple example test):</p>
187 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
188 background:#DDD9C3;margin-left:.1in;margin-right:.1in"><span style="font-family: Courier New,Courier,monospace;">$
189 cd $FEISTY_MEOW_APEX/testkit<br>
190 $ ./test_driver.sh summary<br>
191 ===========================================================<br>
192 Testkit environment loaded.<br>
193 TESTKIT_ROOT=/opt/feistymeow.org/feisty_meow/testkit<br>
194 TESTKIT_CFG_FILE=/opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
195 TMP=/Users/fred/.tmp<br>
196 TEST_TEMP=/Users/fred/.tmp/testkit_logs_fred<br>
197 ===========================================================<br>
198 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
199 TestKit running from: /opt/feistymeow.org/feisty_meow/testkit<br>
201 /opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
202 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
203 Full set of tests:<br>
204 1: /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
206 ======================================================================<br>
207 Wed Aug 12 14:11:00 EDT 2020: Now running test 1:
208 /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
209 Test output file:
210 /Users/fred/.tmp/testkit_logs_fred/run_2020_08_12/test_log.vKf7J3<br>
211 OK: successful test run for test
212 /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
215 Results table for this test run:<br>
218 /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
220 Total testing duration: 00:00 hh:mm (1 seconds total)<br>
221 OK: All 1 Tests Ran Successfully.</span><br>
223 <p class="Textbody">The above shows the "summary" view, which does not allow
224 the individual tests to output to the console. If the "summary" flag
225 is not passed, then the output from all the test runs is also shown.</p>
226 <p class="Textbody">Even when the summary view is used, all output files can
227 be examined after the run. For example, in the above, the mentioned
228 output file "test_log.vKf7J3" can be checked to see exactly what happened
230 <p class="Textbody">A test with a failure in it will have “FAIL” next to the
231 test that failed, and the final output line will start with
232 “FAILURE”. For example:</p>
233 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
234 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
235 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
236 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">01:
237 OKAY – AckPfft_Tests/Gorp_Tests/deslagToaster.sh<br>
238 02: FAIL – AckPfft_Tests/Gorp_Tests/spumeMerchantry.sh<br>
239 03: OKAY – AckPfft_Tests/Gorp_Tests/octopusLauncher.sh<br>
241 22: OKAY -- Snargle_Tests/scramTests/scramForPetunias.sh</span></p>
242 <span style="font-family: Courier New,Courier,monospace;"> </span>
243 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
244 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">FAILURE:
245 1 Tests Failed out of 22 Tests.</span></p>
247 <p>A failed test will also return a non-zero value from the test execution,
248 enabling the run of a test suite to be tested for success when launched
249 externally, such as from a continuous integration system.</p>
250 <h2>Loading the TestKit Environment</h2>
251 <p>If one wishes to run individual tests within the test suite, rather than
252 the entire suite, this is done by loading the TestKit variables into the
253 current shell environment, like so:</p>
254 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
255 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
256 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
257 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">cd
258 <i>{TESTKIT_FOLDER}</i> # replace with actual location of
260 source prepare_tools.sh prepare_tools.sh<br>
261 source $TESTKIT_ROOT/library/process_configuration.sh<br>
262 define_and_export_variables</span></p>
263 <span style="font-family: Courier New,Courier,monospace;"> # Show the
264 important variables.<br>
265 var $TESTKIT_ROOT $TESTKIT_CFG_FILE</span></div>
266 <p>After loading the TestKit environment, one can execute a specific test
267 and see its results, for example:</p>
268 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;background:#DDD9C3;margin-left:.1in;margin-right:.1in">
269 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
270 margin-left:0in;background:#DDD9C3"><span style="font-family: Courier New,Courier,monospace;">cd
272 bash blank_test.sh</span></p>
274 <p>The test will run and output its results to the console (that is, output
275 is sent to standard out and standard error, to be more precise).</p>