3 <meta http-equiv="content-type" content="text/html; charset=utf-8">
4 <title>TestKit Reference Manual</title>
6 <body vlink="purple" link="blue" lang="EN-US">
7 <h1 style=" text-align:center">Feisty Meow® TestKit Reference Manual</h1>
8 <address style=" text-align:center">Version 1.0 ― Updated August 14 2020</address>
9 <h1>The Feisty Meow® TestKit</h1>
10 <p>The TestKit is a collection of scripts that leverages the ShUnit unit
11 testing environment. TestKit provides a pattern for creating test
12 suites using a simple configuration file approach. Full reporting on
13 test runs is provided in a convenient tabular format.</p>
14 <p>Generally, writing a test script using the TestKit is a matter of
15 minutes. A blank test is provided as a template, and that can be
16 expanded with whatever test steps are needed.</p>
17 <p>TestKit (and ShUnit) are implemented in the GNU Bash script language, but
18 a TestKit test script can invoke external applications, written in
19 whatever programming language or scripting tool is desired, using the
20 standard POSIX interfaces.<br>
22 <h2> Getting the TestKit</h2>
23 <p>Follow these steps to download and install a new "vanilla" version of the
26 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
27 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
28 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
29 margin-left:0in;background:#DDD9C3"><span style="font-family: monospace;">sudo
30 mkdir /opt/feistymeow.org<br>
31 sudo chown -R $USER /opt/feistymeow.org<br>
32 cd /opt/feistymeow.org<br>
33 git clone git://feistymeow.org/feisty_meow</span></p>
35 <p>This is the code used to create the Feisty Meow® version of the TestKit.</p>
36 <p>It is possible to check out the TestKit within one's own code base, and
37 then it is possible to retrieve an updated Feisty Meow® TestKit by running
38 "git pull" on the "testkit" folder. This will get the latest version
39 from the Feisty Meow® Codebase without disturbing whatever project's
40 revision control repository contains the TestKit for testing.</p>
41 <h3>Preparing the TestKit on Linux</h3>
42 <p>Linux is the easiest environment for running the TestKit, given that the
43 tests were built using the bash shell within a Linux environment. If
44 some of the packages used in the tests are missing (such as expect and gnu
45 awk), these may need to be installed from the appropriate repository for
46 your Linux distribution. Most distributions include these packages
47 automatically however.</p>
48 <h3> Preparing the TestKit on Mac OS X</h3>
49 <p>The test suite runs well on modern Macs with Intel CPUs. Due to
50 some differences in the versions of a few applications on the Mac, some
51 GNU tools may need to be installed to run the TestKit. These are
52 available via the Brew installer tool. <br>
55 <h3> Preparing the TestKit on MS-Windows</h3>
56 <p>The Cygwin Unix emulation system is required to run the TestKit on
57 Windows. This package is available at: <a href="http://cygwin.com/install.html">http://cygwin.com/install.html</a></p>
58 <p>The default packages selected by Cygwin are the starting point of the
59 install. In addition to those packages, the following packages are
60 also required (see list below). Rather than using the cygwin setup
61 program for this task, the next section describes how to install Cygwin
62 with the apt-cyg tool. Apt-cyg is the preferred method, since it
63 involves less interaction with the somewhat clunky Cygwin installer.
64 If necessary, it is possible to install all the packages without apt-cyg
65 just by using the Cygwin setup program. To find each of these
66 packages more easily, try switching the “View” button on the Cygwin setup
67 program to “Full” to get an alphabetized list.</p>
68 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
69 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
70 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
71 margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3">bc <br>
103 <h3>Apt-cyg Installation Process</h3>
104 <p>The apt-cyg program brings the convenience of the Debian and Ubuntu
105 installer application (apt-get) to Cygwin. This program does require
106 a couple of additional setup steps. This material is drawn from the
107 apt-cyg home page: <a href="https://github.com/transcode-open/apt-cyg">https://github.com/transcode-open/apt-cyg</a></p>
108 <p class="MsoListParagraphCxSpFirst" style="text-indent:-.25in">1.
109 Install the basic Cygwin packages with setup.exe (rather than the long
110 list above), but add these two packages which are not selected by default:</p>
115 <p class="MsoListParagraphCxSpLast" style="text-indent:-.25in">2.
116 Download and install the apt-cyg program from within a Cygwin bash prompt:</p>
117 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
118 background:#DDD9C3;margin-left:.5in;margin-right:.1in">
119 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
120 margin-left:0in;background:#DDD9C3">lynx -source
121 rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg<br>
122 install apt-cyg /bin</p>
124 <p class="MsoListParagraph" style="text-indent:-.25in">3.
125 Install the packages required for the TestKit:</p>
126 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
127 background:#DDD9C3;margin-left:.5in;margin-right:.1in">
128 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
129 margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3">apt-cyg install bc
130 crypt cygutils emacs email expect gcc-g++ git gitk gvim \<br>
131 inetutils less make mutt ncftp openssh perl procps python
133 shutdown time unzip util-linux vim xinit xterm zip</p>
135 <p class="MsoListParagraph" style="text-indent:-.25in">4.
136 The installation will run for a while but then should conclude with all
137 required packages installed.</p>
138 <h2> Setting up a Test Suite</h2>
139 <p>Tunning tests in TestKit uses a configuration file called
140 “testkit.config” to define the environment and, optionally, which test
141 scripts to run. This file is the main switchboard that defines where
142 the tests will find users, home directories, queues, containers, and so
143 forth. The configuration file can be specified via the environment
144 variable “TESTKIT_CFG_FILE”. This variable can be set to any
145 location, enabling the configuration file to reside in a directory other
146 than the toolkit folder. If the variable is not defined, then the
147 testing config file defaults to “$TESTKIT_ROOT/testkit.config”.</p>
148 The TESTKIT_ROOT variable is frequently referred to in command
149 examples. It is set up automatically by the prepare_tools script (see
151 <h2>Running a Test Suite</h2>
152 <p>Once the TestKit configuration file has been established, running a whole
153 test suite can be accomplished with this command:<br>
155 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
156 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
157 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
158 margin-left:0in;background:#DDD9C3"> bash <i>{TESTKIT_FOLDER}</i>/test_driver.sh
160 <p>Where the <i>{TESTKIT_FOLDER}</i> should be replaced with whatever path
161 the TestKit is stored in.</p>
162 <p>Alternatively, if the TESTKIT_ROOT folder is already established, the
163 tests can be run with:</p>
164 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
165 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
166 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
167 margin-left:0in;background:#DDD9C3"> bash "$TESTKIT_ROOT/test_driver.sh"</p>
170 <h3> What to Expect From the Test Run</h3>
171 <p>The test_driver.sh script will output a few informative lines of text
172 before printing a table of the tests that it intends to run.</p>
173 <p>After the test plan is shown, all of the tests listed will be executed in
174 the order they are listed in, and they will each produce output.
175 Each individual test (usually a separate bash script) produces a summary
176 at the end of its run with a count of tests and a report of the tests
177 success or failure.</p>
178 <p>At the end of all the tests in the suite, the table of tests is printed
179 again with the results for each test. For example, this is a test
180 run that had no errors in any test (that's good, since it is our super
181 simple example test):</p>
182 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
183 background:#DDD9C3;margin-left:.1in;margin-right:.1in">$ cd
184 $FEISTY_MEOW_APEX/testkit<br>
185 $ ./test_driver.sh summary<br>
186 ===========================================================<br>
187 Testkit environment loaded.<br>
188 TESTKIT_ROOT=/opt/feistymeow.org/feisty_meow/testkit<br>
189 TESTKIT_CFG_FILE=/opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
190 TMP=/Users/fred/.tmp<br>
191 TEST_TEMP=/Users/fred/.tmp/testkit_logs_fred<br>
192 ===========================================================<br>
193 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
194 TestKit running from: /opt/feistymeow.org/feisty_meow/testkit<br>
196 /opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
197 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
198 Full set of tests:<br>
199 1: /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
201 ======================================================================<br>
202 Wed Aug 12 14:11:00 EDT 2020: Now running test 1:
203 /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
204 Test output file:
205 /Users/fred/.tmp/testkit_logs_fred/run_2020_08_12/test_log.vKf7J3<br>
206 OK: successful test run for test
207 /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
210 Results table for this test run:<br>
212 01: OKAY -- /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
214 Total testing duration: 00:00 hh:mm (1 seconds total)<br>
215 OK: All 1 Tests Ran Successfully.<br>
217 <p class="Textbody">The above shows the "summary" view, which does not allow
218 the individual tests to output to the console. If the "summary" flag
219 is not passed, then the output from all the test runs is also shown.</p>
220 <p class="Textbody">Even when the summary view is used, all output files can
221 be examined after the run. For example, in the above, the mentioned
222 output file "test_log.vKf7J3" can be checked to see exactly what happened
224 <p class="Textbody">A test with a failure in it will have “FAIL” next to the
225 test that failed, and the final output line will start with
226 “FAILURE”. For example:</p>
227 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
228 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
229 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
230 margin-left:0in;background:#DDD9C3">01: OKAY –
231 AckPfft_Tests/Gorp_Tests/deslagToaster.sh<br>
232 02: FAIL – AckPfft_Tests/Gorp_Tests/spumeMerchantry.sh<br>
233 03: OKAY – AckPfft_Tests/Gorp_Tests/octopusLauncher.sh<br>
235 22: OKAY -- Snargle_Tests/scramTests/scramForPetunias.sh</p>
236 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
237 margin-left:0in;background:#DDD9C3">FAILURE: 1 Tests Failed out of 22 Tests.</p>
239 <p>A failed test will also return a non-zero value from the test execution,
240 enabling the run of a test suite to be tested for success when launched
241 externally, such as from a continuous integration system.</p>
242 <h2>Loading the TestKit Environment</h2>
243 <p>If one wishes to run individual tests within the test suite, rather than
244 the entire suite, this is done by loading the TestKit variables into the
245 current shell environment, like so:</p>
246 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
247 background:#DDD9C3;margin-left:.1in;margin-right:.1in">
248 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
249 margin-left:0in;background:#DDD9C3">cd <i>{TESTKIT_FOLDER}</i> # replace
250 with actual location of TestKit.<br>
251 source prepare_tools.sh prepare_tools.sh<br>
252 source $TESTKIT_ROOT/library/process_configuration.sh<br>
253 define_and_export_variables</p>
254 # Show the important variables.<br>
255 var $TESTKIT_ROOT $TESTKIT_CFG_FILE</div>
256 <p>After loading the TestKit environment, one can execute a specific test
257 and see its results, for example:</p>
258 <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;background:#DDD9C3;margin-left:.1in;margin-right:.1in">
259 <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
260 margin-left:0in;background:#DDD9C3">cd examples<br>
261 bash blank_test.sh</p>
263 <p>The test will run and output its results to the console (that is, output
264 is sent to standard out and standard error, to be more precise).</p>