--- /dev/null
+<html>
+ <head>
+ <meta http-equiv="content-type" content="text/html; charset=utf-8">
+ <title>TestKit Reference Manual</title>
+ </head>
+ <body vlink="purple" link="blue" lang="EN-US">
+ <h1 style=" text-align:center">$BRANDING TestKit Reference Manual</h1>
+ <h3 style=" text-align:center">Author: Chris Koeritz</h3>
+ <address style=" text-align:center">
+ Version 1.0 ― Updated August 14 2020</address>
+ <h1>The $BRANDING TestKit</h1>
+ <p>The TestKit is a collection of scripts that leverages the ShUnit unit
+ testing environment. TestKit provides a pattern for creating test
+ suites using a simple configuration file approach. Full reporting on
+ test runs is provided in a convenient tabular format.</p>
+ <p>Generally, writing a test script using the TestKit is a matter of
+ minutes. A blank test is provided as a template, and that can be
+ expanded with whatever test steps are needed.</p>
+ <p>TestKit (and ShUnit) are implemented in the GNU Bash script language, but
+ a TestKit test script can invoke external applications, written in
+ whatever programming language or scripting tool is desired, using the
+ standard POSIX interfaces.<br>
+ </p>
+ <h2> Getting the TestKit</h2>
+ <p>Follow these steps to download and install a new "vanilla" version of the
+ TestKit:<br>
+ </p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3"><span style="font-family: monospace;">sudo
+ mkdir /opt/feistymeow.org<br>
+ sudo chown -R $USER /opt/feistymeow.org<br>
+ cd /opt/feistymeow.org<br>
+ git clone git://feistymeow.org/feisty_meow</span></p>
+ </div>
+ <p>This is the code used to create the $BRANDING version of the TestKit.</p>
+ <p>It is possible to check out the TestKit within one's own code base, and
+ then it is possible to retrieve an updated Feisty Meow® TestKit by running
+ "git pull" on the "testkit" folder. This will get the latest version
+ from the Feisty Meow® Codebase without disturbing whatever project's
+ revision control repository contains the TestKit for testing.</p>
+ <h3>Preparing the TestKit on Linux</h3>
+ <p>Linux is the easiest environment for running the TestKit, given that the
+ tests were built using the bash shell within a Linux environment. If
+ some of the packages used in the tests are missing (such as expect and gnu
+ awk), these may need to be installed from the appropriate repository for
+ your Linux distribution. Most distributions include these packages
+ automatically however.</p>
+ <h3> Preparing the TestKit on Mac OS X</h3>
+ <p>The test suite runs well on modern Macs with Intel CPUs. Due to
+ some differences in the versions of a few applications on the Mac, some
+ GNU tools may need to be installed to run the TestKit. These are
+ available via the Brew installer tool. <br>
+ </p>
+ <h3> </h3>
+ <h3> Preparing the TestKit on MS-Windows</h3>
+ <p>The Cygwin Unix emulation system is required to run the TestKit on
+ Windows. This package is available at: <a href="http://cygwin.com/install.html">http://cygwin.com/install.html</a></p>
+ <p>The default packages selected by Cygwin are the starting point of the
+ install. In addition to those packages, the following packages are
+ also required (see list below). Rather than using the cygwin setup
+ program for this task, the next section describes how to install Cygwin
+ with the apt-cyg tool. Apt-cyg is the preferred method, since it
+ involves less interaction with the somewhat clunky Cygwin installer.
+ If necessary, it is possible to install all the packages without apt-cyg
+ just by using the Cygwin setup program. To find each of these
+ packages more easily, try switching the “View” button on the Cygwin setup
+ program to “Full” to get an alphabetized list.</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
+margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3">bc <br>
+ crypt <br>
+ cygutils <br>
+ emacs <br>
+ email <br>
+ expect <br>
+ gcc-g++<br>
+ git <br>
+ gitk <br>
+ gvim <br>
+ inetutils <br>
+ less <br>
+ make <br>
+ mutt <br>
+ ncftp <br>
+ openssh <br>
+ perl <br>
+ procps<br>
+ python <br>
+ sharutils <br>
+ shutdown <br>
+ subversion <br>
+ time <br>
+ unzip <br>
+ util-linux <br>
+ vim<br>
+ wget<br>
+ xinit <br>
+ xterm <br>
+ zip </p>
+ </div>
+ <p> </p>
+ <h3>Apt-cyg Installation Process</h3>
+ <p>The apt-cyg program brings the convenience of the Debian and Ubuntu
+ installer application (apt-get) to Cygwin. This program does require
+ a couple of additional setup steps. This material is drawn from the
+ apt-cyg home page: <a href="https://github.com/transcode-open/apt-cyg">https://github.com/transcode-open/apt-cyg</a></p>
+ <p class="MsoListParagraphCxSpFirst" style="text-indent:-.25in">1.
+ Install the basic Cygwin packages with setup.exe (rather than the long
+ list above), but add these two packages which are not selected by default:</p>
+ <ul>
+ <li>subversion</li>
+ <li>wget</li>
+ </ul>
+ <p class="MsoListParagraphCxSpLast" style="text-indent:-.25in">2.
+ Download and install the apt-cyg program from within a Cygwin bash prompt:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.5in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3">lynx -source
+ rawgit.com/transcode-open/apt-cyg/master/apt-cyg > apt-cyg<br>
+ install apt-cyg /bin</p>
+ </div>
+ <p class="MsoListParagraph" style="text-indent:-.25in">3.
+ Install the packages required for the TestKit:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.5in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:0in;
+margin-left:0in;margin-bottom:.0001pt;background:#DDD9C3">apt-cyg install bc
+ crypt cygutils emacs email expect gcc-g++ git gitk gvim \<br>
+ inetutils less make mutt ncftp openssh perl procps python
+ sharutils \<br>
+ shutdown time unzip util-linux vim xinit xterm zip</p>
+ </div>
+ <p class="MsoListParagraph" style="text-indent:-.25in">4.
+ The installation will run for a while but then should conclude with all
+ required packages installed.</p>
+ <h2> Setting up a Test Suite</h2>
+ <p>Tunning tests in TestKit uses a configuration file called
+ “testkit.config” to define the environment and, optionally, which test
+ scripts to run. This file is the main switchboard that defines where
+ the tests will find users, home directories, queues, containers, and so
+ forth. The configuration file can be specified via the environment
+ variable “TESTKIT_CFG_FILE”. This variable can be set to any
+ location, enabling the configuration file to reside in a directory other
+ than the toolkit folder. If the variable is not defined, then the
+ testing config file defaults to “$TESTKIT_ROOT/testkit.config”.</p>
+ The TESTKIT_ROOT variable is frequently referred to in command
+ examples. It is set up automatically by the prepare_tools script (see
+ below).
+ <h2>Running a Test Suite</h2>
+ <p>Once the TestKit configuration file has been established, running a whole
+ test suite can be accomplished with this command:<br>
+ </p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3"> bash <i>{TESTKIT_FOLDER}</i>/test_driver.sh
+ </p> </div>
+ <p>Where the <i>{TESTKIT_FOLDER}</i> should be replaced with whatever path
+ the TestKit is stored in.</p>
+ <p>Alternatively, if the TESTKIT_ROOT folder is already established, the
+ tests can be run with:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3"> bash "$TESTKIT_ROOT/test_driver.sh"</p>
+ </div>
+ <p></p>
+ <h3> What to Expect From the Test Run</h3>
+ <p>The test_driver.sh script will output a few informative lines of text
+ before printing a table of the tests that it intends to run.</p>
+ <p>After the test plan is shown, all of the tests listed will be executed in
+ the order they are listed in, and they will each produce output.
+ Each individual test (usually a separate bash script) produces a summary
+ at the end of its run with a count of tests and a report of the tests
+ success or failure.</p>
+ <p>At the end of all the tests in the suite, the table of tests is printed
+ again with the results for each test. For example, this is a test
+ run that had no errors in any test (that's good, since it is our super
+ simple example test):</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">$ cd
+ $FEISTY_MEOW_APEX/testkit<br>
+ $ ./test_driver.sh summary<br>
+ ===========================================================<br>
+ Testkit environment loaded.<br>
+ TESTKIT_ROOT=/opt/feistymeow.org/feisty_meow/testkit<br>
+ TESTKIT_CFG_FILE=/opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
+ TMP=/Users/fred/.tmp<br>
+ TEST_TEMP=/Users/fred/.tmp/testkit_logs_fred<br>
+ ===========================================================<br>
+ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
+ TestKit running from: /opt/feistymeow.org/feisty_meow/testkit<br>
+ TestKit config file:
+ /opt/feistymeow.org/feisty_meow/testkit/testkit.config<br>
+ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br>
+ Full set of tests:<br>
+ 1: /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
+ <br>
+ ======================================================================<br>
+ Wed Aug 12 14:11:00 EDT 2020: Now running test 1:
+ /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
+ Test output file:
+ /Users/fred/.tmp/testkit_logs_fred/run_2020_08_12/test_log.vKf7J3<br>
+ OK: successful test run for test
+ /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
+ <br>
+ <br>
+ Results table for this test run:<br>
+ <br>
+ 01: OKAY -- /opt/feistymeow.org/feisty_meow/testkit/examples/blank_test.sh<br>
+ <br>
+ Total testing duration: 00:00 hh:mm (1 seconds total)<br>
+ OK: All 1 Tests Ran Successfully.<br>
+ </div>
+ <p class="Textbody">The above shows the "summary" view, which does not allow
+ the individual tests to output to the console. If the "summary" flag
+ is not passed, then the output from all the test runs is also shown.</p>
+ <p class="Textbody">Even when the summary view is used, all output files can
+ be examined after the run. For example, in the above, the mentioned
+ output file "test_log.vKf7J3" can be checked to see exactly what happened
+ during the test.</p>
+ <p class="Textbody">A test with a failure in it will have “FAIL” next to the
+ test that failed, and the final output line will start with
+ “FAILURE”. For example:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3">01: OKAY –
+ AckPfft_Tests/Gorp_Tests/deslagToaster.sh<br>
+ 02: FAIL – AckPfft_Tests/Gorp_Tests/spumeMerchantry.sh<br>
+ 03: OKAY – AckPfft_Tests/Gorp_Tests/octopusLauncher.sh<br>
+ …<br>
+ 22: OKAY -- Snargle_Tests/scramTests/scramForPetunias.sh</p>
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3">FAILURE: 1 Tests Failed out of 22 Tests.</p>
+ </div>
+ <p>A failed test will also return a non-zero value from the test execution,
+ enabling the run of a test suite to be tested for success when launched
+ externally, such as from a continuous integration system.</p>
+ <h2>Loading the TestKit Environment</h2>
+ <p>If one wishes to run individual tests within the test suite, rather than
+ the entire suite, this is done by loading the TestKit variables into the
+ current shell environment, like so:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;
+background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3">cd <i>{TESTKIT_FOLDER}</i> # replace
+ with actual location of TestKit.<br>
+ source prepare_tools.sh prepare_tools.sh<br>
+ source $TESTKIT_ROOT/library/process_configuration.sh<br>
+ define_and_export_variables</p>
+ # Show the important variables.<br>
+ var $TESTKIT_ROOT $TESTKIT_CFG_FILE</div>
+ <p>After loading the TestKit environment, one can execute a specific test
+ and see its results, for example:</p>
+ <div style="border:solid windowtext 1.0pt;padding:1.0pt 4.0pt 1.0pt 4.0pt;background:#DDD9C3;margin-left:.1in;margin-right:.1in">
+ <p class="Code-Box" style="margin-top:6.0pt;margin-right:0in;margin-bottom:6.0pt;
+margin-left:0in;background:#DDD9C3">cd examples<br>
+ bash blank_test.sh</p>
+ </div>
+ <p>The test will run and output its results to the console (that is, output
+ is sent to standard out and standard error, to be more precise).</p>
+ <p><br>
+ </p>
+ <h3></h3>
+ </body>
+</html>
projects / feisty_meow.git / commitdiff