Testing for statistic.pl ------------------------ Satanjeev Banerjee bane0025@d.umn.edu 12th February, 2002 1. Introduction: ---------------- We have tested statistic.pl, a component of Bigram Statistics Package version 0.5. Following is a description of the aspects of statistic.pl that we have tested. We provide the scripts and files used for testing so that later versions of statistic.pl can be tested for backward compatibility. 2. Phases of Testing: --------------------- We have divided the testing into two main phases: Phase 1: Testing statistic.pl's response to erroneous conditions. Phase 2: Testing statistic.pl's behaviour under normal conditions. 2.1. Phase 1: Testing statistic.pl's response to erroneous conditions: ---------------------------------------------------------------------- The script for this phase is 'error-handling.sh'. To run the tests contained in this script, type "error-handling.sh" at the command prompt. This script performs several subtests: 2.1.1. Subtest 1: ----------------- This test checks the response of statistic.pl when an ngram in the input ngram file does not have the expected number of tokens. The expected number of tokens is equal to the value specified by the --ngram switch, or 2 if this switch is not enforced. This test is run with the file test-1.sub-1.cnt as the input ngram file. This file has trigrams in it, and so statistic.pl should be run with --ngram 3. Subtest a does not do so, and rather relies on the default '2'. This should result in the error in file test-1.sub-1-a.reqd. Note that statistic.pl always requires a library to run with. If we use a particular library here, future changes to that particular library may cause these tests to break down through no fault of statistic.pl. Hence we provide library test "test_1_sub_3_d.pm" for this subtest (and similar library files for the other subtests). This library always returns a 0 value. Subtest b checks the same thing using --ngram 4 for which too we should get an error. Finally subtest c checks for --ngram 3 for which there should be no error. 2.1.2. Subtest 2: ----------------- This test checks to see what happens when the number of frequency values in the input file does not match with the number expected. The number of frequency values expected are either equal to the frequency combinations set by switch --set_freq_combo, or equal to the default number that count.pl creates from the current size of ngrams. File test-1.sub-2.cnt has trigrams in it, but with only 4 frequency values as opposed to the possible 7. This file was created using test-1.sub-2.freq_combo.txt file with the --set_freq_combo switch. So statistic.pl should also be run with --set_freq_combo test-1.sub-2.freq_combo.txt. Subtest a checks what happens if we dont provide this file! The expected error message is in file test-1.sub-2-a.reqd. Subtest b runs it with the requisite frequency combination file, and so there should be no error for this run. 2.1.3. Subtest 3: ----------------- The statistic library being loaded by statistic.pl must implement two mandatory functions: initializeStatistic() and calculateStatistic(). This test checks what happens when these mandatory functions are not defined in the statistic library file. In subtest a we use statistical library file test_1_sub_3_a.pm which neither defines nor exports the symbols &calculateStatistic and &initializeStatistic. The error that statistic.pl should give is in file test-1.sub-3-a.reqd. In subtest b we use statistical library file test_1_sub_3_b.pm which defines initializeStatistic but not calculateStatistic, while in subtest c we use statistical library file test_1_sub_3_c.pm which defines calculateStatistic but not initializeStatistic. In both these cases, statistic.pl should give an error, as in files test-1.sub-3-b.reqd and test-1.sub-3-c.reqd respectively. Statistical library test-1.sub-3-d.pm defines both, and so we should not get any error on this file, as tested in subtest d. 2.1.4. Subtest 4: ----------------- This subtest checks statistic.pl's reaction when less than the required number of files are provided to it on the commandline. Subtest a checks statistic.pl's reaction when only a library file is provided while subtest b does the same for when only the library and the destination files are provided. 2.1.5. Subtest 5: ----------------- It is possible that the input ngram file does not have the main ngram frequency. This might be alright, except that if a --frequency cut-off is provided, it would be impossible to execute (since the ngram frequency is missing). In this situation, statistic.pl provides a warning that the frequency cut-off requested is being ignored. This subtest checks for this warning. 2.2. Phase 2: Testing statistic.pl's behaviour under normal operation: ---------------------------------------------------------------------- The script for this phase is 'normal-op.sh'. To run the tests contained in this script, type "normal-op.sh" at the command prompt. This script performs several subtests: 2.2.1. Subtest 1: ----------------- This subtest checks if set_freq_combo and get_freq_combo are working fine or not. File test-2.sub-1-a.cnt has trigrams in it but without all the frequency values. We will use switch --set_freq_combo to provide statistic.pl with the necessary frequency combination file (test-2.sub-1-a.freq_combo.txt) and then use --get_freq_combo to check if we get exactly the same file again. 2.2.2. Subtest 2: ----------------- In this subtest we observe the effect of using switch --frequency. In subtest a, we use --frequency 2 to cut off all the ngrams with frequency 1 while in subtest b, we use --frequency 3 to cut off everything but the top ngram (which has a frequency of 3). 2.2.3. Subtest 3: ----------------- This subtest tests the --rank switch. Subtest a runs statistic.pl with --rank 6 and subtest b with --rank 3. 2.2.4. Subtest 4: ----------------- This subtest checks the --precision switch. Subtest a runs statistic.pl with --precision 0 which implies that we dont want any places of decimal. Note that this is not an erroneous condition! Subtest b runs it with --precision 5 and subtest c with --precision 10. 2.2.5. Subtest 5: ----------------- In this subtest we check the effect of using the switch --score. Subtest a runs statistic.pl using a score cutoff of 0.8 and subtest b does the same with a cutoff of 1.2. 2.2.6. Subtest 6: ----------------- This subtest tests the --format switch. The expected output is in file test-2.sub-6.reqd. 2.2.7. Subtest 7: ----------------- This subtest tests the --extended switch. The input file, test-2.sub-7.cnt, has extended information (since it was created by count.pl with the switch --extended). Subtest a runs statistic.pl on this file with the --extended switch. The expected output, which should have all the extended information preserved, as well as some additional information, is in file test-2.sub-7-a.reqd. Subtest b runs statistic.pl on this file without the --extended switch. The expected output, which should have no extended information, is in file test-2.sub-7-b.reqd. 3. Evaluation of execution time of count.pl on big files: --------------------------------------------------------- The following experiments were conducted on machine csdev01 at the Univ of Minnesota, Duluth, Computer Science Department laboratory. scarp.2.cnt was obtained by running count.pl on the etext "The Scarlet Pimpernel" by Baroness Orczy obtained from the following url: ftp://ibiblio.org/pub/docs/books/gutenberg/etext93/scarp10.txt wc scarp.2.cnt: 45792 137374 1051531 1> time statistic.pl dice scarp.dice scarp.2.cnt 23.0u 0.0s 0:24 95% 0+0k 0+0io 0pf+0w 2> time statistic.pl ll scarp.ll scarp.2.cnt 35.0u 0.0s 0:36 96% 0+0k 0+0io 0pf+0w 3> time statistic.pl x2 scarp.x2 scarp.2.cnt 30.0u 0.0s 0:31 95% 0+0k 0+0io 0pf+0w 4> time statistic.pl leftFisher scarp.leftFisher scarp.2.cnt 71.0u 0.0s 1:12 98% 0+0k 0+0io 0pf+0w 5> time statistic.pl mi scarp.mi scarp.2.cnt 28.0u 0.0s 0:29 96% 0+0k 0+0io 0pf+0w 4. Conclusions: --------------- Program statistic.pl has been run with a variety of switches, and its response has been noted. These tests can be used to check for backward compatibility of newer versions.