Summary: In this laboratory, you will learn some valuable tools that can be used to automate the testing of C programs.
Prerequisites: Shell commands, I/O re-direction, functions in C
Contents:
a. Review Exercise 2 (Finding Prime Numbers) from the Functions laboratory. You should have written a program that prompts the user for positive integer values and prints a message stating whether the value entered is prime or composite. Your program should also print an error message if a non-positive integer is entered. It should continue prompting and reporting until the user enters the EOF character (ctrl-d) or a non-integer value.
Presumably, you did some informal testing of your program as you wrote it. Do you remember what test cases you used? Suppose I now asked you to modify your program to add new functionality. After doing so, you would of course test your new functions. Particularly diligent programmers (such as yourself) would also want to re-test the existing functionality to ensure that the new changes did not break anything.
On the other hand... that would be such a pain. Right?
The trouble is, the bigger the program, the more of a pain it becomes to test it properly. At the same time, the bigger the program, the more important it is to test it properly! In this laboratory, we will explore ways to automate the testing process, with the intent of making testing easier in the long run.
b. Think of a set of test cases that will thoroughly test your isprime.c program. What test cases should you include?
It is troublesome, but true, that there is as much art as science in testing programs well, given that one goal of testing is to think of unusual occurrances that may not come readily to mind.
At the very least, be sure that your cases include an example of each response required by the problem specification (i.e., prime number, composite number, zero or negative integer, non-integer value, ctrl-d). You should also pay particular attention to "boundary cases" that may arise: in the context of isprime.c, input values of 0, 1, and perhaps 2 seem like reasonable candidates for boundary cases.
c. Note that isprime.c is written to accept input repeatedly from the user. To "automate" such a user, enter your test data in a file with one input value per line, so that the newline character in the file simulates the user pressing the enter key. You do not need to type ctrl-d into your test data file to indicate the end of the file: just end the file, and your program will correctly detect when it reaches the end of the file.
Now rebuild your program and run it, re-directing it to get its input from the test data file. For example, your run command might look like this:
./a.out < isprime-test.dat
Obviously, you will want to examine your output for correctness.
Your output from running the program this way may look strange because the input prompts appear, but the user input does not. Depending on the situation, you may want to comment the prompts out of your code, or you may want to just put up with odd looking output.
The value of testing C programs in this way is that it allows you to use the same test cases multiple times without retyping them. Why is this useful? Consider the possibility that the first time you test a given case, your program gives an incorrect response. Once you fix the problem, you will want to test it again, and you will want to be sure that you have tested it on the same data. Further, you will want to re-test all of your previously working cases to make sure that your most recent change did not cause other cases to fail.
It is good practice (though a somewhat difficult habit to get started) to maintain a set of test cases for each program you write. This makes it easy to re-test your entire program when a new change is made. Re-running all your test cases for each new change is known as system testing.
How can we automate testing when a good set of test cases requires that the program be run multiple times? (For example, we may want to test multiple cases that cause the program to terminate.)
For such a program, we can create a suite of test data files, each including a subset of our test cases. Then, instead of running the program from the command line multiple times and modifying the I/O re-direction part of the command each time, we can create a shell script that runs the program repeatedly.
a. Review Exercise 3 from the Functions Laboratory. You should have a program from this exercise that allows the user to input data points, prints the distance between them, and then exits.
To test this program, we will probably want to run it multiple times. Create a set of (at least 3) test data files for this purpose.
b. We are now ready to create our shell script. A shell script consists of a sequence of shell commands placed in a file. We can then run the full sequence of commands in batch mode, by executing the script, rather than typing each of the commands individually.
There are two things we must do to allow a given file to be run as a script.
We then fill the file with the commands that we want run. Consider the following simple example.
#! /bin/sh #------------------------------------------------------------------------ # A script that builds a C program called distance.c, and runs it against # a suite of test cases. #------------------------------------------------------------------------ # build the program (executable file is named "distance") gcc -Wall -ansi distance.c -o distance -lm #run the program with test cases ./distance < distance.dat1 #The reason for each test case can go here. ./distance < distance.dat2 ./distance < distance.dat3
As illustrated by the example, comments in a script file begin with the character #. You should recognize the other commands in the file. (Note that the very first line of the script has a special syntax of its own; it is not commented out.)
Enter (or copy) the given script into a file. There are no restrictions on the name of a script file, but you may want to call it test-distance or something similar.
Run the script to test all of your test cases with a single command:
./test-distance
Whee!
You can also send the output of the script to a file with:
./test-distance > test-distance.out