Checking Program Output

For many of the assignments in this class, you will need to compare your program’s output to some known correct output. This is fairly easy to do for a few cases, but it can get quite annoying if you have to do it repeatedly. While there are many heavy-weight systems for testing programs, we can rely on some basic functionality in the shell and a standard command line utility to build a simple testing system.

Imagine we have a simple program called sorter that is supposed to read in values and print them out in sorted order. The first step is to write a simple shell script that can run our program on some basic test cases. We’ll call this file testcases.sh, with the contents below:

#!/bin/bash
echo 1 2 3 4 5 | ./sorter 
echo 5 4 3 2 1 | ./sorter 
echo 9 3 0 5 3 | ./sorter 

The first line of this file tells Linux that it should run this code with the bash program, which also happens to be the default shell. Before you can run the program you have to make it executable. You can do this from your shell:

$ chmod u+x testcases.sh

This grants the user (you) permission to execute the file testcases.sh.

Now we have a script that can run our program on a few simple inputs. The next step is to make a file that holds the expected output for each run. We’ll call this correct-outputs.txt:

1 2 3 4 5
1 2 3 4 5
0 3 3 5 9

Each line corresponds to the output from a single run of the program. Given these correct outputs, now we can run testcases.sh to get the real output from our program. We don’t necessarily want to see every line of output if we have many test cases, so we’ll instead capture the output in a file. Then, we can use the diff command to compare output.txt to correct-output.txt. It would be convenient if we could use make to run our tests, so we’ll actually add the code to do this to Makefile. At the end of Makefile, we’ll add the following lines:

test:: sorter
	./testcases.sh > output.txt
	diff output.txt correct-output.txt

Make sure your indented lines are intented with Tab, not spaces. Makefiles are picky about whitespace, and you could get a cryptic error if you indent incorrectly here.

Now to run our tests, we just run make test. If everything matches, make test will complete successfully. If it doesn’t match, make test will exit with an error and print out the lines that differ in the two output files. You could split up your test cases to make it easier to check output values if you have many long test cases, but this approach should work well for small numbers of tests that don’t produce too much output.