Program Testing: Choose Inputs That Expose the Mistake

Choose a test because of what it can reveal

A program can produce a plausible answer for one input and still contain a mistake. A useful test starts with a rule, then chooses an input whose correct response can be worked out independently. Compare that expected result with the actual result. Do not treat whatever the program prints as the answer key.

This fictional storage rack has 24 slots. A small program receives the number of slots to occupy and reports how many will remain free. Each example below contains just one fault. Consider each faulty version separately rather than combining their mistakes.

Write the input contract first

Accept a numeric whole-number value from 1 through 24 inclusive. For an accepted value x, free slots = 24 − x. Reject other whole-number values with “Reject range”. Reject non-whole numbers and text with “Reject type”. Rejected inputs must not produce an accepted free-slot result.

Inputs in this paper model have a type. Numeric 12 is different from the text “12”; there is no automatic text-to-number conversion. A numeric value with no fractional part, such as 12.0, is a whole-number value under this rule. The helper IS_WHOLE_NUMBER safely returns false for text and fractional numbers. Subtracting text from a number causes a run-time error in this model.

A normal test such as 12 exercises an ordinary accepted case. Boundary tests probe where the result changes. For this inclusive range, choose 0, 1, 24 and 25 to test immediately outside and exactly on both limits. Invalid-type tests such as 2.5 and text “oak” examine a different rule. The categories can overlap: zero is both near a boundary and invalid.

A correct reference design

The following pseudocode implements the stated contract. Its helper checks the value without attempting unsafe arithmetic. Indentation shows which response belongs to each decision; these are language-neutral instructions.

READ x
IF NOT IS_WHOLE_NUMBER(x) THEN
    OUTPUT "Reject type"
ELSE IF x < 1 OR x > 24 THEN
    OUTPUT "Reject range"
ELSE
    free ← 24 - x
    OUTPUT "Accept", free
END IF

For x = 12, derive 24 − 12 = 12 from the rack capacity. For x = 24, derive zero free slots. Those expectations come from the problem, so they can expose a mistake in the code rather than repeating it.

Worked case 1: the endpoint is accidentally excluded

Version A uses x ≥ 24 as a rejection condition. This changes the upper boundary from inclusive to exclusive.

READ x
IF NOT IS_WHOLE_NUMBER(x) THEN
    OUTPUT "Reject type"
ELSE IF x < 1 OR x >= 24 THEN
    OUTPUT "Reject range"
ELSE
    free ← 24 - x
    OUTPUT "Accept", free
END IF

The input 12 passes, so that test alone misses the fault. Input 24 reveals it: the rule expects acceptance with zero free slots, while A rejects it. Input 25 is also useful, but both the correct and faulty versions reject 25; it cannot distinguish these two versions by itself. Replace ≥ with > in that one rejection test.

Worked case 2: the arithmetic uses the wrong operation

Version B has the same validation as the correct reference but changes only one line to “free ← 24 + x”. For input 12 it reports 36 free slots, while independent subtraction gives 12. For input 1 it reports 25 instead of 23; for 24 it reports 48 instead of 0.

The range tests can still work perfectly while the calculated result is wrong. Check the exact output, not merely whether the program reaches its accepted branch. Replace addition with subtraction, then repeat more than the one test that first exposed the error.

Worked case 3: validation happens too late

Version C calculates before establishing that arithmetic is safe:

READ x
free ← 24 - x
IF NOT IS_WHOLE_NUMBER(x) THEN
    OUTPUT "Reject type"
ELSE IF x < 1 OR x > 24 THEN
    OUTPUT "Reject range"
ELSE
    OUTPUT "Accept", free
END IF

Numeric 12 still works. Numeric 2.5 is eventually rejected, although unnecessary arithmetic happened first. Text “oak” causes the subtraction to fail before the rejection message can be produced. A later validation statement cannot protect an earlier unsafe operation. Move the subtraction into the accepted branch, after both checks.

Different programming languages may convert or reject values differently. The run-time-error result here follows the explicit no-conversion model above; it is not a claim that every language treats text identically.

Complete comparison of the selected tests

A logical error can leave a program running but producing the wrong decision or number, as in A and B. C produces a run-time failure for the text inputs in this model. Testing tells us which expectations are violated; debugging investigates and corrects the cause.

Retest record after the corrections

For corrected A, B and C, run all eight rows again. Each now gives the expected result in the comparison table: 0 and 25 reject range; 1, 12 and 24 accept with 23, 12 and 0 free; 2.5, text “oak” and text “12” reject type. This is a paper retest of the corrected reference design, not a claim about an unspecified installed program.

Add two checks that were not in the original set. Numeric 12.0 should accept with 12 free because it has no fractional part. Numeric −1 should reject range. These help check the stated type rule and a second value below the minimum. Retesting old cases after a change also guards against a correction that accidentally breaks previously correct behaviour.

Three solved test-selection tasks

1. You may choose 12 or 24 to distinguish A from the reference. Which is better? Choose 24. It is the exact upper endpoint that A incorrectly excludes. Twelve is accepted by both and therefore does not expose this particular fault.
2. A tester expects 14 free slots for input 12, but the program reports 12. Is there a bug? Not on that evidence. Recalculate: 24 − 12 = 12. Correct the expected answer. A disagreement is a reason to investigate both the program and the expectation.
3. A test set contains only 1, 12 and 24. Add one input that reveals C’s unsafe ordering. Add text “oak”. Arithmetic cannot process it under the stated model, whereas validation should reject it safely. Adding 25 checks range rejection but does not expose the text failure.

Reference for expected-response testing

The Python Software Foundation’s unittest documentation explains checking a specific response to given inputs and comparing results with expectations. The storage-slot rule, pseudocode, fault variants and answers here are original and do not require Python.