Oracle

Concepts

• An oracle is any software, process, or data that provides the test case designer with the expected output of a test case. <bib>vincenzi-etal:2007:slides</bib>

• Test oracle is a mechanism used for generating expected results for software under test. <bib>jin-etal:2008, p. 1</bib>

• Oracles are external sources of information about functions. <bib>howden:1986, p. 1</bib>

• An oracle is any software, process, or data that advices the test case designer whether the output of a test case is acceptable. <bib>delamaro:slides:2009</bib>

• An oracle is a method for checking whether the system under test has behaved correctly on a particular execution. <bib>baresi-young:2001, p. 1</bib>

• A test oracle is a mechanism against which testers can check the output of a program and decide whether it is correct. <bib>chan-etal:2006, p. 1</bib>

• Test oracle is a program which evaluates the output of other programs. <bib>andrews-etal:2002, p. 1</bib>

• A test oracle is a means, automated or manual, for checking the correctness of the outputs computed by the module under test. <bib>hoffman-strooper:1991, p. 935</bib>

Facts

• An oracle decides whether output values are correct against what is specified. <bib>vincenzi-etal:2007</bib>

• An oracle is required to determine whether a fault was revealed. <bib>vincenzi-etal:2007</bib>

• An oracle can be represented by a domain expert. <bib>vincenzi-etal:2007</bib>

• The types of oracle are: kiddie oracle, regression test suite oracle, validated data oracle, purchased test suite oracle, existing software oracle. <bib>vincenzi-etal:2007:slides</bib>, <bib>delamaro:slides:2009</bib>

• The basic kinds of oracles used in functional testing and analysis are input-output oracles, trace oracles, and interface oracles. <bib>howden:1986, p1</bib>

• It is difficult to choose the characteristics to be considered when comparing the test case output and the expected output. <bib>oliveira:2010</bib>

• The implementation of an oracle is not trivial because results can last just fractions of second and cannot be easily observed or be disturbed by noise or hidden behind a hardware-software architecture (hidden results) and the observation or recording process may introduce further errors or uncertainties into the execution results. <bib>tse-etal:2007, 80</bib>

• The implementation of an oracle is not trivial because generation of the problem of automating the generation of correct reference results against which to evaluate component outputs. <bib>hummel-atkinson:2005, p. 1</bib>

• The implementation of an oracle is as subject to errors as is the program it was designed to. <bib>oliveira:2010, p. 15</bib>