How do you decide you have tested enough ?

Question

This question is related to software testing only.

Answer 1

This is a very difficult question to really answer correctly. According to Godel's incompleteness theorem, a system is either complete or inconsistent. Although Godel proved it for the natural numbers, i have taken the liberty of using it as an analogy for software systems.

A practical way to proceed is to test throughout the life cycle and to correct errors. As the software gets used and errors get fixed, (without introducing new errors), it becomes more reliable. The existence of various software environments (different operating systems, transaction monitors, database systems, interfaces to other applications and services, etc.) complicates matters. Particlarly when you have little knowledge and/or control over the environmental systems and software.

So what to do about the initial and subsequent releases. One way is to classify errors into severe, medium and minor. The classification depends on the application and impact of the application. Then have a release policy depending on the number and type of errors detected in testing.

To illustrate, consider the following applications:

real time flight control software for a passenger aircraft, passenger reservation software for airlines or trains, on-line stock trading software, word processing software, chat and email software.

Clearly the societal impact of errors in the above applications differs considerably.

Some organizations also use quantitative models to predict residual-error distributions based on on-going test results.

Testing, maintenance and support are part of the software life cycle and i think its very difficult to say "i've tested enough", simply because it's difficult to define "enough".

To build and maintain good software you need good requirements and analysis, good design which is maintainable and testable, good practices, good tools, good environments, good infrastructure, good communication and most of all good people.

Answer 2

1

Answer 3

Ideally, when you have tested in all known environments and all was correct 3-5 times in a row in each environment (PC, Mac, IE, Netscape - whatever).