Heuristics

What are heuristics and when to use them?

A significant part of testing activities involves making quick judgements and getting first impressions. Often, these are followed by more detailed investigations and experiments. To support these activities, heuristics can be of great help. 

 

Definition A heuristic is a strategy derived from previous experiences with similar problems, which employs a practical method, not guaranteed to be optimal, perfect, or rational, but instead sufficient for reaching an immediate goal. It can involve using a rule of thumb, an educated guess, an intuitive judgment, a guesstimate, profiling, or common sense.

Heuristics for static testing 

The results of a review (of documents, program code or whatever source) tend to rely on the reviewer’s experience. We would like to have some consistency over multiple reviews of various people. Heuristics can be of great help. The following sections show heuristics that can be used during refinements, reviews and other forms of static testing. Most heuristics for static testing can be used as a checklist to investigate if the test object complies with the various parts of the heuristic.  

SMART 

SMART is an acronym of the following terms:  
     Specific – provide a specific description with sufficient details  
     Measurable – quantify an indicator of progress and/or success  
     Achievable – is it possible within the restrictions and conditions of the situation  
     Relevant – does it relate to the objectives for the specific situation  
     Time-bound – are timelines clearly stated  

There are many variations and extensions of the SMART acronym, all have the same goal of assisting the reviewer. 

INVEST 

INVEST is an acronym of the following terms, specifically meant to review user stories [Wake 2003]:  
     Independent – of all other user stories  
     Negotiable – not a specific contract for features (captures the essence, not the details)  
     Valuable – to the customer  
     Estimable – to a good approximation  
     Small – so as to fit within an iteration  
     Testable – in principle, even if there isn’t a test for it yet  

More about INVEST can be found in section “Description of quality measures“.

Always, never and other absolute words

When reviewing a text, such as a user story or requirements specification, look for words like always and never and try to imagine situations that do not comply with these absolute terms.

Is it logical?

When reviewing a text, such as a user story or a requirement specification, ask yourself if the description is logical, this can be done in various ways. Is it consistent with other systems, with the previous version of the same system, with the expectations of the targeted users, with the corporate image of the company that will supply it to its customers, and most importantly: is it fit for purpose (i.e. does it solve the user’s problem).

Heuristics for preparing dynamic tests

Preparing dynamic tests can be roughly done in two ways: using test design techniques to create fully prepared test scenarios that are executed at a later stage, and using experience in an exploratory approach where one (or a few) test cases are created and executed right away.

The distinction between heuristic and test design technique is not a strict one. In her book Elizabeth Hendrickson describes CRUD as a heuristic [Hendrickson 2013] where we describe it as part of the test design technique Data Cycle Test (DCyT). (Note: CRUD is an acronym for Create, Read, Update, Delete, the actions that relate to the lifecycle of a data item.)

Numeric information

Below a few ideas on how to test numeric information (based on [Hendrickson 2013]).

• When testing a numeric input field try:
  Zero, positive value, negative value, integer value, decimal point (or decimal comma), hexadecimal value, highest possible value, lowest possible value, value higher than highest possible, value lower than lowest possible, non-numeric input, no input (empty field).
• When testing an alphabetic input field try:
  One letter, a few letters, many letters, the maximum number of letters, more than the maximum number of letters, special letters (e.g. ä, ç, ñ etc.), special characters (e.g. %, &, $ etc.), numbers, no input (empty field).
• When testing a date input field try:
  Today’s date, yesterday’s date, tomorrow’s date, wrong date (e.g. 32nd day, 13th month), 29 February in a leap year and in a non-leap-year, a date a century ago, a century in the future, a future birthdate, a birthdate of more than 100 years ago, DDMMYYYY format, MMDDYYYY format, YYYYMMDD format, any other applicable formats.
• When testing a time field try:
  Different time formats (e.g. HH:MM, HH:MM:SS, HH:MM:SS:THT), 00:00, 24:00, try more than 24 hours, try system time formats such as UNIX-seconds-since-01-01-1970.

Other kinds of heuristics

Below a few examples of other heuristics.

• When testing a screen with multiple options try:
  One option, a few options, many options, all options, too many options, too few options.
• When testing a process try:
  Shortest normal way through the process, longest normal way through the process, alternative ways through the process, exceptional ways through the process, interrupting the process halfway, restart the process after interrupting, reverse the process.
• When testing output try:
  Getting every possible category of output, output approximations if the exact output is difficult to predict.
• When testing a graphical user interface try:
  Back and forward during the process, save the screen and later start the process with the saved screen, zoom the screen to maximum, minimum and average format, use different screen sizes, use different types of devices (computer, tablet, smartphone, smartwatch).
• Nobody would ever do this…
  Keep a list of real-life examples of actions that real users have taken that no professional IT person had ever dreamt were possible. For example:
  • The instruction says “hit any key”. The user observes there is no key on the keyboard marked “any” and cannot complete the task.
  • The instruction says “make a copy of this USB stick”. The user walks to the photo-copier, puts the USB stick on the glass plate and pushes “copy”.

You will find more examples in your day-to-day life at the office.