Lab 1: Exploratory testing

Purpose

In this assignment, you will get to practice exploratory testing, as in, learning about and documenting functionality of a program while you test it. You will read an introductory paper on Exploratory testing by Jonathan Bach, to understand what exploratory testing means. Exploratory testing is intended to be used when we need to test software where the documentation and specifications are not the only things of interest, but we are rather interested in details about what the software actually does. Writing test cases by translating a specification to test cases verbatim, or following a script to perform “testing” is not generally what humans excel at, or like to do: we can even write automated approaches for test case generation and execution that you encounter in the other labs in this course. However, humans are very well suited for the creative, intellectually rigorous exploration of software. When exploring, we may want to figure out how it should work, or figure out ways in which it may not work. In that sense, testing is an integral part of achieving a better understanding of the product, and indeed the problem the product itself is supposed to solve.

For passing grade you need to complete part 1, for a grade 4 or of 5 you also need to complete part 2.

Part 1 - Mandatory

In this assignment, you shall produce test cases based on your exploration of a product. The products themselves are small web applications, and you shall describe their behaviour by creating a set of test cases of expected behaviour. Group your test cases logically, according to functionality that you believe is related.

machine

Above is an illustration of one of these applications. As you approach this task, you will need to describe how you explore the functionality given, and not just click at random. You shall leave room for experimentation, but describe, broadly, what you are interested in finding out, and formulate hypotheses along the way. When working in pairs, one of you shall take notes of your activities as you explore these applications. Explain aloud to each other how you think as you go about exploring these applications, and formulate hypotheses that you test. Switch places between each black-box machine that you explore, so that each one of you gets to explore and describe the functionality by sitting at the keyboard, and the other person taking notes and helping with the process of understanding what and how to test the application.

Preparations

Read the paper on Exploratory testing by Jonathan Bach that explains the basics of exploratory testing, as well as how to use sessions when exploring software. Create a session report structure to record the information on the tests executed: inputs, expected outputs, actual output, observations, and any additional information you might find useful. Choose four of the Black box machines created by James Lyndsay that you would like to explore. The older puzzles might require Flash, but the newer ones (22 and up) work without needing any plugins. For each machine, create an exploratory testing charter similar to the examples. It should be detailed enough that it gives you some guidance in testing. There should be a clear relationship between the charter and your test activities. Remember, exploratory testing is not random testing and your charter should reflect a testing strategy, as well as document your findings during testing.

Session-based exploratory testing

Given the charter, you are to engage in session-based testing of the product to find out how it works. Make sure that you switch places and take turns exploring the software artifacts, and talk aloud to help each other take notes on how you interpret the functionality of each machine. Continue until you have a description of how the machine works (no more than one or a few sentences), that may be used as a basis for further regression testing. You may need to revise your initial assumptions and iteratively refine your work, but that is part of the process and expected. As you move from one blackbox machine to the next, try to find faster exploration heuristics and detecting how the machines work and note how you plan to explore them faster than the previous iteration. Develop new charters as you go along, if needed.

Part 2 - Optional

Choose a web-based application, for example a Google service, like Google Maps or Google Translate or an online store. Based on the functionality you expect, create a test charter. You cannot test all the application in a single session, so you need to decide what functionality to focus. Include the link to the application you are testing in your test charter.

Reporting your results

See the instructions on the course page general submission instructions.

For part 1, upload your test session transcripts, along with your modifications to your charter between test sessions. For each black-box machine, outline a testing strategy, describe how you select your test cases and conclude with your description of the behavior. Finally, include a short reflection on whether you believe your changes helped you test faster or more productively.
For part 2, upload the test charter with the test session transcript as well as a description of the features you decided to test and of the overall testing strategy.