What does Property-Based Testing mean for developers? Well, a debate persists in software as to the exact nature, division and scope of “testing” activities. We need to check that a particular software component produces the expected output with the expected inputs provided. Yet we also need to expose the behaviour of this component against unforeseen or unpredictable inputs.
The first activity is usually referred to as “checking”, while the second is called “exploring” and both contribute to defining the software components as “tested”.
The boundary that divides these two different activities is often linked to the level of knowledge of the component being examined. However, we must be very careful not to confuse the knowledge of the issue under analysis with the illusion of knowledge.
Introducing the Black Swan Theory
A common example of the negative effects of the illusion of knowledge is the so-called “black swan theory” introduced by Nassim Nicholas Taleb in 2007. In short, if it is assumed that all swans are white and that, therefore, black swans cannot exist, it means the existence of a single black swan can bring down the whole theory.
In the field of testing, the illusion of knowledge is linked to the cognitive bias that, want it or not, we apply both when we write code and when we design and structure the various tests and test cases necessary to confirm the functionality of the code itself.
Some types of automatic tests, such as unit tests, only allow confirmation of the behaviour. An example is the confirmation of the occurrence of a certain condition. What could happen unexpectedly in code like the following?
<script src=”https://gist.github.com/elleuca/267b5db40dd72c03234c5eb76b9835c5.js”></script>
It is preferable to add the various limit cases of interest. However, the illusion of knowing the range within the price we will pass to the function will vary. This could lead to the belief that it will never provide a value that can produce an unexpected result.
Have you ever tried to use Shakespeare’s entire work in Japanese and Korean as an input to your method that takes a valid string as a parameter? It’s an impossible case, so why should we use it as input for a test?
Qualities of Good Testing
What we could expect from good testing? At Codemotion Rome 2019, Kenny Baas and João Rosa shared their ideas about how testing should work to shift from just confirming to learning. When your software breaks, it is caused by at least one mistake. It happens, but you can leverage those mistakes to learn more about what you built or the actual context of the problem you are trying to solve.
Learn and Challenge Code Through Property Based Testing
If you want to learn more about the context, or the domain you are working on, your tests should describe and communicate what is happening. Domain-Driven Design can help you to write code that is based on the same language developers speak with each other and other domain experts. Tests should help you to document how your software behaves, not assert that it can pass a pile of anonymous data points.
To challenge the code you wrote, you need to move away from the comfort zone of known valid inputs and feed it unforeseen data that can disprove your illusion of safety. If you can do so, failures in tests will reveal an actual problem in your system. If not, the tests will only assert that your system will work when it flows as expected.
Such kind of testing is called “property based testing” and helps to introduce a degree of fuzziness in test inputs, allowing to move from single abstraction examples (a.k.a. unit tests) to an active scanning of system behaviour.
In the next article of this series, we’ll explore more about how to start building property based testing. Stay tuned…
