- Developers, Don’t Write Functional Tests!
- Why Functional Tests don’t belong in a Build Environment
Over the past few years I have noticed that the distinction between functional tests and unit tests has blurred in a lot of projects. I think that using the features of modern testing frameworks like JUnit and TestNG to push functional tests into unit tests is the wrong approach, because it shifts the focus of the test from the test perspective to the development perspective. In this blog post, I explain in detail how I have come to this conclusion.
In order to facilitate the discussion I want to elaborate the arguments using a simplified example, the development of an online shop. Let’s call the artefact of the development effort a product. And please note, that we talk about automated functional tests only.
Let’s focus first on the information flow of the development process. The stakeholder of the shop to be built has a mental picture of the shop in his mind and creates a story board and some business requirements (see figure 1). The analyst studies the story board and the business requirements, analyzes the business processes, and creates a specification with detailed functional and non-functional requirements. The developer reads the detailed requirements and to some degree all other documents created so far and analyzes them, too. The developer creates additional documents (architecture, design, technical requirements), produces the source code (i.e. Java, HTML, SQL …) and the build files.
Imprecise Information Flow
On the way from the stakeholder to the developer, there is a lot of loss or change or even creation of information. The written story board and the business requirements deviate already from the original business model in the mind of the stakeholder.
The analyst reads and (mis-)interprets the written documents, builds a model of his own, and creates further documents, the content of which deviates even further from the mental picture of the stakeholder.
Eventually, the developer reads some or all of the documents, builds a development model in his mind and creates architecture and design documents, finally also code and configuration, which in their functionality will be quite distant to the intention of the stakeholder.
Lots of meetings are organized to try and minimize the discrepancies in understanding caused by imprecise information flow; numerous development iterations are planned to narrow the gap between the intended shop and the created shop.
An additional effort to reach the goal is invested in the area of quality assurance. Functional tests are created to check whether the shop works as specified. Information loss occurs here as well: The tester reads some or all of the created documents (perhaps including the development documents) and builds a model of the shop and how to test it in his mind. Based on this model, the tester creates test descriptions (specifications, test cases …), and finally functional test code, test data and a test configuration.
The Functional Tester as an Independent Institution
The tester builds functional tests from the perspective of the end user of the system and his scope is the entire system including neighbouring systems, i.e. end-to-end. This is usually much closer to the perspective of the stakeholder than the perspective of the developer, who wants to test, what he has built and not what should have been built. The functional tests execute paths through the application (use cases), simulating expected behaviour. First the main paths as described in the story board are tested, then less important paths.
The tester goes through the same steps as the analyst and the developer, but (more or less) independent from them. So due to the imprecise information flow and the bugs in the production code and the test code, the resulting functional tests will not match the implementation of the shop. The tester finds those mismatches, which result in the failing of test cases. He then discusses those mismatches with the developer and sometimes with the analyst or even the stakeholder. Together, they clarify, whether the implementation of the shop or of the test case is wrong (or both), resulting either in fixes to the test case implementation or fixes to the shop implementation. The number of defects in the product found depends (among other things) on the test coverage of the functional tests, the completeness of the specification, and how independent the tester and the developer are working and thinking.
The Four-Eye Principle for Functional Testing
This approach ensures that the documents of the stakeholder and the analyst, and to some degree also those of the developer, are studied twice and independently and from different perspectives (four-eye-principle). So there is a higher chance to detect flaws in the document. This is especially important considering the fact that up to 50% of the defects in a product result from wrong requirements. See e.g. the article Software Defect Prevention – In a Nutshell.
In addition, the likelihood that a misinterpretation of requirements leads to a defect in the product is diminished, because of the two independent interpretations of the document by the developer and tester. In particular the different perspective and focus of the tester (“How can I test this?”) adds a new dimension in thinking, which often leads to finding gaps in the requirements. There is a crude and rude (to the tester) argument for it: the developer has a creational mind-set and the tester a destructive one. A tester of course has to be creative as well, to be able to create good functional tests, but the intention to focus on checking the details and thereby bring down the application, is very helpful.
Contrasting Unit and Functional Tests
A unit test has a different focus than a functional test. It is created in order to check that the functionality promised in the API of the class is fulfilled. I know that a unit test checks a function and can be called a functional test, but I do not want to discuss terminology here. A unit test is a test on a very small scale. It tests a small box with well defined input and well defined output. It is created to stabilize the code and to help the developer change or reorganize existing code. Most classes are not created to fulfil exactly one requirement but are part of a component, which fulfils a set of requirements.
The developer has not the business requirements in his mind while constructing the class, but his own set of requirements for this part of the product, so that it fulfils its tasks within the component (see figure 2). Let’s have a look at a class implementing a caching algorithm or a special sorting algorithm: often there is no direct business requirement for this, but there are concrete needs from the component under construction (development requirements) for such a class. So the developer knows very well what to expect from the class and is therefore the right person to develop the test for such a class. Here we have an ideal case regarding the flow of information. Everything happens within the mind of one person only: the developer. So there is no misinterpretation of documents or misunderstanding in discussions.
Summarizing the argumentation, I want to emphasize the following points:
- A strong distinction between functional tests and unit tests is important.
- Unit tests are written by the developer.
- Functional tests are created by the tester.
In the next post, I will discuss, why I do not want functional tests to run in the build environment.