Blog

Dirty Systems

March 1st, 2017

It is often the case when creating tests for systems that persist data that we must consider the entry level state of that system before the tests begin. For example, if we want to test the ability to create a new account we must first be sure that account does not already exist otherwise we run the risk of a false positive. If the system supports the deletion of accounts, for example, it makes the task somewhat easier but the fundamental problems do not go away. What if the delete function was not run or failed on a previous test run? What if someone manually created an account by that name on the system prior to the tests being run?

The bottom line is, when testing systems that persist data, if they are dirty this must be taken into consideration and we must attempt to minimize the level of of system dirtiness.

To better understand this concept consider a clean system as one where we may return the system to a constant known state before each test run. Ideally, we could reset the entire system and wipe clean any data stores, but this is often not possible. Isolating a system and allowing it to only be used for testing purposes is often a good starting point which is why many organizations have QA servers, but even dedicated servers can become dirty if the environment is not controlled.

If a system does not allow accounts for example to be deleted we must somehow persist state between test runs. This is accomplished in Test Case Central using data stores but many frameworks do not support this key feature. When persisting test state between test runs is not possible we must revert to system state detection. For example, assume our tests always produce accounts with email addresses of the following form 'test_001@example.com'. We could construct a pre-test step which queries the system and determines what the last email address we created was and then provide this as input to the test process.

While it may seem odd at first, it is actually good practice for a test system to have this pre-test phase where the constraints imposed by the testing process are validated before testing begins, even when testing will be performed on a clean system. While at first this level of validation may seem paranoid or recursive in the long run it will reduce issues like false positives.

Ken