Test Atomicity

Description

NCrunch will execute tests with very high frequency and under conditions that may stress test code much more heavily than a standard test runner. The nature of NCrunch's batching and task runner process re-use can also subject tests to environmental constraints that did not exist when these tests were designed.

When introducing NCrunch to an existing solution with a complex set of existing tests, it's important to understand the manner in which NCrunch can execute tests and the implications this may have for test code.

Tests executed by NCrunch should be atomic, consistent and repeatable.

Sequence-Dependent Tests

Description

When working with a standard test runner, it is very common that tests will only be run in a fixed sequence. Much of this can be attributed to the way in which manual test runs tend to be for either the entire test suite, or only a small part of the test suite in isolation. The fixed sequence used by many standard test runners can hide interesting state-related issues residing in the tests being executed.

Problem

Tests that have been designed to run in a fixed sequence can fail intermittently when executed by NCrunch. NCrunch can execute any arrangement of tests in any order. Consider the following test code:

public class SequenceDependentFixture
{
	private int counter = 0;

	[Test]
	public void FirstTest()
	{
		Assert.That(counter, Is.EqualTo(0));
		counter++;
	}

	[Test]
	public void SecondTest()
	{
		Assert.That(counter, Is.EqualTo(1));
		counter++;
	}
}

The tests above will fail intermittently in a solution being processed by NCrunch. The tests assume that they will be executed in a specific sequence as part of the fixture, although NCrunch's test pipeline does not guarantee this behaviour.

Problems can also be exhibited with the second test if it is run in isolation with a standard test runner, so our assumption is that this test fixture has only ever been run in sequence from start to finish.

Solution

This problem should ideally be solved in the test code itself. Tests need to be engineered to be atomic and not sequence dependent. The scenario described above could only be logically valid if it is executed as a single test, for example:

public class SequenceDependentFixture
{
	private int counter = 0;

	[Test]
	public void RunTests()
	{
		FirstTest();
		SecondTest();
	}

	public void FirstTest()
	{
		Assert.That(counter, Is.EqualTo(0));
		counter++;
	}

	public void SecondTest()
	{
		Assert.That(counter, Is.EqualTo(1));
		counter++;
	}
}

For sequence dependent tests that fail due to corrupt or unexpected state left behind by other tests, it may be possible to work around the problem by using the Isolated attribute.

Test Runner Re-use

Description

For performance and parallelisation reasons, NCrunch will re-use test runner processes multiple times, running tests in batches (as shown in the processing queue). NCrunch keeps test runner processes active and will call back into them regularly, instructing the relevant test frameworks to execute tests within the suite.

Problem

This behaviour can be problematic for tests that assume their test environment will only be built and torn down a single time. For example, consider the following test code:

[SetUpFixture]
public class SetUpEnvironment
{
    public static int TimesEnvironmentSetUp = 0;

    [SetUp]
    public void DoSetUp()
    {
        TimesEnvironmentSetUp++;
    }
}

public class FixtureRelyingOnEnvironment
{
    [Test]
    public void CheckEnvironment()
    {
        Assert.That(
            SetUpEnvironment.TimesEnvironmentSetUp, 
            Is.EqualTo(1));
    }
}

The CheckEnvironment test above can fail intermittently when executed by NCrunch. This is because the test assumes the SetUpFixture's SetUp method will only ever be called once, where NCrunch's reuse of the testing process can result in a new call to the SetUpFixture for each test batch executed.

Solution

This problem can only be solved in the test code itself. Test code should be made resilient to multiple calls into the test process. The following code has been adjusted with this in mind:

[SetUpFixture]
public class SetUpEnvironment
{
    public static int TimesEnvironmentSetUp = 0;
    private static bool EnvironmentInitialised = false;

    [SetUp]
    public void DoSetUp()
    {
        if (EnvironmentInitialised == false)
        {
            TimesEnvironmentSetUp++;
            EnvironmentInitialised = true;
        }
    }
}

public class FixtureRelyingOnEnvironment
{
    [Test]
    public void CheckEnvironment()
    {
        Assert.That(
            SetUpEnvironment.TimesEnvironmentSetUp, 
            Is.EqualTo(1));
    }
}