Test Pipeline

The Test Pipeline is a conceptual term used to describe the behaviour of NCrunch's engine when it executes tests fed to it from the build output. The pipeline itself consists of a series of prioritised tasks containing physical groups of tests.

The logic used to group the tests into tasks takes into consideration a number of factors, including:

• Length of test execution
• Distribution of tests between their parent fixtures
• Length of fixture setup and teardown
• Resource requirements (as controlled by ExclusivelyUsesAttribute, InclusivelyUsesAttribute and SerialAttribute)
• Framework type (NUnit, MSpec, etc)
• Priority

The engine will use this information to try to construct a pipeline that will give the most relevant feedback within the shortest possible time. Prioritisation of these tasks will also take into consideration changes that you have recently made to your codebase - as impacted tests are considered much higher value than others.

It should be noted that it is physically possible for a test to exist in the pipeline multiple times. This is because the engine can be executing multiple versions of your source code at any one time, and therefore a test can actually be running concurrently with another version of itself. This behaviour is configurable.

Data returned from tests after their execution (such as status and code coverage) will be batched together and will only be returned in bulk when the entire task finishes executing. The engine takes this into consideration when building the pipeline and ensures that large and long running tests are placed into their own distinctive tasks to avoiding holding up any other badly needed information.

While you work through your codebase, the test pipeline will be in a constant state of flux. The engine will not interrupt tests while they are partway executing, but it will regularly clear out pending test tasks from the pipeline when new data becomes available from builds. The processing queue window gives a good physical view of the test pipeline along with the status of the work and priorities etc. It can sometimes be useful to work with this view open so as to have some information about the progress of the engine.

Because the engine can theoretically pipeline tests in any grouping or order, you should take care to write high quality tests that do not make dangerous assumptions about their execution order or the state of their application domain.

To ensure the processing queue is not obstructed, NCrunch will rigorously enforce test timeouts.