Framework Utilisation Types

Purpose

These configuration settings control the manner in which NCrunch analyses test project assemblies to find tests.

Available Options

Static Analysis

When set to StaticAnalysis, NCrunch will find tests within the test assembly by analysing the assembly in its binary form and using the CLR metadata available. This analysis happens as part of the instrumentation process NCrunch uses to manipulate the built assembly.

Static analysis is the fastest way for NCrunch to find tests, but it relies on inferred knowledge of the workings of the applicable test framework. Because in most cases static analysis does not involve direct interrogation of the test framework for information, it is less aligned with test framework standards and may not find tests that rely on edge features specific to the test framework.

Dynamic Analysis

When set to DynamicAnalysis, NCrunch will perform an additional analysis step against the test assembly after it has been built, in the form of an analysis task (visible in the processing queue). This analysis task involves creating a runtime application domain that contains the loaded test assembly. The application domain allows the analysis task to extract test information from the assembly through integration with the applicable test framework, which usually relies on runtime reflection to find tests.

Dynamic Analysis is considered to be the most reliable and standardised approach to discovering tests within an assembly, although it has a higher overhead as runtime reflection is generally slower than static analysis, and requires an application domain to be built up-front.

Specific Framework Considerations

NUnit

By default, NUnit is set to use dynamic analysis. This is because NUnit contains a number of features (such as the TestCaseSource attribute) that make use of user code in the discovery of test cases. Fully aligned support for these dynamic test cases is impossible when using static analysis, as static analysis can only inspect user code without executing it.

When using NUnit with static analysis, many of its edge features (such as parameterised fixtures and test cases) will work only through the collapsing of multiple test cases into single tests. This does allow these features to work under NCrunch, although the experience is less optimal than when using dynamic analysis.

NUnit's [Random] attribute will only work with a utilisation type of static analysis.

The performance difference between static analysis and dynamic analysis is not significant for NUnit.

MSTest

There is no functional difference between static and dynamic analysis when using MSTest with NCrunch. Because dynamic analysis still relies on the use of a runtime application domain, static analysis will always provide better performance and should always be used for this framework unless unexpected problems are experienced with it.

MSpec

As MSpec does not contain features that require execution of user code to discover tests, there should be little reason to use it with dynamic analysis. Static analysis will perform more efficiently with this framework and should be used in all situations unless unexpected problems are experienced.

Gallio/Xunit

NCrunch relies on Gallio's abstractions when discovering Xunit tests.

There are subtle differences in behaviour between static and dynamic analysis when using Gallio. Dynamic analysis is more closely integrated with the underlying framework (i.e. MbUnit/Xunit), and will generally provide better support for specific edge features within the framework, such as xUnit tests declared in nested types and the DisplayName xUnit Fact attribute parameter.

Gallio's static analysis performs significantly better than its dynamic analysis. As such, this framework is set to use static analysis by default. It is strongly recommended that you avoid using dynamic analysis with Gallio/Xunit for large test assemblies, as this can greatly increase your test cycle times.