Framework Utilisation Types

Purpose

These configuration settings control the manner in which NCrunch analyses test project assemblies to find tests.

Available Options

Static Analysis

When set to StaticAnalysis, NCrunch will find tests within the test assembly by analysing the assembly in its binary form and using the CLR metadata available. This analysis happens as part of the instrumentation process NCrunch uses to manipulate the built assembly.

Static analysis is the fastest way for NCrunch to find tests, but it relies on inferred knowledge of the workings of the applicable test framework. Because in most cases static analysis does not involve direct interrogation of the test framework for information, it is less aligned with test framework standards and may not find tests that rely on edge features specific to the test framework.

Dynamic Analysis

When set to DynamicAnalysis, NCrunch will perform an additional analysis step against the test assembly after it has been built, in the form of an analysis task (visible in the processing queue). This analysis task involves creating a runtime application domain that contains the loaded test assembly. The application domain allows the analysis task to extract test information from the assembly through integration with the applicable test framework, which usually relies on runtime reflection to find tests.

Dynamic Analysis is considered to be the most reliable and standardised approach to discovering tests within an assembly, although it has a higher overhead as runtime reflection is generally slower than static analysis, and requires an application domain to be built up-front.

Specific Framework Considerations

NUnit v2

By default, NUnit v2 is set to use dynamic analysis. This is because NUnit contains a number of features (such as the TestCaseSource attribute) that make use of user code in the discovery of test cases. Fully aligned support for these dynamic test cases is impossible when using static analysis, as static analysis can only inspect user code without executing it.

When using NUnit v2 with static analysis, many of its edge features (such as parameterised fixtures and test cases) will work only through the collapsing of multiple test cases into single tests. This does allow these features to work under NCrunch, although the experience is less optimal than when using dynamic analysis.

NUnit's [Random] attribute will only work with a utilisation type of static analysis.

The performance difference between static analysis and dynamic analysis is not significant for NUnit v2.

NUnit v3

The introduction of NUnit v3 included a new API targeted towards test runners such as NCrunch. This API provides a number of benefits in terms of compatibility and cross-version reliability. However, the API does not support static analysis. NCrunch can only work with NUnit v3 using dynamic analysis. Because of this, NUnit v3 is controlled only by an on/off switch instead of a framework utilisation type.

MSTest

There is no functional difference between static and dynamic analysis when using MSTest with NCrunch. Because dynamic analysis still relies on the use of a runtime application domain, static analysis will always provide better performance and should always be used for this framework unless unexpected problems are experienced with it.

MSpec

As MSpec does not contain features that require execution of user code to discover tests, there should be little reason to use it with dynamic analysis. Static analysis will perform more efficiently with this framework and should be used in all situations unless unexpected problems are experienced.

From NCrunch v3.31, dynamic analysis is no longer supported for MSpec and this framework is instead enabled with an on/off switch instead of a framework utilisation type. From v3.31 upwards, MSpec tests are always discovered using static analysis.

Gallio/Xunit v1

NCrunch relies on Gallio's abstractions when discovering Xunit V1 tests.

There are subtle differences in behaviour between static and dynamic analysis when using Gallio. Dynamic analysis is more closely integrated with the underlying framework (i.e. MbUnit/Xunit), and will generally provide better support for specific edge features within the framework, such as Xunit tests declared in nested types and the DisplayName Xunit Fact attribute parameter.

Gallio's static analysis performs significantly better than its dynamic analysis. As such, this framework is set to use static analysis by default. It is strongly recommended that you avoid using dynamic analysis with Gallio/Xunit for large test assemblies, as this can greatly increase your test cycle times.

Xunit v2+

NCrunch V2.6 introduced direct integration with Xunit V2. Xunit V1 tests will continue to execute using the older Gallio adapter, while Xunit V2+ tests are run through NCrunch calling directly into the Xunit test runner.

Static analysis may give some performance benefit for discovering Xunit V2 tests, but as it does not support all the features of Xunit (including custom trait discoverers, nullable theory enum parameters and collection fixtures), dynamic analysis is enabled by default. Under v3.9 and above, NCrunch does not support static analysis for Xunit v2 and will use only dynamic analysis.