Heavily Executed Code Under Test

Description

In order to analyse code coverage and collect performance information, NCrunch will manipulate built assemblies to insert instrumentation that can collect this data. This instrumentation is transparent while working with code or while using the debugger, and has been designed not to alter the logic of the code under test in any way.

Possible Problems

A side effect of instrumenting code is that the instrumentation can add considerable weight to the code. This problem is present in all tools that profile an application's execution for any reason. In most situations, the extra weight added to the code is not noticeable. This is because .NET code tends to rely on libraries and frameworks for performance critical routines, and only rarely contains heavy loops of frequently executed statements. However, in some situations the difference can be significant. Examine the following test code:

[Test]
public void DoLotsOfSomethingSmall()
{
    var strings = new StringBuilder(10000000);

    for (int i = 0; i < 10000000; i++)
        strings.Append("a");
}

When executed with a standard test runner on a mid-range development computer, this test takes about 180ms to execute. When running the test with the default configuration under NCrunch, the test takes about 3 seconds to execute. The instrumentation NCrunch has added to the heavily iterated 'Append' method call has reduced the performance of this code.

This problem can be much easier to notice where it causes tests to fail due to timeout under NCrunch.

Solutions

Code Coverage Suppression

There are several ways to solve this problem. The best approach is usually to suppress code coverage for the performance critical region of code. This can be done using NCrunch's inline suppression comments, for example:

[Test]
public void DoLotsOfSomethingSmall()
{
    var strings = new StringBuilder(10000000);

    //ncrunch: no coverage start
    for (int i = 0; i < 10000000; i++)
        strings.Append("a");
    //ncrunch: no coverage end
}

The comments above will force NCrunch to avoid instrumenting the performance critical region of the code. This means no code coverage, performance information or RDI data will be reported for this region, but the code will run at full speed.

Runtime Data Inspection Settings

If you are using RDI, it can be worth using RDI inline directives to adjust RDI for less aggressive data collection over CPU-intensive regions of code.

Configuration Changes

There are two less preferable alternatives to the above solution:

• Turn off performance analysis for the assembly containing the code by setting the Analyse line execution times configuration setting to false. The instrumentation added by performance analysis is much heavier than that used for code coverage analysis, so this should make quite a difference in the execution speed of all code within the project.
• Turn off all instrumentation for the assembly containing the code by setting the Instrument output assembly configuration setting to false. This will stop NCrunch from manipulating the built assembly, disabling all code coverage and performance analysis while allowing all code within the project to run at full speed.
• Turn off all RDI data collection for the assembly containing the code by setting the Enable RDI configuration setting to false. NCrunch will then avoid adding any RDI instrumentation to the assembly, making all RDI data for the related project completely unavailable.