.NET Standard Test Projects

Description

Under .NET, it is possible to create .NET Standard projects that make use of test frameworks and declare tests. Some test runners will implicitly choose their own platform and may be able to run these tests directly. When NCrunch detects a .NET Standard project making use of a test framework, it will give an error and will not execute any tests in the project.

Why are .NET Standard Test Projects a problem for NCrunch?

.NET Standard itself is not a runtime platform. .NET Standard projects can only be loaded by the CLR indirectly, as references from other projects targeting specific platforms. For example, it is not possible to create a working console application that targets .NET Standard. However, it IS possible to create a .NET Core console application that references a .NET Standard library.

This means that for a test runner to execute tests within .NET Standard libraries, it must arbitrarily choose a platform on which the tests should be executed. For example, it is possible for a test runner to simply load the .NET Standard test assembly into a .NET Framework v4.6 runtime environment and execute it as though it were actually a .NET Framework v4.6 library.

However, such a choice is not as simple as it would appear. Any default platform chosen by the test runner must be available on the machine used to execute the tests. Because different machines have different versions of .NET and .NET Core installed, the chosen platform would be entirely dependent on the environment used to execute tests. This environment could be different between team members or grid nodes, giving inconsistent results even though the codebase used is exactly the same. Such indeliberate inconsistencies are not desirable in software testing scenarios.

Developer software testing is most effective when it is performed using the platform(s) on which the software is expected to be executed in a live environment. By choosing to make your test project target .NET Standard, you are effectively choosing not to care which platform should be used to execute your code. Given the high likelihood of differences between runtime platforms, you actually should care about this very much. With no certain knowledge of your live environment or the platform(s) on which your .NET Standard project is intended to be run, NCrunch is completely unqualified to make this choice for you.

Solutions

The solution to this problem depends on why the project is targeting .NET Standard.

If the project targets .NET Standard simply as an accidental choice or because no thought was given to this issue, you should retarget the project to a runtime platform (such as .NET Core or .NET Framework).

If the project is being used to test code intended to run on multiple platforms (for example, a set of libraries to be published on Nuget), you should instead multi-target the test project to each of the platforms your code is expected to be run on. NCrunch will then internally duplicate the test project and the tests it contains, giving extensive test coverage of all chosen platforms.

If the project is intended to also be shared with a different test runner (for example, to target iOS/Android), you should multi-target it to both .NET Standard and a platform that is supported by NCrunch, then set the Ignore this component completely for the .NET Standard NCrunch derivative of this project to TRUE so that NCrunch's error is suppressed.

If the project actually doesn't contain any tests and is only being used as a utility project referencing test frameworks, you should use NCrunch project-level configuration to disable the test frameworks involved to suppress the error.