Control Cores Used For Test Execution

Purpose

When enabled, this setting instructs the engine to attempt to force tasks to run on specifically allocated CPU cores only.

CPU cores are allocated rotationally to build, test and analysis tasks and are not shared between tasks.

Considerations are made for tests that make use of UsesThreadsAttribute. Tests marked to use more than a single CPU core will have proportional cores allocated to them.

Implementation and Limitations

When this setting is enabled, all tasks in NCrunch's processing queue become subject to CPU affinity control. This includes build tasks, analysis tasks, and testing tasks.

The affinity controller considers a range of factors, including:

• CPU core control settings in the VS/Rider plugins
• Tests making use of UsesThreadsAttribute to declare the number of threads they require
• The physical architecture of CPUs on the machine, with particular attention to NUMA nodes
• Limitations of Win32 APIs for controlling affinity, particularly around processor groups and 32-bit environments

The Windows operating system has an interesting history in its capabilities around CPU core affinity settings. The existing API set was originally designed around 32-bit limitations, and its ability to handle affinity controls on systems with more than 32 cores has resulted in a range of extensions and limitations that can be difficult to reliably navigate.

The Win32 APIs have the following limitations around CPU affinity control:

• It is not possible for a 32-bit process to control affinity for anything other than the first 32 CPU cores in a processor group. This is because the APIs responsible only accept affinity bitmasks with a max size of 32.
• It is not possible to assign affinity at process level beyond the 'current' processor group. This means full affinity control on systems with more than 64 cores is effectively impossible at process level and must be done at thread level instead.
• Under newer versions of Windows, processes and threads without an affinity mask applied to them are free to 'float' to any CPU core on the system. However, there is no effective API that can be used to reset an affinity mask once it is applied. This means there is no way to get a process or thread back to the fully floating state once it gets messed with. The process must be restarted.
• The APIs used to assign thread affinity can only work with a single processor group at a time. This means it is impossible to assign affinity across processor groups. This limits the ability of NCrunch's allocator to handle tests declaring a need for many cores.
• Process affinity is inherited through a process tree, but Thread affinity is not.

To mitigate the impact of these limitations, NCrunch's allocator does the following:

• Affinity is always assigned at process level where ever possible. It is also assigned at thread level regardless of this. This means that allocation works more reliably on systems with 64 cores or less.
• Threads are 'discovered' by NCrunch using instrumentation only. This means that threads executing user code should be reliably caught and assigned the appropriate affinity, but threads controlled by external or platform code are able to hide from the system and may still cause overstepping.
• 32-bit test processes control their own affinity only if the required mask can fit within the 32-bit limit. Where this is not the case, they pipe an IPC request to the 64-bit engine process to perform the API calls on their behalf.
• An error will be shown when attempting to control CPU cores past the first 32 cores when NCrunch is running its engine in a 32-bit process.
• Significant effort is made to avoid splitting test process affinity between processor groups or NUMA nodes.

It is important to note that affinity control over NCrunch tasks is not absolute. Tasks often make use of externals that are outside of the reach of NCrunch's core allocation. An example of this is VBCSCompiler.exe, which is typically shared by MSBuild tasks for compile steps. Where core allocation is not effective, the O/S's thread scheduler takes full control of affinity and may do so with unpredicable results.

Recommendations

Enabling this setting will instruct the engine to attempt to take control over the usage of CPU resources for processes that it controls. When the engine controls CPU resources, it can work to prevent tests from overstepping their expected resources (through multi-threaded or multi-process execution). Overstepping can result in tests consuming far more CPU resources than expected, which can cause resource starvation on the machine, resulting in other test runs timing out.

When CPU resources are controlled by the engine, the O/S is prevented from pooling them between tasks. This can result in decreased overall throughput in testing cycles due to underutilisation.

In most situations, there is no need to enable this setting. Sensible use of UsesThreadsAttribute can allow the engine to compensate for overstepping. However, some teams may find this setting useful when trying to address issues with test timeouts in resource constrained environments, particularly when making use of RDI.