Using native DLL and resource files in BenchmarkDotNet projects

In a previous post, I described the NativeMemoryProfiler which I implemented in the BenchmarkDotNet library. It caused that I got many questions about using native DLL files in benchmark projects. Many people want to use post-build events but they do not work. There is an issue dotnet/BenchmarkDotNet#946. I know a better solution, so I decided to describe this topic.

In this post, I would like to show you, how to use native DLL or resource files in benchmark projects.

If you only need solution, you can navigate to the solution section, but if you would like to understand the problem you need to read the whole article.

The story

When you run your benchmark project, the BenchmarkDotNet generates an isolated project that references to your project. This generated project is built and run, during each run of your benchmark. Let’s create a really simple benchmark project that shows us this mechanism.

In the beginning, we have to create a new project. I’m going to use dotnet CLI, so that you can follow the steps below on Windows, Linux or macOs.

mkdir UsingResourcesWithBenchmarkDotNet
cd UsingResourcesWithBenchmarkDotNet
dotnet new console

Then we need to add the BenchmarkDotNet nuget to this project.

dotnet add package BenchmarkDotNet

After configuring the project, we can implement our benchmark. Below is a Program.cs file:

using System;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

namespace UsingResourcesWithBenchmarkDotNet
{
    class Program
    {
        static void Main(string[] args) =>
            BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
    }

    public class Benchmark
    {
        [Benchmark]
        public int Sum()
        {
            int result = 0;
            for (int i = 0; i < 100; i++)
            {
                result += i;
            }

            return result;
        }
    }
}

Now we can run this project with a --keepfiles option. If you add this option, all auto-generated files will be kept after running the benchmark.

dotnet run -c Release -- --filter *Sum* --keepfiles

Now, bin directory should contain all generated files. In our case, we were run only one benchmark so all files should be in DefaultJob directory.

λ cd bin\Release\netcoreapp3.0\DefaultJob
λ ls  # or dir for windows console
BenchmarkDotNet.Autogenerated.csproj  DefaultJob.bat  DefaultJob.notcs  bin/  obj/

As you can see, BenchmarkDotNet generates a new project for us, calledBenchmarkDotNet.Autogenerated.csproj. For instance, when you want to run your benchmark for many frameworks, BenchmarkDotNet will generate many projects for you. Each generated project has its own bin directory and is run from this location.

Since BenchmarkDotNet does not run benchmarks from default bin directory, you can not use any recourses from there. Therefore, post-build events do not work because they are running during building the benchmark project, not auto-generated projects.

Solution

The solution is really simple. Instead of post-build events, you should use None or Compile project item. Below you can see an example of copying NativeDll.dll from solutionDir\x64\Release directory into bin directory of the benchmark project, as well as into bin directories of all auto-generated projects that refer to the benchmark project. If you set Visible to true, then the file will be visible in the solution explorer.

<ItemGroup>
  <None Include="$(MSBuildThisFileDirectory)..\x64\$(Configuration)\NativeDll.dll">
    <Link>NativeDll.dll</Link>
    <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    <Visible>True</Visible>
  </None>
</ItemGroup>

What’s more, you do not have to add file by file. You can also copy the entire directory using this method.

  <ItemGroup>
    <None Include="$(MSBuildThisFileDirectory)..\Resources\**\*.*">
      <Link>%(Directory)\%(Filename)%(Extension)</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>True</Visible>
    </None>
  </ItemGroup>

You can also copy all files of a given type, for example, all DLL files from a directory:

  <ItemGroup>
    <None Include="$(MSBuildThisFileDirectory)..\Resources\**\*.dll">
      <Link>%(Directory)\%(Filename)%(Extension)</Link>
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <Visible>True</Visible>
    </None>
  </ItemGroup>

Summary

If you are creating a project with benchmarks using BenchmarkDotNet, you should include additional files as if you were creating a nuget package.

If you think this post is useful, let me know in the comments below.