Set up Running C# Selenium Tests in BuildWise CT Server

How to run C# Selenium Tests in BuildWise, a Continuous Testing server

This article shows you how to set up Selenium WebDriver tests written in C# on a Continuous Testing (CT) server, BuildWise.

Before you start

Make sure you have .NET installed (version 7 is fine), some C# tests and BuildWise ready to go.

1. Install .NET

The below article has instructions to install .NET. Skip the VS Code specific steps as we won’t be using VS Code here.

Set up and Run Selenium C# Tests in Visual Studio Code on macOS

Note: Make sure you restart your machine after .NET is installed.

2. Clone Sample Selenium C# tests

There are sample C# Selenium tests available here, under the e2e-vstest-selenium. Alternatively, use your own Selenium C# tests.

3. Install and run BuildWise

Then, install BuildWise, a CT server. You can follow this article’s 3rd and 4th steps to install and run the server.

Set Up the CT Server for C#

1. Test on the command line first

Compile the tests with dotnet build, then run them to verify the tests actually execute. If you are using the sample tests, you can use rake ui_tests:quick to run the tests and verify your .NET setup.

If you see an error like:

Could not find a test logger with AssemblyQualifiedName, URI or FriendlyName 'junit'.

You need to add the JunitXml.TestLogger package to your project and recompile the project. Add this via NuGet Package Manager, or simply add this line to your csproj file:

<PackageReference Include="JunitXml.TestLogger" Version="3.0.124"/>

2. Set up BuildWise project

Create a BuildWise project like you normally would. Avoid using special characters in project name/identifier (i.e. use “Sharp” instead of “#”). You can ignore the “Test framework” dropdown for now.

It is not necessary to specify “UI Test Folder” because the test script classes don’t execute (instead, the compiled classes do).

3. Add a Compile task

Go to the project settings (click the project name on the home page).

Add a new build step to run dotnet build.

Then in the drop-down under Build Steps, select VSTest (C#).

4. Start a build by clicking “Build Now”

Check that the Git Pull and Compile step were successful.

If you were using the sample e2e-vstest-selenium, your UI tests have run successfully. However, if not, they probably didn’t. This is because the standard Rakefile does not work for C# yet.

The rest of the article show you how to update your Rakefile to make C# tests supported.

Customising the Rakefile

Support Compiled Languages

Add the following line to the top of the Rakefile.

$ignore_checking_test_file_exists = true 

This tells buildwise.rake (sub module) to not check if the file exists before including it in the test execution. This mode is only applies to test script in non-compiled languages. Otherwise, you will get an empty file list.

Add a function to extract Test Class List

In C#, after building, the concept of test file names are gone in the context test execution. We need to get a list of TestClass in DLL, which you can get by invoking avstest command.

Here a function (in Ruby) to get that.

def get_test_classes_list
  tmp_dir = File.expand_path File.join(File.dirname(__FILE__),  "tmp")  
  FileUtils.rm_rf(tmp_dir)if File.exist?(tmp_dir)
  FileUtils.mkdir_p(tmp_dir)
  
  project_dir = File.expand_path File.dirname(__FILE__)
  FileUtils.chdir(project_dir)
  cmd = "dotnet vstest  #{dll_file} /ListFullyQualifiedTests /ListTestsTargetPath:#{tmp_dir}/Tests.txt"  
  output = `#{cmd}`
  
  puts "Generate qualified test case names to file: #{tmp_dir}/Tests.txt}"
  sleep 0.1
  test_lines = []
  File.read("#{tmp_dir}/Tests.txt").split.each  do |line|
    test_class = File.basename(line, File.extname(line))
    test_lines << test_class
  end
  test_lines.uniq!
  test_lines.sort!
  return test_lines
end

The above snippet uses vstest ListFullyQualifiedTests/ to retrieve test case names. Then, using Ruby, extracts the test case names out.

SeleniumRecipes.AssertionTest.TestAssertTitle => SeleniumRecipes.AssertionTest
SeleniumRecipes.AssertionTest.TestAssertPageText =>SeleniumRecipes.AssertionTest

Then applyuniq and sort to get test classes in alphabetical order.

Saving the Test Result

BuildWise relies on the test result in order to display the UI Test details. After each test class executes, save the test results (JUnit XML) to a specified folder.

  specs_to_be_executed = buildwise_determine_specs_for_quick_build(all_test_classes, excluded_spec_files);
  specs_to_be_executed.each do |test_class|    
    cmd = "dotnet test  #{dll_file}  --filter #{test_class} --logger 'junit;LogFilePath=#{reports_dir}/TestResult-#{test_class}.xml'"

    # TODO, might be vstest better, but generate trx format, junit need another assembly        
    # cmd = "dotnet vstest ./bin/Debug/net7.0/e2e-vstest-selenium.dll /Tests:#{test_class} /Logger:trx"

    # cmd = "dotnet vstest ./bin/Debug/net7.0/e2e-vstest-selenium.dll /Tests:#{test_class} /logger:JUnitLogger;TestResultsFile==./#{reports_dir}/TestResult-#{test_class}.xml'"
    puts cmd
    result = system(cmd)
    if exit_code == 0 && !result
      exit_code = -1
    end   
    sleep 0.25
  end

Run the test dotnet test for each test class and generate a Junit report under reports_dir with name TestResults-{CLASS}.xml

BuildWise will take care of the rest!

Why use dotnet test, not dotnet vstest?
By default, vstest comes with trx format logger. It’s still XML, but different. For consistency, I have used dotnet test and the JUnitLogger.

These are all of the changes required to get your C# Selenium tests working on the BuildWise CT server. If you get stuck, a working Rakefile is available in the buildwise-samples repository.

A Sample Test Report on BuildWise Server