Wednesday, October 21, 2009

Running MS Team System tests in NUnit

Earlier this week I saw this question on Stack Overflow asking if it was possible to run MS Team System Tests under NUnit.  At first, this sounds like a really odd request.  After all, why not just convert your tests to NUnit and be done with it? 

I can think of a few examples where you may need interoperability – for example, two teams collaborating where team #1 has corporate policy for MSTest, while team #2 doesn’t have access to a version of Visual Studio with Team System.  Being able to target one platform but have the tests run in either environment is desirable.  Whatever the reason, it’s an interesting code challenge.

Fortunately, the good folks at Exact Magic Software published an NUnit addin that can adapt Team System test fixtures to NUnit.  Since NUnit identifies candidate fixtures simply by examining attribute names, the addin’s job is fairly straight forward, but it also recognizes MS specific features such as TestContext and DataSource driven tests.

Unfortunately, the NUnit adapter uses features that reside within the nunit.core assembly which ties it to a specific version of NUnit.  In this case, the addin will only work with NUnit 2.4.6.  This is a frustrating design problem that plagues many NUnit addins, including my own open source project.

Note: NUnit 3.0 plans to solve this problem by moving the boundary between framework and test runner so that each version of the framework knows how to run it’s own tests.  The GUI or host application will be able to load and execute the tests in a version independent manner.  Hopefully this means that addins will target and execute within a specific framework version, but will work in different versions of the user-interface.

The current version of NUnit (2.5.2) is a stepping stone between the 2.4.x framework and the upcoming overhauled 3.0 version.  As part of this transition, there’s a lot of breaking changes between older versions.  For me, this translates into a problem where Exact Magic’s project simply won’t compile if you update the dependencies.  Since I need to do similar work for my own project, this was good exercise.

I’ve reworked their code and put it here.

A few known caveats:

  • The unit tests for the adapter attempt to perform reflection calls to internals fields or methods within the NUnit.Core that have been renamed or no longer exist.  I haven’t bothered fixing these tests.
  • Does not appear to have support for AssemblyInitialize or AssemblyCleanup, though these could be adapted by adding an EventListener when the test run start and finish.  I may add this feature.
  • The adapter doesn’t wrap functionality contained within MS Test, it simulates it’s behavior.  While it has the same result, it won’t capture the exact same nuances of Microsoft’s implementation.

If you have any concerns or questions, let me know.

Cheers.

2 comments:

  1. Hey Bryan. I've got a bunch of .NET 4 projects with MSTests that I'd love to be able to run with NUnit through this addin, but 2.5.2 won't load their assemblies because it won't run on .NET 4. I was thinking I'd update the addin, but I worried about two things:

    1) You reported several tests failing when you built it. I'm seeing 18 failures across both test projects while building as-is from your SVN sources--no changes of my own yet.
    2) You mentioned that there are dependencies on the specific NUnit version I want to run against. Since 3.0 still isn't out (and no idea if the newer addin model you described made it into the alpha that's available right now), should I just build against 2.6.3 (latest) and cross my fingers?

    I know it's been a while since you worked on this, but any pointers you can give will be greatly appreciated. I have >5k unit tests I need to show running as-is on NUnit as a proof of concept before I can start writing actual NUnit tests.

    ReplyDelete
  2. Hey Jesse,

    I think enough has changed since 2.5.2 and 2.6.3 where NUnit has added support for .NET 4/4.5, etc.

    To my understanding, Charlie Poole has a NUnit adapter for Visual Studio 2012 here: http://nunit.org/index.php?p=vsTestAdapter&r=2.6.3

    ReplyDelete