My current approach to Context/Specification

One of my most popular posts from last year is an article on test naming guidelines, which was written to resemble the format used by the Framework Design Guidelines.  Despite the popularity of the article, I started to stray from those guidelines over the last year.  While I haven’t abandoned the philosophy of those guidelines, in fact most of the general advise still applies, I’ve begun to adopt a much different approach for structuring and organizing my tests, and as result, the naming has changed slightly too.

The syntax I’ve promoted and followed for years, let’s call it TDD or original-flavor, has a few known side-effects:

• Unnecessary or complex Setup – You declare common setup logic in the Setup of your tests, but not all tests require this initialization logic.  In some cases, a test requires entirely different setup logic, so the initial “arrange” portion of the test must undo some of the work done in the setup.
• Grouping related tests – When a complex component has a lot of tests that handle different scenarios, keeping dozens of tests organized can be difficult.  Naming conventions can help here, but are masking the underlying problem.

Over the last year, I’ve been experimenting with a Behavior-Driven-Development flavor of tests, often referred to as Context/Specification pattern.  While it addresses the side-effects outlined above, the goal of “true” BDD is to attempt to describe requirements in a common language for technical and non-technical members of an Agile project, often with a Given / When / Should syntax.  For example,

Given a bank account in good standing, when the customer requests cash, the bank account should be debited

The underlying concept is when tests are written using this syntax, they become executable specifications – and that’s really cool.  Unfortunately, I’ve always considered this syntax to be somewhat awkward, and how I’ve adopted this approach is a rather loose interpretation.  I’m also still experimenting, so your feedback is definitely welcome.

An Example

Rather than try to explain the concepts and the coding style in abstract terms, I think it’s best to let the code speak for itself first and then try reason my way out.

Note: I’ve borrowed and bended concepts from many different sources, some I can’t recall where.  This example borrows many concepts from Scott Bellware’s specunit-net.

public abstract class ContextSpecification
{
    [TestFixtureSetUp]
    public void SetupFixture()
    {
       BeforeAllSpecs();
    }
    
    [TestFixtureTearDown]
    public void TearDownFixture()
    {
       AfterAllSpecs();
    }

     [SetUp]
     public void Setup()
    {
       Context();
       Because();
    }

    [TearDown]
    public void TearDown()
    {
       CleanUp();
    }

    protected virtual void BeforeAllSpecs() { }
    protected virtual void Context() { }
    protected virtual void Because() { }
    protected virtual void CleanUp() { }
    protected virtual void AfterAllSpecs() { }   
}

public class ArgumentBuilderSpecs : ContextSpecification
{
    protected ArgumentBuilder builder;
    protected Dictionary<string,string> settings;
    protected string results;

    protected override void Context()
    {
       builder = new ArgumentBuilder();
       settings = new Dictionary<string,string>();
    }

    protected override void Because()
    { 
       results = builder.Parse(settings);
    }

    [TestFixture]
    public class WhenNoArgumentsAreSupplied : ArgumentBuilderSpecs
    {
       [Test]
       public void ResultsShouldBeEmpty()
       {
           results.ShouldBeEmpty();
       }
    }

    [TestFixture]
    public class WhenProxyServerSettingsAreSupplied : ArgumentBuilderSpecs
    {
       protected override void Because()
       {
           settings.Add("server", "proxyServer");
           settings.Add("port", "8080");

           base.Because();
       }

       [Test]
       public void ShouldContainProxyServerArgument()
       {
           results.ShouldContain("-DhttpProxy:proxyServer");
       }

       [Test]
       public void ShouldContainProxyPortArgument()
       {
           results.ShouldContain("-DhttpPort:8080");
       }
  }
}

Compared to my original flavor, this new bouquet has some considerable differences which may seem odd to an adjusted palate.  Let’s walk through those differences:

• No longer using Fixture-per-Class structure, where all tests for a class reside within a single class.
• Top level “specification” ArgumentBuilderSpecs is not decorated with a [TestFixture] attribute, nor does it contain any tests.
• ArgumentBuilderSpecs derives from a base class ContextSpecification which controls the setup/teardown logic and the semantic structure of the BDD syntax.
• ArgumentBuilderSpecs contains the variables that are common to all tests, but setup logic is kept to a minimum.
• ArgumentBuilderSpecs contains two nested classes that derive from ArgumentBuilderSpecs.  Each nested class is a test-fixture for a scenario or context.
• Each Test-Fixture focuses on a single action only and is responsible for its own setup.
• Each Test represents a single specification, often only as a single Assert.
• Asserts are made using Extension methods (not depicted in the code example)

Observations

Inheritance

I’ve never been a big fan of using inheritance, especially in test scenarios as it requires more effort on part of the future developer to understand the test structure.  In this example, inheritance plays a part in both the base class and nested classes, though you could argue the impact of inheritance is negated since the base class only provides structure, and the derived classes are clearly visible within the parent class.  It’s a bit unwieldy, but the payoff for using inheritance is found when viewing the tests in their hierarchy:

While technically you could achieve a similar effect by using namespaces to group tests, but you lose some of the benefits of encapsulation of test-helper methods and common variables.

Although we can extend our contexts by deriving from the parent class, this approach is limited to inheriting from the root specification container (ArgumentBuilderSpecs).  If you were to derive from WhenProxyServerSettingsAreSupplied for example, you would inherit the test cases from that class as well.  I have yet to find a scenario where I needed to do this.  While the concepts of DRY make sense, there’s a lot to be said about clear intent of test cases where duplication aids readability.

Extra Plumbing

There’s quite a bit of extra plumbing to be able to create our nested contexts, and it seems to take a bit longer to write tests.  This delay is either caused by grappling with new context/specification concepts, writing additional code for subclasses or more thought determining which contexts are required.  I’m anticipating that it gets easier with more practice, and some Visual Studio code snippets might simplify the authoring process.

One area where I can sense I’m slowing down is trying to determine if I should be overriding or extending Context versus Because.

Granular Tests

In this style of tests, where a class is created to represent a context, each context performs only one small piece of work and the tests serve as assertions against the outcome.  I found that the tests I wrote were concise and elegant, and the use of classes to structure the tests around the context helped to organize both the tests and the responsibility of the subject under test.  I also found that I wrote fewer tests with this approach – I write only a few principle contexts.  If a class starts to have too many contexts, it could mean that the class has too much responsibility and should be split into smaller parts.

Regarding the structure of the tests, if you’re used to fixture-per-class style of tests, it may take some time to get accustomed to the context performing only a single action.  Though as Steven Harman points out, visualizing the context setup as part of the fixture setup may help guide your transition to this style of testing.

Conclusion

I’m enjoying writing Context/Specification style tests as they read like a specification document and provide clear feedback of what the system does.  In addition, when a test fails, the context it belongs to provides additional meaning around the failure.

There appear to be many different formats for writing tests in this style, and my current format is a work in progress.  Let me know what you think.

Presently, I’m flip flopping between this format and my old habits.  I’ve caught myself a few times where I write a flat structure of tests without taking context into consideration – after a point, the number of tests becomes unmanageable, and it becomes difficult to identify if I should be tweaking existing tests or writing new ones.  If the tests were organized by context, the number of tests becomes irrelevant, and the focus is placed on the scenarios that are being tested.