Monday, October 29, 2012

Unit Testing your Repository Query Logic

  • Have you ever tried to unit test your Query Logic using repositories??

  • If you do, have you ended writing integration tests instead of unit tests??


Even if you have not tried, you might find this post useful.


I´m going to show you a simple way to unit test your Query Logic. It doesn’t matter what back-end system you are using (relational database, in-memory database, xml file, etc etc etc) nor what ORM tool you are using (EF, NHibernate) as long as you encapsulate the code to interact with these external services using an independent component and that you can defer query execution on your back-end system using an IQueryable object.


In order to do it, we need to write the code in a test-friendly way.


Let´s begin with some concepts and principles related to this problem.


  • SRP. Single Responsibility Principle. I believe this is the most important principle out there that should be followed. When you break it, you are creating code that is going to be harder to test. There is however one exception to the rule, when you are writing a controller and I don’t mean a MVC controller, I mean the concept used in the more general sense of the controller in Responsibility Driven Design as a Role Stereotype

  • DI. Dependency Injection


    Dependency injection is a software design pattern that allows a choice of component to be made at run-time rather than compile time


    Dependency Injection as specified before is a Design Pattern and DI by itself does not require the use of a IoC container although, it is used with one most of the time but this is not required.

  • Law of Demeter


    The Law of Demeter (LoD) or Principle of Least Knowledge is a design guideline for developing software, particularly object-oriented programs. In its general form, the LoD is a specific case of loose coupling


    • Each unit should have only limited knowledge about other units: only units "closely" related to the current unit.
    • Each unit should only talk to its friends; don't talk to strangers.
    • Only talk to your immediate friends.

For more info watch this Misko Hevery video


So let’s see how a typical Query repository looks like:


public interface IQueryRepository
{
    IQueryable<Movie> GetMovies(string title);
}

public class QueryReprository : IQueryRepository
{
    public IQueryable<Movie> GetMovies(string title)
    {
        var ctx = new MoviesContext();
        var movies = ctx.Movies.AsQueryable();

        if (!string.IsNullOrWhiteSpace(title))
        {
            movies = movies.Where(x => x.Title.Contains(title));
        }

        return movies;
    }
}

Looks familiar? If you write a repository like this, you won't be able to test your query logic in isolation.


There are basically four problems with this implementation that prevent our code to be unit tested:


  • We are using the evil anti-test keyword new inside our repository to create the EF context

  • Using the EF context directly, to retrieve the IQueryable<Movie> collection to perform the query on it instead of just recieve the collection itself

  • Even when the GetMovies method looks extremely simple, it is violating the SRP. This method is doing too much work.


    • It is creating the EF context

    • It is getting the Movies collection

    • It implements the query logic

  • The GetMovies violates the Law of Demeter, because it’s using directly the EF context, which doesn't actually requires. What we need is to work directly with the movies collection instead


All these problems make impossible to unit test our query logic, if you have repositories like this one, you would have to write integration tests instead.


Refactoring the ugly code


Now let’s refactor this simple code to make it test-friendly.


Removing the new operator


In order to remove the new operator we need to use DI.


First we need to write a custom factory in charge to create the EF context for us. (I wrote a blog about a thread-safe implementation)


This is a simplified version


public interface IContextResolver
{
    TContext GetCurrentContext<TContext>() where TContext : DbContext;

    void ReleaseContext();
}

public class ContextResolver : IContextResolver
{
    private HttpContextBase httpContextBase;
    private string connectionString;
    private const string ContextPerRequestItemName = "EF_Context_Per_Request_Item_Name";

    public ContextResolver(HttpContextBase httpContextBase, string connectionString = "")
    {
        Condition.Requires(httpContextBase).IsNotNull();

        this.httpContextBase = httpContextBase;
        this.connectionString = connectionString;
    }

    public TContext GetCurrentContext<TContext>() where TContext : DbContext
    {
        var ctx = this.GetContext<TContext>();

        if (ctx == null)
        {
            ctx = this.CreateContext<TContext>();
        }

        Condition.Ensures(ctx).IsNotNull();
        return ctx;
    }

    public void ReleaseContext()
    {
        var ctx = this.GetContext<DbContext>();

        if (ctx != null)
        {
            ctx.Dispose();
        }
    }

    private TContext GetContext<TContext>() where TContext : DbContext
    {
        return this.httpContextBase.Items[ContextPerRequestItemName] as TContext;
    }

    private TContext CreateContext<TContext>() where TContext : DbContext
    {
        var ctx = default(TContext);

        if (string.IsNullOrWhiteSpace(this.connectionString))
        {
            ctx = (TContext)Activator.CreateInstance(typeof(TContext));
        }
        else
        {
            ctx = (TContext)Activator.CreateInstance(typeof(TContext), this.connectionString);
        }

        this.httpContextBase.Items[ContextPerRequestItemName] = ctx;

        Condition.Ensures(ctx).IsNotNull();
        return ctx;
    }
}

This class is simply acting as a generic factory to instantiate EF Contexts using the Context Per Request pattern


Isolate the EF Context from our repository


This might seem odd at first, but when you clearly identify and separate the responsibilities of each object, your code will become automatically easier to unit-test


So let’s start by identifying what needs to be separated.


The problem in the last code sample is that the repository communicates directly with the EF context to retrieve the IQueryable collection of objects in order to create the query.


What we really need is the collection of objects to work on


Since the IQueryable interface is capable of execute deferred queries, it’s perfect to expose the initial collection of objects to perform the query.


So let’s add the following objects


public interface IMoviesDataProvider
{
    IQueryable<Movie> GetMoviess();
}

This simple interface will be in charge to provide the initial objects as an IQueryable collection to perform the query


public class MoviesDataProvider : IMoviesDataProvider
{
    private IContextResolver contextResolver;

    public MoviesDataProvider(IContextResolver contextResolver)
    {
        this.contextResolver = contextResolver;
    }

    public IQueryable<Movie> GetMoviess()
    {
        var context = this.contextResolver.GetCurrentContext<MoviesContext>();
        var items = context.Movies.AsQueryable();

        Condition.Ensures(items).IsNotNull();
        return items;
    }
}

Notice how the implementation has become really simple. This implementation will be in charge to communicate directly to the backend system to expose the data as an IQueryable collection. Since this object will talk directly with the database using EF, we use DI to pass a reference to the IContextResolver object (remember, this object will be in charge to get the correct instance of the EF context using the Context per Request Pattern)


Implement the Query Logic


The next step is to implement your query logic in your repository object.


This is quite simple since now we have identified and isolated single responsibilities in several objects.


So let’s refactor our repository code:


public class MoviesRepository : IMoviesRepository
{
    private IMoviesDataProvider moviesDataProvider;

    public MoviesRepository(
        IMoviesDataProvider moviesDataProvider)
    {
        this.moviesDataProvider = moviesDataProvider;
    }

    public QueryResults<Movie> FilterMoviesByTitle(string title, PagingInfo pagingInfo = null)
    {
        var movies = this.moviesDataProvider.GetMoviess();

        if (!string.IsNullOrWhiteSpace(title))
        {
            movies = movies.Where(x => x.Title.Contains(title));
        }

        return QueryResults.Of(movies.ToList(), movies.Count());
    }
}

    public interface IMoviesRepository
    {
        QueryResults FilterMoviesByTitle(string title, PagingInfo pagingInfo = null);
    }


There are several things to note on this simple code:


  • The repository no longer depends on the EF Context, now it depends on the IMoviesDataProvider interface which can now be easily mocked to return a dummy in-memory collection to work on when unit testing.

  • The Query Logic is being encapsulated in this layer

  • The result of the query is an in-memory object IList (which is wrapped in the QueryResults object) which means that at this point the query is being executed on the server.

  • Since the query has been executed at this point, we can safely return in-memory objects represented by the IList collection


Some people would argue about the use of the IList instead of returning an IQueryable.


With an IQueryable you get more flexibility since you can actually keep modifying the query all the way until the GUI, but in essence, that’s the problem. Since the query will be spread out across the application, there’s no way to test it in isolation.


For more info about IQueryable


Here is the code of the QueryResults and PagingInfo objects:


public sealed class QueryResults
{
    public static QueryResults<TTarget> Of<TTarget>(IList<TTarget> results, int virtualRowsCount)
    {
        Condition.Requires(results).IsNotNull();
        Condition.Requires(virtualRowsCount).IsGreaterOrEqual(0);

        return new QueryResults<TTarget>(results, virtualRowsCount);
    }
}

public class QueryResults<TTarget>
{
    public QueryResults(IList<TTarget> results, int virtualRowsCount)
    {
        Condition.Requires(results).IsNotNull();
        Condition.Requires(virtualRowsCount).IsGreaterOrEqual(0);

        this.VirtualRowsCount = virtualRowsCount;
        this.Results = results;
    }

    public int VirtualRowsCount { get; protected set; }

    public IList<TTarget> Results { get; protected set; }
}

public sealed class PagingInfo
{
    public PagingInfo(
        int page = 1, 
        int pageSize = 10, 
        string orderByField = "", 
        OrderDirection orderDirection = DataLayer.OrderDirection.Ascending)
    {
        Condition.Requires(page).IsGreaterOrEqual(1);
        Condition.Requires(pageSize).IsGreaterOrEqual(1);;

        this.Page = page;
        this.PageSize = pageSize;
        this.OrderByField = orderByField;
        this.OrderDirection = orderDirection;
    }

    public int Page { get; private set; }

    public int PageSize { get; private set; }

    public string OrderByField { get; private set; }

    public OrderDirection OrderDirection { get; private set; }
}

These objects are simple DTO’s


Writing the Unit Tests


Finally we can have fun writing our unit tests:


[TestClass]
public class MoviesRepositoryTests
{
    [TestClass]
    public class TheFilterByTitleMethod
    {
        [TestMethod]
        public void it_should_not_apply_a_filter_when_the_title_is_empty()
        {
            var fixture = new Fixture().Customize(new AutoMoqCustomization());
            var moviesDataProviderMock = fixture.Freeze<Mock<IMoviesDataProvider>>();
            var movies = Builder<Movie>.CreateListOfSize(10).Build().AsQueryable();

            moviesDataProviderMock.Setup(x => x.GetMoviess()).Returns(movies);

            var sut = fixture.CreateAnonymous<MoviesRepository>();

            sut.Should().NotBeNull();

            var res = sut.FilterMoviesByTitle(string.Empty, null);

            res.Should().NotBeNull();
            res.VirtualRowsCount.Should().Be(10);
            res.Results.Should().NotBeNull().And.NotBeEmpty().And.NotContainNulls()
                .And.OnlyHaveUniqueItems().And.ContainInOrder(movies);
        }

        [TestMethod]
        public void it_should_apply_the_filter_when_the_title_parameter_is_not_null()
        {
            var fixture = new Fixture().Customize(new AutoMoqCustomization());
            var moviesDataProviderMock = fixture.Freeze<Mock<IMoviesDataProvider>>();
            var numberOfElements = 10;
            var numberOfMatchingTitles = 3;
            var title = "Testing";
            var movies = Builder<Movie>.CreateListOfSize(numberOfElements)
                .TheFirst(numberOfMatchingTitles)
                    .With(x => x.Title, title)
                .Build().AsQueryable();
            var matchingMovies = movies.Where(x => x.Title.Contains(title));

            moviesDataProviderMock.Setup(x => x.GetMoviess()).Returns(movies);

            var sut = fixture.CreateAnonymous<MoviesRepository>();

            var res = sut.FilterMoviesByTitle(title);

            res.Should().NotBeNull();

            res.VirtualRowsCount.Should().Be(numberOfMatchingTitles);
            res.Results.Should().NotBeNull().And.NotBeEmpty().And.NotContainNulls()
                .And.OnlyHaveUniqueItems().And.ContainInOrder(matchingMovies);
            res.Results.Count.Should().Be(numberOfMatchingTitles);
        }
    }
}

Have fun!

Thursday, October 25, 2012

EntityFramework context per request-lifetime thread-safe with Unit Tests

When using EntityFramework, you need a consistent way to manage the context object lifecycle. The most accepted way to do this is creating a context per HTTP Request


I´m going to show you a basic thread-safe implementation.


Building the component


Context Resolver Contract


public interface IContextResolver
{
    TContext GetCurrentContext<TContext>() where TContext : DbContext;

    void ReleaseContext();
}

This interface is really simple, it just defines two methods.


  • The GetCurrentContext method will be used to get or create the EntityFramework context object.

  • The ReleaseContext method will be in charge to call Dispose in your context in order to release resources.


Context Resolver implementation


public class ContextResolver : IContextResolver
{
    private static object syncRoot = new Object();
    private string connectionString;
    private volatile HttpContextBase httpContextBase;
    private const string ContextItemName = "Context_Per_Request";

    public ContextResolver(HttpContextBase httpContext, string connectionString)
    {
        Condition.Requires(httpContext).IsNotNull();
        Condition.Requires(connectionString).IsNotNullOrWhiteSpace();

        this.connectionString = connectionString;
        this.httpContextBase = httpContext;
    }

    public TContext GetCurrentContext<TContext>() where TContext : DbContext
    {
        TContext context = default(TContext);

        lock (syncRoot)
        {
            context = this.httpContextBase.Items[ContextItemName] as TContext;
        }

        if (context == null)
        {
            lock (syncRoot)
            {
                if (context == null)
                {
                    context = (TContext)Activator.CreateInstance(typeof(TContext), this.connectionString);
                    this.httpContextBase.Items.Add(ContextItemName, context);
                }
            }
        }

        Condition.Ensures(context).IsNotNull().IsOfType(typeof(TContext));
        return context;
    }

    public void ReleaseContext()
    {
        DbContext context = default(DbContext);

        lock (syncRoot)
        {
            context = this.httpContextBase.Items[ContextItemName] as DbContext;
        }

        if (context != null)
        {
            context.Dispose();
        }
    }
}

The implementation is straightforward, the only tricky part is the addition of lock statements when reading and writing the EntityFramework context.


The C# specification clearly defines what operations are "atomic" in section 5.5. The atomic operations are reads and writes of variables of any reference type, or, effectively, any built-in value type that takes up four bytes or less


Source


In order to implement the context per HTTP request pattern, we use HttpContext.Items. The objects we store in this dictionary, will be accessible only during the current HTTP request.


Gets a key/value collection that can be used to organize and share data between an IHttpModule interface and an IHttpHandler interface during an HTTP request.


Usage


You just have to configure your DI container (if any) to inject the correct instance of IContextResolver.


Injecting


For simplicity I will inject it in my MVC controller


private IContextResolver contextResolver;

public HomeController(IContextResolver contextResolver)
{
    this.contextResolver = contextResolver;
}

public ActionResult SimpleData()
{
    var ctx = this.contextResolver.GetCurrentContext<pubsEntities>();

    return View(ctx.jobs);
}

In Global.asax


During the whole request, all the calls to IContextResolver.GetCurrentContext are going to return the same context instance, we now need to dispose the context after the request has been processed. We can do this in the PostRequestHandlerExecute application event in the Global.asax file.


public MvcApplication()
{
    this.PostRequestHandlerExecute += MvcApplication_PostRequestHandlerExecute;
}

void MvcApplication_PostRequestHandlerExecute(object sender, EventArgs e)
{
    var resolver = ServiceLocator.Current.GetInstance<IContextResolver>();

    resolver.ReleaseContext();
}

Notice the use of the ServiceLocator... your first impression might be: “hey dude, that’s an anti-pattern”...


Most of the time it is an anti-pattern, but in this case, we are using it inside the Composition Root Object of the Web Application, therefore we are allowed to use the container, actually I could change the code to use directly my DI container (Ninject - Kernel in this case), but this will get me the same result, even though, I would be adding an explicit dependency to Ninject.


Unit Testing


It’s time to show you the unit tests I created for this small but useful component


Builder


I tend to use builders to create my SUT objects, it seems like a consistent way to create testing objects


public class ConcreteResolverBuilder
{
    private ContextResolver Instance { get; set; }

    private static volatile HttpContextBase httpContextBase;
    private static volatile Dictionary<string, object> items;
    private static object syncRoot = new Object();

    public static HttpContextBase HttpContextBase
    {
        get
        {
            if (httpContextBase == null)
            {
                lock (syncRoot)
                {
                    if (httpContextBase == null)
                    {
                        var mock = new Mock<HttpContextBase>();

                        items = new Dictionary<string, object>();
                        mock.Setup(x => x.Items).Returns(items);

                        httpContextBase = mock.Object;
                    }
                }
            }

            httpContextBase.Should().NotBeNull();
            return httpContextBase;
        }
    }

    public ConcreteResolverBuilder Initialize()
    {
        this.Instance = new ContextResolver(
            HttpContextBase,
            ConfigurationManager.ConnectionStrings["Msts"].ConnectionString);

        return this;
    }

    public ContextResolver Build()
    {
        this.Instance.Should().NotBeNull();
        return this.Instance;
    }

    public static implicit operator ContextResolver(ConcreteResolverBuilder builder)
    {
        return builder.Build();
    }
}

I´m implementing a singleton pattern just because I want to mock an HttpContextBase object to simulate a real HTTP request


I´m also configuring the HttpContextBase.Items properties with Moq to return a Dictionary object


Note that since my intention is to simulate concurrent threads, I'm marking my HttpContextBase and Dictionary members as volatile


The Tests


[TestClass]
public class ContextResolverTests
{
    [TestClass]
    public class TheCreateContextInstanceMethod
    {
        [TestMethod]
        public void it_should_resolve_a_valid_instance()
        {
            ContextResolver sut = new ConcreteResolverBuilder().Initialize();
            var res = sut.GetCurrentContext<PubsContext>();

            res.Should().NotBeNull().And.BeOfType<PubsContext>();
        }

        [TestMethod]
        public void it_should_return_the_same_instance_even_when_calling_it_from_different_threads()
        {
            ContextResolver sut = new ConcreteResolverBuilder().Initialize();
            var firstResolvedContext = sut.GetCurrentContext<PubsContext>();
            var numberOfConcurrentProcess = 10000;
            var contextResolverInstances = new BlockingCollection<DbContext>();
            var tasks = new Task[numberOfConcurrentProcess];

            for (int i = 0; i < numberOfConcurrentProcess; i++)
            {
                tasks[i] = Task.Factory.StartNew(() =>
                {
                    var context = new ConcreteResolverBuilder().Initialize().Build().GetCurrentContext<PubsContext>();

                    contextResolverInstances.Add(context);
                }, TaskCreationOptions.LongRunning);
            }

            Task.WaitAll(tasks);

            contextResolverInstances
                .Should()
                    .NotBeNull()
                .And
                    .NotBeEmpty()
                .And
                    .HaveCount(numberOfConcurrentProcess)
                .And
                    .ContainItemsAssignableTo<DbContext>()
                .And
                    .NotContainNulls();

            var firstResolvedContextHashCode = (firstResolvedContext as object).GetHashCode();

            firstResolvedContextHashCode.Should().NotBe(default(int));

            foreach (var item in contextResolverInstances)
            {
                var itemHashCode = (item as object).GetHashCode();

                itemHashCode.Should().NotBe(default(int));
                itemHashCode.Should().Be(firstResolvedContextHashCode);
                item.Should().Be(firstResolvedContext);

                var areReferenceEquals = ReferenceEquals(item, firstResolvedContext);

                areReferenceEquals.Should().BeTrue();
            }
        }
    }
}

  • The first test is really simple, it just ensures that we can create a context

  • The second test is more interesting, this is where I’m actually testing the concurrency of my implementation.


    • I keep track of the first context created to compare all subsequent contexts with this one.


      var firstResolvedContext = sut.GetCurrentContext<PubsContext>();
      
    • I set the number of concurrent operations


      var numberOfConcurrentProcess = 10000;
      
    • Now I create a list to store all the created contexts. Note that I’m using the BlockingCollection object which is thread-safe


      var contextResolverInstances = new BlockingCollection<DbContext>();
      
    • I’m storing all the parallel tasks in an array


      var tasks = new Task[numberOfConcurrentProcess];
      
    • I create and start all the tasks. On each task, I’m creating a new instance of my SUT (my concrete implementation of: IContextResolver) and adding to contextResolverInstances the result of IContextResolver.GetCurrentContext. Also note that I'm using: TaskCreationOptions.LongRunning to force the TPL to run my tasks in background threads


      for (int i = 0; i < numberOfConcurrentProcess; i++)
                  {
                      tasks[i] = Task.Factory.StartNew(() =>
                      {
                          var context = new ConcreteResolverBuilder().Initialize().Build().GetCurrentContext<PubsContext>();
      
                          contextResolverInstances.Add(context);
                      }, TaskCreationOptions.LongRunning);
                  }
      
    • I wait for all the tasks to complete by calling:


      Task.WaitAll(tasks);
      
    • The rest of the code are assertions, to confirm that just one instance was created among all threads




In the next screenshot you can see how the tasks are being executed on different threads





Thursday, October 4, 2012

WCF REST Data service in Visual Studio 2012 Express for Web

If you are using Visual Studio Express for Web, you might notice that the WCF Data Service template is not available to use.


I’m going to explain a step-by-step guide to create a WCF Data Service.


  • Create an empty Web project with the name WcfDataServiceInExpressEditions:

  • Add the following nuget packages:


  • Add the following settings to the web.config right under <configuration>


    <system.webServer>
        <modules runAllManagedModulesForAllRequests="true"></modules>
    </system.webServer>
    <system.serviceModel>
        <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true" />
    </system.serviceModel>
    
  • Create an ADO.NET Entity Data Model and connect to any database you own and drag a couple of tables to the designer

  • Add a new WCF Service to your project name it: PubsService

  • Delete the file containing the interface IPubsService.cs

  • Change the declaration of your service as follows:


    public class PubsService : DataService<Pubs>
    {
        // This method is called only once to initialize service-wide policies.
        public static void InitializeService(DataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
            // Examples:
            config.SetEntitySetAccessRule("jobs", EntitySetRights.AllRead);
            // config.SetServiceOperationAccessRule("MyServiceOperation", ServiceOperationRights.All);
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V3;
        }
    }
    
  • Change the markup of the .svc file to include the correct factory:


    Factory="System.Data.Services.DataServiceHostFactory, Microsoft.Data.Services"
    

That’s it, run the application, and you will see something like this:



This is the list of all the tables configured in the InitializeService method


To test the queries try something like this:


http://localhost:55575/PubsService.svc/jobs/?$filter=min_lvl gt 150



Download the code of this article


Happy coding