Vasily Sizov. Funky life.

Blog of IT manager, Java EE architect and developer more...

Tuesday, March 31, 2009

Java Mock Frameworks Comparison

In this post I try to tell you what mock frameworks are and what features they provide, and compare existing frameworks.

What is mock?

Let’s look at the example.

As we can see on the diagram Class A depends on Class B and Class C. Class B depends on Class D and Class E.

Ok. What should we do for unit testing of Class A? First of all we should isolate Class A. We should provide for instance of Class A something looks like instances of Class B and Class C, but which give us an opportunity to check behavior of testing instance.

These replacements are called mocks. Simple mocks which can only react like replacement classes with some canned answers to invocations made during tests are called stubs.

The diagram for unit testing of Class A:

Mock frameworks

Mock frameworks are the frameworks that give us an easy way to create mocks. Usually working with mock frameworks includes the following steps:
  1. Creation of a mock. Usually it looks like this (in EasyMock):
    ClassB mock = createMock(ClassB.class);
  2. Definition of the stubbed methods (what the method should do when a call happens). Sometimes definition is combined with expectations. EasyMock example:
    expect(mock.go()).andThrow(new RuntimeException());
  3. Definition of the expectations (how many times this method will be called, etc). EasyMock example:
    expectLastCall().times(3);
  4. Execution of the test code.
  5. Verification of the expectations. EasyMock example:
    verify(mock);

Different frameworks provide their own ways to define the expectations and a variety of features to check the tested object’s behavior.

Features

Let’s make a quick overview of common features provided by mock frameworks.

Value returning for stubbed method

It’s the ability of stubbed method to return value when invocation happens.

Mockito example:

when(mock.isDone()).thenReturn(true);

Exception throwing for stubbed method

It’s the ability of stubbed method to throw an exception. Mockito example:

when(mock.do()).thenThrow(new CantDoException());

Invocations count check

It’s the ability to check count of mock’s method invocations.

EasyMock example:

expect(mock.run()).times(1, 3); // expecting at least one and maximum three invocations

Method arguments check

It’s the ability to validate parameters passed to mock’s method. It could be range check, type check, etc. jMock example:

allowing(calculatorMock).sqrt(with(lessThan(0)); will(throwException(new IllegalArgumentException());

Invocations order for one mock check

It’s the ability to check the order of invocations for one mock. For example, in EasyMock we should create StrictMock with createStrictMock() method. StrictMock will check invocation order automatically.

Iterator-style stubbing

It’s the ability of a stubbed method to return different values for each invocation. Mockito example:

when(mock.getWeekday("My event")).thenReturn("Monday", "Tuesday", "Thursday");

Real objects spy

It’s the ability of a mock-object to act like proxy. It checks the invocations and redirect them to the real object. This feature can be useful for legacy code testing.

Mockito example from documentation:


List list = new LinkedList();
List spy = spy(list);

//optionally, you can stub out some methods:
when(spy.size()).thenReturn(100);
//using the spy calls real methods
spy.add("one");
spy.add("two");

Callback invocation from stubbed method

It’s the ability to invoke custom code from a stubbed method. Mockito documentation example:


when(mock.someMethod(anyString())).thenAnswer(new Answer() {
Object answer(InvocationOnMock invocation) {
        Object[] args = invocation.getArguments();
        Object mock = invocation.getMock();
        return "called with arguments: " + args;
    }
});

Mocking of final classes, methods, static and private fields

This ability could be useful for legacy code testing.

Stubbing of equals() and hashcode() methods

Most of frameworks (if not all) redefine these methods for their own needs. So it’s the usual limitation.

Thread safe

It’s the ability to run test with mock objects in multiple threads, so the mock objects are not shared between threads.

Invocation order between mocks check

It’s the ability to check the order of invocations between several mocks. For EasyMock it can be reached using control object:

IMocksControl control= createStrictControl();
IMyInterface mock1 = control.createMock(IMyInterface.class);
IMyInterface mock2 = control.createMock(IMyInterface.class);

Then framework will remember invocation between these mocks.

Java Mock Frameworks Features Comparison

First of all, please notice that all information about frameworks was taken from their documentation and I can’t guarantee that it is 100% accurate. If you have found any mistakes please feel free to contact me.

jMock EasyMock Mockito SevenMock JMockit rMock Unitils
Actual version 2.6-RC1 (7 Dec 2008) 2.4 (30 Aug 2008) 1.7 (24 Jan 2009) 2.1.0 (17 Jun 2008) 0.94 (30 Jan 2008) 2.0.0 (18 Mar 2007) 2.2 (23 Dec 2008)
Website http://jmock.org http://easymock.org http://code.google.com/p/mockito http://seven-mock.sourceforge.net https://jmockit.dev.java.net http://rmock.sourceforge.net http://www.unitils.org
Features Specification-like definition of mocks’ methods behavior (looks like DSL). «Record/Replay/Verify» model. Basic expectations are defined via calling appropriate mocks’ methods with actual data. Other types of expectations are defined with framework’s methods. «Verify after run» model. Basic expectations are checked after calling (mocks remember their methods invocations). Other types of expectations are defined as rules before running. Method stubbing and expectations are defined using inner classes. Methods stubbing are made using inner classes. Annotations for those methods define basic expectations. Other expectations are defined using DSL-like style. «Record/Replay/Verify» model. Expectations are defined using helper methods. Mockito-like «Verify after run» model.
Value returning for stubbed method + + + +* + + +
Exception throwing for stubbed method + + + +* + + +
Invocations count check + + + -* + + +*
Method arguments check + + + -* + + +
Invocations order for one mock check + + + + - + +
Iterator-style stubbing + + + -* +* - +
Real objects spy - - + - - + -
Callback invocation from stubbed method + + + - - - +
Mocking of final classes, methods, static and private fields - - - - + - -
Mocking of equals() and hashcode() methods - - - -
Thread safe - + + -
Invocation order between mocks check + + + + - + -
Total score 6.7 7.5 8.3 3.5 4.8 5.8 5.6

* — feature can be implemented, but not supported directly by a framework

Syntax comparison

Framework review would be incomplete with features comparison alone. Tests should be easy to read, so syntax is a very important criterion. That’s why I want to demonstrate testing of some example code with these frameworks.

I like the example from EasyMock documentation, so I rewrite it for other frameworks using their documentation as a reference. Let’s assume we have a DocumentManager object which references to Collaborator obects. DocumentManager has

void addDocument(String document)
method. When this method is executed DocumentManager (like Event Source in Observer design pattern) notifies its Collaborators with
void documentAdded(String document)
method. Also DocumentManager has
Boolean removeDocument(String document)
method which inquires its Collaborators with
Integer voteForRemoval(String document)
method. If the sum of votes is larger than zero, DocumentManager notifies its Collaborators about document removal with method
void documentRemoved(String document)
then returns true.

It can look like this:


public class DocumentManager() {
List collaborators = new ArrayList();

    ...

    public void addDocument(String document) {
        ...

        for (Collaborator collaborator : collaborators) collaborator.documentAdded(document);
    }


    public boolean removeDocument(String document) {
        Integer votingResult = 0;
        for (Collaborator collaborator : collaborators) {
          votingResult += collaborator.voteForRemoval(document);
        }


        if (votingResult > 0) {
            ...

          for (Collaborator collaborator : collaborators) collaborator.documentRemoved(document);
          return true;
        }
    }
}

In following examples I avoid the setup phase and concentrate directly on testing. Check it out:

EasyMock


collaboratorMock.documentAdded("Document");
expect(collaboratorMock.voteForRemoval("Document")).andReturn(42);
collaboratorMock.documentRemoved("Document");

replay(collaboratorMock);

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

verify(collaboratorMock);

jMock

context.checking(new Expectations() {{
oneOf(collaboratorMock).documentAdded("Document");
oneOf(collaboratorMock).voteForRemoval("Document"); will(returnValue(42));
oneOf(collaboratorMock).documentRemoved("Document");
}});

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

context.assertIsSatisfied();

Mockito


when(collaboratorMock.voteForRemoval("Document")).thenReturn(42);

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

verify(collaboratorMock).documentAdded("Document");
verify(collaboratorMock).voteForRemoval("Document");
verify(collaboratorMock).documentRemoved("Document");

SevenMock

mockControl.expect(new Collaborator() {

    public void documentAdded(String document) {
        assertEquals("Document", document);
    }

    public byte voteForRemoval(String document) {
        assertEquals("Document", document);
        return 42;
    }

    public void documentRemoved(String document) {
        assertEquals("Document", document);
    }
});

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

mockControl.verify();

JMockit

public void testDoOperationAbc() {
    Mockit.setUpMocks(CollaboratorMock.class);

    documentManager.addDocument("Document");
    assertTrue(documentManager.removeDocument("Document"));
}


@MockClass(realClass = Collaborator.class)
public static class CollaboratorMock {
    @Mock(invocations = 1)
    public void documentAdded(String document) {
        assertEquals("Document", document);
    }

    @Mock(invocations = 1)
    public byte voteForRemoval(String document) {
        assertEquals("Document", document);
        return 42;
    }

    @Mock(invocations = 1)
    public void documentRemoved(String document) {
        assertEquals("Document", document);
    }
}

rMock

collaboratorMock.documentAdded("Document");
collaboratorMock.voteForRemoval("Document");
modify().returnValue(42);
collaboratorMock.documentRemoved("Document");

startVerification();

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

Unitils

collaboratorMock.returns(42).voteForRemoval("Document");

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

collaboratorMock.assertInvoked().documentAdded("Document");
collaboratorMock.assertInvoked().documentRemoved("Document");

Conclusion

In my opinion EasyMock and Mockito are the most easy-to-use and feature-rich frameworks. As for me I would use one of these frameworks. Syntax is quite simple and easy to read.

JMockit is also rather promising framework made with instruments exposed by Java 5 SE JVM and provides mocking of static fields, final classes, etc. It can be quite useful for legacy code testing.

jMock is well-know framework, but in my opinion it has cumbersome syntax and doesn’t provide any additional features to compete with EasyMock or Mockit.

Unitils is more than just Mock framework. It provides different tools to organize your testing environment including mocking, JTA/Hibernate/DataSource support with DBUnit integration, assertion utils, annotation-driven mocks creation and injection, Spring integration and other useful tools. As I said it has its own mock framework which is quite simple and provides the most useful features. On my opinion only using of other Unitils features proves usage of its mock framework. By the way, it also supports EasyMock as mocking module.

rMock and SevenMock – «Neither fish nor fowl».

22 comments:

  1. Jmock n ow has lots of features for thread safety and testing multithreaded code. It also has always had features (e.g. actions, state-machines, tracing) that are not in your list, so I don't see how you can say it has no more features than easyMock.

    ReplyDelete
  2. Thanks for the info! I'll check it!

    ReplyDelete
  3. You're right, thread-safety is implemented in recent version (2.7): http://jira.codehaus.org/browse/JMOCK-213, but this version is not released yet.

    ReplyDelete
  4. Interesting.

    Have you checked https://mocquer.dev.java.net/ ?

    ReplyDelete
  5. Hi,

    In EasyMock 2.5, you can do the Real Object spy using andDelegateTo.

    Also nother framework called Powermock is based on EasyMock and tries to workaround final and private limitations.

    ReplyDelete
  6. 2 Anonymous:
    >> Have you checked https://mocquer.dev.java.net/ ?
    Unfortunately I've overlooked it... Looks very similar to EasyMock.

    ReplyDelete
  7. 2 Henri,

    Thanks for the addition!
    EasyMock 2.5 has been released 2009-05-24.

    In the near future I'm going to update the page with the fresh data.

    ReplyDelete
  8. I would like to know what you meant by saying that the "Iterator-style stubbing" can be implemented in JMockit Framework. This feature can be implemented theoretically by any framework, so what is special in in JMockit ? I am asking since I need it, so if you could tell how to implement it there it would be very helpfull.
    Also, I must add, JMockit is VERY powerfull

    ReplyDelete
  9. Hi!

    I meant that you can implement it in "manual" way. The example which demonstrates the idea:

    @MockClass(realClass = MyClassToTest.class)
    public static class MyClassToTestMock {
    public static List<String> values = Arrays.asList("firstValue", "secondValue", "thirdValue");

    public static int invocationIndex;

    @Mock(invocations = 3)
    public String getSomething() {
    String result = values.get(invocationIndex);
    invocationIndex++;

    return result;
    }
    }

    I hope the idea is clear. I haven't tried it by myself, but I think it will work for you. Of course, you can implement some util class to make the code much pretty.

    ReplyDelete
  10. There is also quite interesting tool http://code.google.com/p/powermock/ which could be based on top of EasyMock or Mockito.

    Anyway - good article. I've added it to dzone.com

    ReplyDelete
  11. Dear bmajsak, thanks for the valuable addition! I'll check it out!

    ReplyDelete
  12. one word, PowerMock!

    ReplyDelete
  13. Great comparison!

    I am the author of the JMockit tool.
    I notice you only considered the JMockit Annotations API (for state-based testing). There is also the behavior-based alternative, "JMockit Expectations" (which was rather new back in January 2008).

    JMockit has seen huge progress since then.
    It now has explicit support for all features in the comparison matrix, except for "thread safety", which is something I am yet to focus on (it will be done for release 1.0, though).

    Except for SevenMock and rMock, the JMockit full distribution contains sample test suites for comparison with all the other tools, and additionally with the PowerMock tool.

    ReplyDelete
  14. Dear Rogério, thank you very much for the great reply!

    I see your web-site has been updated.
    And I feel the time to update my comparison matrix has come :)

    ReplyDelete
  15. Another thing about JMockit: somewhat like Unitils, it also provides more than mocking APIs.

    Another tool in the JMockit toolkit is "JMockit Coverage". It aims to provide the functionality of EMMA and Cobertura (with several improvements), in an easier to use and less intrusive way. On top of that, it also provides "incremental test running", like in the Infinitest IDE plugins. (This tool isn't complete nor mature at this point, but I expect to evolve it significantly in coming months.)

    Yet another tool is "JMockit Hibernate Emulation", which provides a fake implementation of the Hibernate 3 API, including HQL. It's not fully functional yet, mostly due to lack of demand. Plans to extend it to cover JPA exist, but don't ask me for a schedule right now...

    ReplyDelete
  16. Vasily, thank you for this article!

    ReplyDelete
  17. Vasily, useful comparison. Thanks. But as Rogerio mentioned, it will be indeed wise to update the ratings for JMockit as it does not reflect the full potential of the framework

    ReplyDelete
  18. I created my own comparison matrix: http://code.google.com/p/jmockit/wiki/MockingToolkitComparisonMatrix.

    It includes EasyMock/EasyMock CE, jMock, Mockito, Unitils Mock, PowerMock, and JMockit, and is up-to-date (to the best of my knowledge) as of January 3, 2010.

    ReplyDelete
  19. Rogerio,
    Unfortunately I still don't have enough time to update my matrix, so thank you so much for posting here.

    Hope it helps everyone interested in the topic.

    ReplyDelete
  20. Thanks for the informative post on Mockito. If you are willing to discuss more on unit testing with mockit and unit testing for other languages can check out Mockito Mock forum which is exclusively for unit testing for different languages. It is a upcoming forum and needs support to make it strong so we can have forum where people can share their knowledge on
    Testing.

    ReplyDelete
  21. Great article, thanks. I love the one-line examples of mocking "features".

    ReplyDelete
  22. I was just reading it as well and found this to be very helpful. Thanks!

    ReplyDelete