Java Mock Frameworks Comparison

In this post I try to tell you what mock frameworks are and what features they provide, and compare existing frameworks.

What is mock?

Let’s look at the example.

As we can see on the diagram Class A depends on Class B and Class C. Class B depends on Class D and Class E.

Ok. What should we do for unit testing of Class A? First of all we should isolate Class A. We should provide for instance of Class A something looks like instances of Class B and Class C, but which give us an opportunity to check behavior of testing instance.

These replacements are called mocks. Simple mocks which can only react like replacement classes with some canned answers to invocations made during tests are called stubs.

The diagram for unit testing of Class A:

Mock frameworks

Mock frameworks are the frameworks that give us an easy way to create mocks. Usually working with mock frameworks includes the following steps:

1. Creation of a mock. Usually it looks like this (in EasyMock):

   ClassB mock = createMock(ClassB.class);

2. Definition of the stubbed methods (what the method should do when a call happens). Sometimes definition is combined with expectations. EasyMock example:

   expect(mock.go()).andThrow(new RuntimeException());

3. Definition of the expectations (how many times this method will be called, etc). EasyMock example:

   expectLastCall().times(3);

4. Execution of the test code.
5. Verification of the expectations. EasyMock example:

   verify(mock);

Different frameworks provide their own ways to define the expectations and a variety of features to check the tested object’s behavior.

Features

Let’s make a quick overview of common features provided by mock frameworks.

Value returning for stubbed method

It’s the ability of stubbed method to return value when invocation happens.

Mockito example:

when(mock.isDone()).thenReturn(true);

Exception throwing for stubbed method

It’s the ability of stubbed method to throw an exception. Mockito example:

when(mock.do()).thenThrow(new CantDoException());

Invocations count check

It’s the ability to check count of mock’s method invocations.

EasyMock example:

expect(mock.run()).times(1, 3); // expecting at least one and maximum three invocations

Method arguments check

It’s the ability to validate parameters passed to mock’s method. It could be range check, type check, etc. jMock example:

allowing(calculatorMock).sqrt(with(lessThan(0)); will(throwException(new IllegalArgumentException());

Invocations order for one mock check

It’s the ability to check the order of invocations for one mock. For example, in EasyMock we should create StrictMock with createStrictMock() method. StrictMock will check invocation order automatically.

Iterator-style stubbing

It’s the ability of a stubbed method to return different values for each invocation. Mockito example:

when(mock.getWeekday("My event")).thenReturn("Monday", "Tuesday", "Thursday");

Real objects spy

It’s the ability of a mock-object to act like proxy. It checks the invocations and redirect them to the real object. This feature can be useful for legacy code testing.

Mockito example from documentation:

List list = new LinkedList();
List spy = spy(list);

//optionally, you can stub out some methods:
when(spy.size()).thenReturn(100);

//using the spy calls real methods
spy.add("one");
spy.add("two");

Callback invocation from stubbed method

It’s the ability to invoke custom code from a stubbed method. Mockito documentation example:

when(mock.someMethod(anyString())).thenAnswer(new Answer() {
Object answer(InvocationOnMock invocation) {
        Object[] args = invocation.getArguments();
        Object mock = invocation.getMock();
        return "called with arguments: " + args;
    }
});

Mocking of final classes, methods, static and private fields

This ability could be useful for legacy code testing.

Stubbing of equals() and hashcode() methods

Most of frameworks (if not all) redefine these methods for their own needs. So it’s the usual limitation.

Thread safe

It’s the ability to run test with mock objects in multiple threads, so the mock objects are not shared between threads.

Invocation order between mocks check

It’s the ability to check the order of invocations between several mocks. For EasyMock it can be reached using control object:

IMocksControl control= createStrictControl();
IMyInterface mock1 = control.createMock(IMyInterface.class);
IMyInterface mock2 = control.createMock(IMyInterface.class);

Then framework will remember invocation between these mocks.

Java Mock Frameworks Features Comparison

First of all, please notice that all information about frameworks was taken from their documentation and I can’t guarantee that it is 100% accurate. If you have found any mistakes please feel free to contact me.

	jMock	EasyMock	Mockito	SevenMock	JMockit	rMock	Unitils
Actual version	2.6-RC1 (7 Dec 2008)	2.4 (30 Aug 2008)	1.7 (24 Jan 2009)	2.1.0 (17 Jun 2008)	0.94 (30 Jan 2008)	2.0.0 (18 Mar 2007)	2.2 (23 Dec 2008)
Website	http://jmock.org	http://easymock.org	http://code.google.com/p/mockito	http://seven-mock.sourceforge.net	https://jmockit.dev.java.net	http://rmock.sourceforge.net	http://www.unitils.org
Features	Specification-like definition of mocks’ methods behavior (looks like DSL).	«Record/Replay/Verify» model. Basic expectations are defined via calling appropriate mocks’ methods with actual data. Other types of expectations are defined with framework’s methods.	«Verify after run» model. Basic expectations are checked after calling (mocks remember their methods invocations). Other types of expectations are defined as rules before running.	Method stubbing and expectations are defined using inner classes.	Methods stubbing are made using inner classes. Annotations for those methods define basic expectations. Other expectations are defined using DSL-like style.	«Record/Replay/Verify» model. Expectations are defined using helper methods.	Mockito-like «Verify after run» model.
Value returning for stubbed method	+	+	+	+*	+	+	+
Exception throwing for stubbed method	+	+	+	+*	+	+	+
Invocations count check	+	+	+	-*	+	+	+*
Method arguments check	+	+	+	-*	+	+	+
Invocations order for one mock check	+	+	+	+	-	+	+
Iterator-style stubbing	+	+	+	-*	+*	-	+
Real objects spy	-	-	+	-	-	+	-
Callback invocation from stubbed method	+	+	+	-	-	-	+
Mocking of final classes, methods, static and private fields	-	-	-	-	+	-	-
Mocking of equals() and hashcode() methods	-	-	-	-	‌	‌	‌
Thread safe	-	+	+	‌	‌	‌	-
Invocation order between mocks check	+	+	+	+	-	+	-
Total score	6.7	7.5	8.3	3.5	4.8	5.8	5.6

* — feature can be implemented, but not supported directly by a framework

Syntax comparison

Framework review would be incomplete with features comparison alone. Tests should be easy to read, so syntax is a very important criterion. That’s why I want to demonstrate testing of some example code with these frameworks.

I like the example from EasyMock documentation, so I rewrite it for other frameworks using their documentation as a reference. Let’s assume we have a DocumentManager object which references to Collaborator obects. DocumentManager has

void addDocument(String document)

method. When this method is executed DocumentManager (like Event Source in Observer design pattern) notifies its Collaborators with

void documentAdded(String document)

method. Also DocumentManager has

Boolean removeDocument(String document)

method which inquires its Collaborators with

Integer voteForRemoval(String document)

method. If the sum of votes is larger than zero, DocumentManager notifies its Collaborators about document removal with method

void documentRemoved(String document)

then returns true.

It can look like this:

public class DocumentManager() {
List collaborators = new ArrayList();

    ...

    public void addDocument(String document) {
        ...

        for (Collaborator collaborator : collaborators) collaborator.documentAdded(document);
    }


    public boolean removeDocument(String document) {
        Integer votingResult = 0;
        for (Collaborator collaborator : collaborators) {
          votingResult += collaborator.voteForRemoval(document);
        }


        if (votingResult > 0) {
            ...

          for (Collaborator collaborator : collaborators) collaborator.documentRemoved(document);
          return true;
        }
    }
}

In following examples I avoid the setup phase and concentrate directly on testing. Check it out:

EasyMock

collaboratorMock.documentAdded("Document");
expect(collaboratorMock.voteForRemoval("Document")).andReturn(42);
collaboratorMock.documentRemoved("Document");

replay(collaboratorMock);

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

verify(collaboratorMock);

jMock

context.checking(new Expectations() {{
oneOf(collaboratorMock).documentAdded("Document");
oneOf(collaboratorMock).voteForRemoval("Document"); will(returnValue(42));
oneOf(collaboratorMock).documentRemoved("Document");
}});

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

context.assertIsSatisfied();

Mockito

when(collaboratorMock.voteForRemoval("Document")).thenReturn(42);

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

verify(collaboratorMock).documentAdded("Document");
verify(collaboratorMock).voteForRemoval("Document");
verify(collaboratorMock).documentRemoved("Document");

SevenMock

mockControl.expect(new Collaborator() {

    public void documentAdded(String document) {
        assertEquals("Document", document);
    }

    public byte voteForRemoval(String document) {
        assertEquals("Document", document);
        return 42;
    }

    public void documentRemoved(String document) {
        assertEquals("Document", document);
    }
});

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

mockControl.verify();

JMockit

public void testDoOperationAbc() {
    Mockit.setUpMocks(CollaboratorMock.class);

    documentManager.addDocument("Document");
    assertTrue(documentManager.removeDocument("Document"));
}


@MockClass(realClass = Collaborator.class)
public static class CollaboratorMock {
    @Mock(invocations = 1)
    public void documentAdded(String document) {
        assertEquals("Document", document);
    }

    @Mock(invocations = 1)
    public byte voteForRemoval(String document) {
        assertEquals("Document", document);
        return 42;
    }

    @Mock(invocations = 1)
    public void documentRemoved(String document) {
        assertEquals("Document", document);
    }
}

rMock

collaboratorMock.documentAdded("Document");
collaboratorMock.voteForRemoval("Document");
modify().returnValue(42);
collaboratorMock.documentRemoved("Document");

startVerification();

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

Unitils

collaboratorMock.returns(42).voteForRemoval("Document");

documentManager.addDocument("Document");
assertTrue(documentManager.removeDocument("Document"));

collaboratorMock.assertInvoked().documentAdded("Document");
collaboratorMock.assertInvoked().documentRemoved("Document");

Conclusion

In my opinion EasyMock and Mockito are the most easy-to-use and feature-rich frameworks. As for me I would use one of these frameworks. Syntax is quite simple and easy to read.

JMockit is also rather promising framework made with instruments exposed by Java 5 SE JVM and provides mocking of static fields, final classes, etc. It can be quite useful for legacy code testing.

jMock is well-know framework, but in my opinion it has cumbersome syntax and doesn’t provide any additional features to compete with EasyMock or Mockit.

Unitils is more than just Mock framework. It provides different tools to organize your testing environment including mocking, JTA/Hibernate/DataSource support with DBUnit integration, assertion utils, annotation-driven mocks creation and injection, Spring integration and other useful tools. As I said it has its own mock framework which is quite simple and provides the most useful features. On my opinion only using of other Unitils features proves usage of its mock framework. By the way, it also supports EasyMock as mocking module.

rMock and SevenMock – «Neither fish nor fowl».