I started the same way (writing mocks by hand) and by now I have almost completely switched over to EasyMock.
I find that using EasyMock is usually both faster and more flexible.
Typically the very first time I need a mock, I can have it in a couple of lines of code with EasyMock, whereas by hand I need to implement the needed interface (fair enough, this can be generated by an IDE like IntelliJ) and then add the necessary code to produce the needed response and/or to allow sensing the effects of calls to it.
Fine, one might say, that's a one-time cost only. The next time I can just reuse the hand written mock happily... I found that's often not the case. In another test I might need a mock of the same class with different behaviour. E.g. different methods are called, and/or different result is expected. A specific case is when the mock is expected to throw an exception in one test case, but not in another. Fine, I can add some parameters to it which control the behaviour dynamically. Then for the next test, some more parameters to control more behaviour... so I end up with an ever more complicated mock implementation, which is a dependency for ever more unit tests - posing also a risk of inadvertently breaking older tests.
As opposed to this, with EasyMock I can configure my mocks independently for each test. Thus the behaviour is explicitly controlled and visible within the unit test code itself, and there is no risk of side effects.
Not to mention that with EasyMock you can verify that the required methods are called in the required order, if you need to (and I do, every now and then). Implementing this by hand (especially in a generic fashion) would be quite pain in the neck, for no added benefit.