When compiling code that operates on List<> objects, the compiler must use interface calls, which care more expensive than regular virtual calls on concrete types.
Of course, in a piece of code where the compiler sees the object being instantiated and can prove that something that is declared to be a List<> is always an ArrayList<>, the compiler should be able to just emit a regular call instead of an interface call, but methods which are not inlined and operate on already instantiated objects would not benefit from this optimization.
The ubiquitously used optimization that speeds up virtual calls, namely inline caches (frequently Polymorphic Inline Caching or PIC, not to be confused with Position Independent Code), benefits from the observation that instances of only a single subclass are ever accessed through a variable of a certain declared type. In this scenario, after the code has been running instrumented for a while the JIT can optimistically guess that a List<> object will only ever be an ArrayList<>, generate a trap in case the guess was wrong, and fall through with the ArrayList<> call.
Modern processors execute the check very quickly (because they are superscalar and have good branch prediction) so you don't notice the cost of all those virtual call and single implementation interface calls. But it does make the VM work hard, instrumenting, generating and patching all that code.
For server software running in steady state on HotSpot it's irrelevant but for fast startup on a mobile device it might make a difference - I don't know how good is Google's VM.
A nice article about it by Dr. Cliff Click, Jr. (custom big iron hardware, hotspot-deived VM):
http://www.azulsystems.com/blog/cliff-click/2010-04-08-inline-caches-and-call-site-optimization
And of course "inline caching" on wikipedia.