The semantics of LinkedHashMap are still those of a Map, rather than that of a LinkedList. It retains insertion order, yes, but that's an implementation detail, rather than an aspect of its interface.
The quickest way to get the "first" entry is still entrySet().iterator().next(). Getting the "last" entry will entail iterating over the whole entry set.
edit: However, if you're willing to go beyond the JavaSE API, Apache Commons Collections has its own LinkedMap implementation, which has methods like firstKey and lastKey, which do what you're looking for. The interface is considerably richer. Commons Collection does not use generics, but you can get a generic-ised version here.
edit: Looks like some buffoon is farting about with the web server on commons.apache.org, it keeps failing requests... I wonder if they're using IIS....