Looking at the (mature) codebase at my new job, there is an interface, and only one class implements it (as far as I can tell). Can/should I get rid of the interface?
No way! Its has zero harmful effects and one day somebody can swap an implementation without having to refactor tons of code.
Yes, there is a point to interfaces, as there could conceivably be more than one implementation in the future. You can of course go overboard here and create interfaces for everything, so there is a balance to be found.
Today, no. Six months from now after the project has grown tenfold in complexity? Who knows. To your point, if this is legacy code, there is little value to an interface that is implemented once, but there is also no point to going through the refactoring involved in removing it. "If it works, don't fix it".
Apart from the reasons given in the other answers, only implementing an interface allows intercepting methods by building proxies without using instrumentation. This is used for logging, encapsulating operations in a transaction, etc. It is used by several frameworks.
Even if people don't quite agree on the number of bug per 1000 lines of code, those two seem correlated (http://stackoverflow.com/questions/862277/what-is-the-industry-standard-for-bugs-per-1000-lines-of-code). Less lines == less bugs.
Also, changing the name of the implementation for the name of the interface should be all there is to do here.
If the refactoring effort is minimal and it reduces the code, I'll be rather for the supression of the interface.
Since you mention a mature base, I'd like to mention another real world example: Hibernate's Session.
Session is an interface implemented only by one class: SessionImpl. However, if you've used hibernate in the past, or read its source code, you probably have noticed how Session is used everywhere, instead of SessionImpl.
Is this wrong design? Definetly no. Its called 'The substitution principle', or 'programming towards interfaces'. It means that by using the interface instead of the implementation, you can extend your code without any effort, just instantiating your new classes accordingly, but always using the interface. Will it still be wrong if no one creates a new class implementing Session? No, still not wrong.
My two cents.
In addition to the good answers already provided - if at some point in the future that one class needs to be mocked for testing purposes, it's a lot easier to do so when there's already an interface available!
I would say it depends, but in most cases an interface is a good idea on certain classes. Mocking is made easier using interfaces and when you use an IoC container, then interfaces start making a lot of sense, especially when you start implementing services shared across the container. You will then be able to decouple the implementation of the service from the class needing the service.