I think LINQ is fairly solid, and there aren't a lot of big caveats. Nearly every "problem" I've run into is the result of deferred execution, and it's not really a problem, but rather a different way of thinking.
The biggest issue I've faced - LINQ is a game changer (or at least a rule bender) when it comes to profiling for performance. The deferred execution can make it much more difficult to profile an application at times, and can also dramatically change the runtime performance characteristics in unexpected ways. Certain LINQ operations seem almost magical with how fast they are, and others take a lot longer than I expected - but it's not always obvious from the code or profiler results.
That being said, in general, the deferred execution more than makes up for the cases where it's slowed down hand-coded routines. I much prefer the simpler, cleaner code to the code it replaced.
Also, I have found that the more I use LINQ to Objects, the more I have to rethink my design and rework my collections in general.
For example, I had never realized how often I was exposing IList instead of IEnumerable when it wasn't absolutely necessary until I started using linq to objects frequently. I now completely understand why MS design guidelines warn against using IList too often (for example, don't return IList just for the Count property, etc). When I'd have methods that took IList, passing through the IEnumerable results from a linq query requires .ToList() or a reworking of the method's API.
But it's almost always worth the rethinking - I've found many places where passing an enumerable and using LINQ resulted in a huge perf. gains. The deferred execution is wonderful if you think about it, and take full advantage of it. For example, using .Take() to restrict a collection to the first 2 elements if that's all that's needed was a bit more challenging pre-linq, and has dramatically sped up some of my nastier loops.