ORMs can certainly be performance killers. I've measured performance of Entity Framework (v1) and LINQ to SQL against ADO.NET (both datasets and datareaders). EF performance was simply unacceptable, especially in web apps where data contexts would be created and discarded very frequently. LINQ to SQL wasn't too bad, but wouldn't qualify for a high-performance application. The difference between the two is that EF is a two-layer model, without generated code to optimze the mapping between layers. LINQ to SQL is a single layer, and doesn't offer nearly as much customization; the trade-off is that LINQ queries map more closely to the relational model, so there is much less overhead.
The concept of an ORM is certainly valuable; it results in much cleaner code at the application level. But there's no such thing as a free lunch. I'm currently writing a custom ORM that maps a single data model onto both SQL Server and Oracle, with the ability to switch between servers with a simple app config setting. However, there's no LINQ IQueryable provider, and all queries are written or generated as dynamic SQL and run as ADO.NET queries. All database interaction is interfaced to the application as method calls for specific operations, and IEnumerables of entity classes returned as results. Performance is within 10% of straight ADO.NET coding, but that level of performance was a requirement from the start of the project.
While the core components of this ORM could be used in any project, the only way to get ORM and high performance is to avoid any on-the-fly mapping, either between layers (as EF does) or translating LINQ into SQL. (It's very painful to write that, because I dearly love LINQ, but the mapping cost is too much. I did make sure that LINQ to Objects works both within the ORM and in application usage with the results.)