That would really depend on the usage, but IMO it's the best choice for a cross-platform drawing API.
In practice, people do use Cairo in reasonably performance-sensitive operations without problems. MonoDevelop uses Cairo for its text editor and Banshee uses Cairo for a bunch of custom widgets including its list view. I believe the GTK toolkit and Moonlight browser plugin use Cairo for all their drawing.
In theory Cairo can be hardware-accelerated, backed by OpenGL, though I don't know the current status of this backend. An nice advantage of Cairo is that it has a decent PDF backend, which is also useful for printing.
No matter what you use to draw, you're going to have to do various kinds of caching and other optimizations. For example, in MonoDevelop's text editor we cache the visible Pango text layouts, since laying out text is much more expensive than actually rendering it. And scrolling works much more smoothly if you blit some of the existing surface and only redraw the newly exposed region. And some operations (such as gradients) are much slower than others.