I've implemented some sites with Facebook connect. The short answer is yes; it's flaky. Load times can be very long, API calls often fail, images don't load, and there's the awful reality of having to use 3rd party cookies. The documentation is decent, but often dated.
You'll just have to weigh the gains you get (huge community, really cool cross-pollination, template posting, friend-linking) versus the headaches of being a FB developer. The platform is super young though and it'll be worth it to implement in the long run. I wouldn't recommend replacing a traditional user/pass system completely right now, but it's great for linking accounts and posting "wall posts" to user feeds (which is pretty easy to implement).
One of my biggest gripes with FB development is that you have to create new applications for every environment. Have a local server? Application. Have a dev server? App. The live server? Another app. Each is tied to a base domain, and there's no way to add multiple domains (though sub-domains work). You have to configure each app for each environment. Also, the fact that you can't "edit" wall templates is tough as well; you have to delete and start over as far as I know (even if you just want to correct spelling or something).
It's weird. Some aspects of FB development are an absolute JOY to work with. And some are the worst development experiences I've had.
/2¢