Google Checkout and PayPal both differ from "traditional" payment processing solutions in that they don't require nearly the same amount of setup work to get going. They combine the traditionally separate roles of credit card processing and merchant accounts. The upsides include a simpler implementation and cheaper rates. The downside is that they tend to control much more of the experience - Google Checkout and PayPal both have their own login systems, for example, which your users would likely see as part of the payment workflow. If you were unhappy with Google Checkout or PayPal (both have horror stories about withholding payments to merchants, for example) you'd generally have a harder time than if the roles were split between the payment processor and the merchant account host.
If you want to go with a fully customized solution, I'd recommend ActiveMerchant. It primarily provides a unified, easy-to-use API for various credit card processing companies, including PayPal.
In the end, the most important factor is what your customers are comfortable with. A lot of non-US based sales would skew heavily towards PayPal or use cell phone payments; an early adopter technology crowd would be more likely to have the Google accounts necessary to use Google checkout, whereas most other predominantly US-based online shops would be best served accepting credit cards.