I want to create WCF services using aspnet membership to authorize the user. However, if I just need the certificate for encryption purposes only, does it matter if it is a self signed certificate or do you need to buy a certificate from a vendor?
You could use self-signed certificates, but the problem there is that you have to install the certificate on each and every machine that will use the certificate for encryption.
If you have a large number of machines that the clients will run on, this can easily become prohibitive from a maintenance and configuration point of view and it would easily justify the purchase of a certificate from an authority.
A self-signed certificate regardless of whether you authenticate the user carries the same risks. Best practice is to use a real cert in a production environment. With some very cheap prices these days, it's not much of a financial burden to take on any more either (unless you're a stickler for Verisign).
It's been discussed and noted that self-signed certs in a production WCF environment also come with heavy performance issues: