This is the expected default behavior. WsHttpBinding by default will use Windows credentials for authenticating your callers. This works in your domain - but fails outside it.
What you need to do is either have a second endpoint that also uses the wsHttpBinding, but different security settings, or you need to switch the entire authentication mechanism to something that works inside and outside your domain - either username/password against e.g. ASP.NET membership system on your server-side, or you'll need to install a certificate on the caller's machines.
Update: the ultimate reference would be the WCF Security Guidance on Codeplex which has plenty of How-To's and Step-by-Step instructions on how to do specific security-related operations in WCF.