I ran into this issue a few days ago and the solution is a bit detailed but here are the important bits. In AuthorizeAttribute the OnAuthorization method returns a HttpUnauthorizedResult when authorization fails which makes returning a custom result a bit difficult.
What I ended up doing was to create a CustomAuthorizeAttribute class and override the OnAuthorization method to throw an exception instead. I can then catch that exception with a custom error handler and display a customized error page instead of returning a 401 (Unauthorized).
public class CustomAuthorizeAttribute : AuthorizeAttribute
{
    public virtual void OnAuthorization(AuthorizationContext filterContext) {
        if (filterContext == null) {
            throw new ArgumentNullException("filterContext");
        }
        if (AuthorizeCore(filterContext.HttpContext)) {
            HttpCachePolicyBase cachePolicy = filterContext.HttpContext.Response.Cache;
            cachePolicy.SetProxyMaxAge(new TimeSpan(0));
            cachePolicy.AddValidationCallback(CacheValidateHandler, null /* data */);
        }
        else {
            // auth failed, redirect to login page
            // filterContext.Result = new HttpUnauthorizedResult();
            throw new HttpException ((int)HttpStatusCode.Unauthorized, "Unauthorized");                
        }
    }
}
then in your web.config you can set custom handlers for specific errors:
    <customErrors mode="On" defaultRedirect="~/Error">
        <error statusCode="401" redirect="~/Error/Unauthorized" />
        <error statusCode="404" redirect="~/Error/NotFound" />
    </customErrors>
and then implement your own ErrorController to serve up custom pages.
On IIS7 you need to look into setting Response.TrySkipIisCustomErrors = true; to enable your custom errors.