The big thing to keep in mind is that Forms authentication works based on cookies. When the client browser sends a request to the server, it sends the cookie with each request. If the server requires a specific cookie (ASP.NET makes this a requirement if you're enforcing authentication on a particular URL) and the cookie isn't there, this is when you get the unauthorized error.
In order to get your jQuery code, the client has to request it from a host page, which could be an ASP.NET page that is protected via Forms Authentication. This is the first step - securing access to the jQuery code itself. The next thing you need to do is secure access to the RESTful endpoint.
If you put the .svc endpoint (assuming you're hosting the RESTful WCF service in IIS) in the same directory as part of your application that is being protected by Forms Authentication via Web.config settings, then ASP.NET will reject requests to the .svc endpoint before WCF is told about the request. In this fashion, you can wrap your WCF service in Forms Authentication without having to do any complicated coding within the service.
Since jQuery, when making HTTP requests of your endpoint, is doing so from within the context of your browser, HTTP requests made from jQuery (so long as the jQuery was loaded from the same root as your forms authenticated site) will contain the authentication cookie.