You will need to enable ASP.NET compatibility mode on the WCF service in order to enable forms authentication.
The Java client application could send username and password over a secure connection and your WCF service authenticates the user via FormsAuthentication.Authenticate(username, password)
or FormsAuthentication.SetAuthCookie
.
You will then need to use a cookie store on the Java client side in order to pass the authentication cookie on every consecutive request (and update it when it gets refreshed), but this should be a built-in feature of your HTTP-client.
The .aspx page must run on a server with the same machine key as the WCF service.
Conclusion: Yes, it is possible, but for me it is not clear to which ".NET application" you refer to?
Edit: I think its clear now, you will need to be able to set the browser cookies. If you cant do this directly from your java application, a workaround would be to let the WCF service communicate that the user is authenticated and then set the cookie on the .aspx site request.