This is one of my biggest bugbears of RIA Services :). The good thing is that they are in the process of enabling complex types to be returned from invoke methods (according to the user voice site for RIA Services). However, that won't help you now :).
The best way to implement your requirement to return complex objects is indeed to set up a standard WCF Service to return complex types when required. You do so in the normal way that you would expose a WCF in any web project. You then consume it in your Silverlight application in much the same way you would a WCF Service in any other type of project (by adding it as a service reference). The main difference when consuming it (compared to other types of projects) is that Silverlight only supports asynchronous calls. One difference to be aware of from RIA Services is that whereas RIA Services automatically determines the URI of the domain services (based upon where the XAP file originated from), you will need to handle that aspect yourself (very important when you try to deploy your application :). Do this either by setting the URI in the app.config file, or use the method I use in the code accompanying this article here: http://www.silverlightshow.net/items/Building-a-Framework-for-Silverlight-Line-Of-Business-Applications.aspx.
In regards to your final question, yes, RIA Services uses standard ASP.NET authentication, so the cookie that it uses to maintain your authentication status is also used by any WCF Services in the same project. I talk a bit about security with WCF Services in this article if you want more information: http://www.silverlightshow.net/items/Building-a-Silverlight-Line-Of-Business-Application-Part-3.aspx.
Hope this helps...
Chris