We've done this recently by writing new WCF services that expose the service in a simple way that VB6 can handle. Like you say VB6 can't handle complex types as they are not generated by a proxy like .net would do for you. So our solution was to wrap (facade?) the existing service in something that VB6 could call.
Our service returned a string which contained an XML document that could be parsed in VB6 into the required object i.e. return an XML document containing 10 clients and then parse this into a collection of Client objects.
Once the WCF service was created you need to grab the WSDL and save it locally or point the code to the WSDL location, we had some trouble when trying to get the WSDL via HTTPS which is why we saved it locally.
To consume them we used the SOAP toolkit from MS.
Here's some example code for creating the request and parsing the result :-
Set m_ServiceClient = New MSSOAPLib30.SoapClient30
With m_ServiceClient
.MSSoapInit "c:\service.wsdl"
.ConnectorProperty("Timeout") = "30000"
End With
lLoading = ParseResultToLoading(m_ServiceClient.GetLoadingByCountryIdProductId(llProductId, _
Countries(), _
Duration, _
lsTravellerType, _
Traveller.MedicalScore, _
Traveller.InitialAmount, _
Traveller.AdditionalWeekAmount, _
IssueDate))
Private Function ParseResultToLoading(XMLString As String) As Loading
Dim x As Loading
Dim objXML As New MSXML2.DOMDocument
Dim objElem As MSXML2.IXMLDOMElement
If Not objXML.LoadXML(XMLString) Then
Err.Raise objXML.parseError.ErrorCode, , objXML.parseError.reason
End If
Set objElem = objXML.selectSingleNode("//LoadingInfo")
Dim objSub As MSXML2.IXMLDOMElement
' iterate its sub-nodes
For Each objSub In objElem.childNodes
Select Case UCase$(objSub.tagName)
Case "FIXED"
x.Fixed = objSub.Text
Case "FIXEDNET"
x.FixedNet = objSub.Text
'any other values that are required...
End Select
Next
ParseResultToLoading = x
End Function
You can see in the code example that we passed arrays, strings, longs and dates into the service call without any problems. Can't use any .net return types (apart from the obvious ones!) as VB6 won't understand what they are. You can use a similar design to push information back to the service.
Regarding the exceptions to throw in your library we just used standard .net code to throw any exceptions we needed to or allowed existing exceptions to bubble up and you could get exception information via the MSSOAPLib30.SoapClient30 object it contains various fault properties.
All worked really well in the end, had no problems with it running and the best bit is that VB6 now uses exactly the same functionality as other .net clients.
Hope that gives you an idea of how we did it and that it may help you with your decisions. let me know if anything doesn't make sense or you want clarification on any of it.