There may be better ways, which I'm open to using but this works for me and it's flexible.
In your Web Application's Web.config, add a variable in AppSettings and store the base URL, notice that I'm not storing the location of the SVC file, I will append that later. This is because I have multiple SVC that I usually point to. You may choose to do it differently.
 <appSettings>
    <add key="ServiceURI" value="http://localhost:64457/"/>
 </appSettings>
In my Web Application's Web Page, add a param called InitParms, this allows you to add a list of key, pair values (seperated by comma that will be read by the XAP file)
<div id="silverlightControlHost">
    <object data="data:application/x-silverlight," type="application/x-silverlight-2"
        width="100%" height="100%" ID="Xaml1" >
        <param name="InitParams" value="ServiceURI=<%= ConfigurationManager.AppSettings("ServiceURI") %>" />
In the Silverlight App.xaml.vb, load all the InitParms into a Resource or where ever you want
    Private Sub Application_Startup(ByVal o As Object, ByVal e As StartupEventArgs) Handles Me.Startup
    If e.InitParams IsNot Nothing Then
        For Each k As Generic.KeyValuePair(Of String, String) In e.InitParams
            Me.Resources.Add(k.Key, k.Value)
        Next
    End If
Then in any of my XAML files I can initialize the service with the configured URI, I have a method like this
Private Sub InitializeService()
    Dim uri As String = App.Current.Resources("ServiceURI")
    If uri Is Nothing OrElse uri = String.Empty Then
        'if there is no value added in the web.config, I can fallback to default values
        _client = New ServiceClient
    Else
        'Notice I hardcoded the location of the SVC files in the client and append there here, you may choose not to do this
        Dim uri_withservice As String = uri & "svc/secure/Service.svc"
        _client = New ServiceClient("CustomBinding_Service", New EndpointAddress(uri_withservice))
    End If
End Sub