Silverlight Storing Reference Data

Question

I have a text file with a list of 300,000 words and the frequency with wich they occur. Each line is in the format Word:FequencyOfOccurence. I want this information to be accessible from within the C# code. I can't hard code the list since it is too long, and I'm not sure how to go about accessing it from a file on the server. Ideally I'd ideally like the information to be downloaded only if it's used (To save on bandwidth) but this is not a high priority as the file is not too big and internet speeds are always increasing. It doesn't need to be useable for binding. The information does not need to be editable once the project has been built.

Answer 1

You could create a WCF service on the server side that will send the data to the Silverlight application. Once you retrieve the information you could cache it in-memory inside the client. Here's an example of calling a WCF service method from Silverlight.

Another possibility is to embed the text file into the Silverlight assembly that is deployed to the client:

using (var stream = Assembly.GetExecutingAssembly()
                            .GetManifestResourceStream("namespace.data.txt"))
using (var reader = new StreamReader(stream))
{
    string data = reader.ReadToEnd();
    // Do something with the data
}

Answer 2

Based on your comments, you could download the word list file if you are required to have a very thin server layer. The XAP file containing your Silverlight application is nothing more than a ZIP file with all the referenced files for your Silverlight client layer. Try adding the word list as content that gets compiled into the XAP and see how big the file gets. Text usually compresses really well. In general, though, you'll want to be friendly with your users in how much memory your application consumes. Loading a huge text file into memory, in addition to everything else you need in your app, may untimately make your app a resource hog.

A better practice, in general, would be to call a web service. The service could would perform whatever look up logic you need. Here's a blog post from a quick search that should get you started: (This was written for SL2, but should apply the same for SL3.)

Calling web services with Silverlight 2

Even better would be to store your list in a SQL Server. It will be much easier and quicker to query.

Answer 3

Here is another alternative. Zip the file up and stick it in the clientBin folder next to the apllication XAP. Then at the point in the app where the content is needed do something like this:-

 public void GetWordFrequencyResource(Action<string> callback)
 {
     WebClient client = new WebClient();
     client.OpenReadAsync += (s, args) =>
     {
       try
       {
         var zipRes = new StreamResourceInfo(args.Result, null)
         var txtRes = Application.GetResourceStream(zipRes, new Uri("WordFrequency.txt", UriKind.Relative));
         string result = new StreamReader(txtRes.Stream).ReadToEnd();

         callback(result);
       }
       catch
       {
         callback(null);  //Fetch failed.
       } 

     }
     client.OpenReadAsync(new Uri("WordFrequency.zip", UriKind.Relative"));
 }

Usage:-

 var wordFrequency = new Dictionary<string, int>();
 GetWordFrequencyResource(s =>
 {
    // Code here to burst string into dictionary.
 });
 // Note code here is asynchronous with the building of the dictionary don't attempt to 
 // use the dictionary here.

The above code allows you to store the file in an efficient zip format but not in the XAP itself. Hence you can download it on demand. It makes use of the fact that a XAP is a zip file so Application.GetResourceStream which is designed to pull resources from XAP files can be used on a zip file.

BTW, I'm not actually suggesting you use a dictionary, I'm just using a dictionary as simple example. In reality I would imagine the file is in sorted order. If that is the case you could use a KeyValuePair<string, int> for each entry but create a custom collection type that holds them in an array or List and then use some Binary search methods to index into it.