How to read remote XML file the most efficiently way ?

Question

i am developping a little app that retrieves an XML file, located on a remote server (http://xxxx.com/myfile.xml) This file is relatively big, and it contains a big list of geolocations with other informations that i need to use for my app. So i read this file remotely once and insert it into a little SqlCE file (database.sdf)

So If I need to be accessing geolocation #1, i ll just make a SELECT statement into this DATABASE instead of loading the whole XML file everytime.

But i would like to know if its possible to do this without using .sdf files ?

What is the most efficient way (fastest)?

Saving the big XML file once locally and load if everytime i start my app to load it in a dataset ? this is would make the app a bit long to load everytime

Saving the big XML file once locally and reading the nodes one by one to look for geolocation #1 ?

Or is it possible to retrieve geolocation #1 from the remote xml directly(http://xxxx.com/myfile.xml) without reading the whole file ?

Do you have any other ideas?

thanks

Answer 1

Load the big XML file, convert it into an appropriate different data structure, save it to a file in an efficient format. (XML really isn't terribly efficient.)

I believe Marc Gravell's Protocol Buffers implementation works on the Compact Framework...

(None of the protobuf implementations are deemed production-ready yet, but a couple are close. We need testers!)

Answer 2

Re protobuf-net, there isn't a separate download for the CF version at the moment, but there is a csproj in the source for both CF 2.0 and CF 3.5.

To clarify on your question; actually protobuf-net doesn't even use a .proto file (at the moment); a .proto file just describes what the data is - protobuf-net simply looks at your classes and infers the schema from that (similar to how XmlSerializer / DataContractSerializer etc work). So there is not .proto - just the classes that look like your data.

However, before you embark on creating classes that look like your data, I wonder if you couldn't simply use GZIP or [PK]ZIP to compress the data, and transfer it "as is". Xml generally compresses very well. Of course, finding a GZIP (etc) implementation for CF then becomes the issue.

Of course, if you want to use protobuf-net here, I'll happily advise etc if you get issues...

The other option is for your CF app to call into a web-service that has the data locally...

Answer 3

Why would you pull the entire file down to the CE device for this? It's a bandwidth waste and certainly doing the lookup on an embedded processor is going to be way slower than on the server regardless of storage format. You should have a service (Web, WCF or whatever) that allows you to ask it for the single geolocation you want.