C#: capture data, display to user, save to disk, upload to database: BEST approach

Question

I have a program that captures live data from a piece of hardware on a set interval. The data is returned as XML. There are several things I would like to do with this data (in order): -display it to user -save it to disk -eventually, upload it to database

My current approach is to take the XML, parse it into a hashtable so I can display the correct values to the user.

Next, I want to save the XML to a file on disk. For each data capture session I am planning on creating a unique XML file and I will dump all the data into it.

Finally, I would like to reparse the XML and upload it to a MySQL database. The data cannot be immediately uploaded to the database.

This seems really inefficient method of solving this problem and I would love some advice.

Is it a waste of hd space to save the data as XML?

Is it THAT inefficient to have to reparse the XML in order to write it to a database?

Thank you!

---

To clarify: a typical XML response will be ~1kb and are captured at a rate of about 1 response every 15-60 seconds.

I think I do want to store the XML as XML on the disk because the data is very valuable and a pain to reproduce (if it is even possible). Thank you!

Answer 1

That mostly depends on the amount and rate of data you're moving. If you have to upload to the database seldom and it only takes a few seconds, the flexibility of XML is surely good to have. If you're never using the locally stored data except to upload it to the database and parsing takes a few minutes you might want to rethink the strategy.

Answer 2

When you receive a new XML document from the source, directly save to disk and parse it to display to the user.

With a background process, or user initiated, read the xml files from disk and send them to the server based on created date (so you can retrieve only the latest ones) for insert into MySql.

Answer 3

perhaps there should be a seperate thread for the data fetching and the processing.

Answer 4

the application can't be mulithreaded. Thanks for all the input so far!

Answer 5

if there is a necessity to view the records we can parse and save to db