C#
I have a XML file that contains serealized datatable. Each row represent and event. The amount of events can reach up to milions, and loading all this data can take a very long time, and can cause the program to crash (No memory).
Is there a way to load only the last X rows of an XML file to the datatable, and then save them without overwriting the existing rows (in the XML) that does not exist in the datatable?
It sounds like you should be using a database. If you're trying to manipulate a bunch of data via an XML file as the main storage method, you're going to run into a huge number of crippling problems. I suggest you change your approach to a more proven method.
XML is not well suited to such a task; it isn't very dense (but density is desirable with high volumes), expensive to process, and non-appendable.
As has already been suggested a database may be appropriate, but there are also file formats that could help here (storing the simple directly; no db abstraction). The biggest stumbling block, however, is the "last n" requirement. Unless you store that offset separately this is very tricky (assuming each record isn't fixed size).
Let me know if you want me to suggest something more specific.