Methodology for saving values over time

Question

I have a task, which I know how to code (in C#), but I know a simple implementation will not meet ALL my needs. So, I am looking for tricks which might meet ALL my needs.

1. I am writing a simulation involving N number of entities interacting over time.

2. N will start at around 30 and move in to many thousands.

   a. The number of entities will change during the course of the simulation.
   

   b. I expect this will require each entity to have its own trace file.

3. Each Entity has a minimum of 20 parameters, up to millions; I want to track over time.

   a. This will most likely required that we can’t keep all values in memory at all times. Some subset should be fine.

   b. The number of parameters per entity will initially be fixed, but I can think of some test which would have the number of parameters slowing changing over time.

4. Simulation will last for millions of time steps and I need to keep every value for every parameter.

5. What I will be using these traces for:

   a. Plotting a subset (configurable) of the parameters for a fixed amount of time from the current time step to the past.

   i. Normally on the order of 300 time steps.
   
   
   ii. These plots are in real time while the simulation is running.
   

   b. I will be using these traces to re-play the simulation, so I need to quickly access all the parameters at a give time step so I can quickly move to different times in the simulation.

   i. This requires the values be stored in a file(s) which can be inspected/loaded after restarting the software.
   
   
   ii. Using a database is NOT an option.
   

   c. I will be using the parameters for follow up analysis which I can’t define up front so a more flexible system is desirable.

My initial thought:

1. One class per entity which holds all the parameters.

2. Backed by a memory mapped file.

3. Only a fixed, but moving, amount of the file is mapped to main memory

4. A second memory mapped file which holds time indexes to main file for quicker access during re-playing of simulation. This may be very important because each entity file will represent a different time slice of the full simulation.

Answer 1

Just for the memory part...

1.You can save the data as xElemet (sorry for not knowing much about linq) but it holds an XML logic.

2.hold a record counter.

after n records save the xelement to an xmlFile (data1.xml,...dataN.xml)

It can be a perfect log to any parameter you have with any logic you like:

<run>
  <step id="1">
     <param1 />
     <param2 />
     <param3 />
  </step>
  .
  .
  .
  <step id="N">
     <param1 />
     <param2 />
     <param3 />
  </step>
</run>

This way your memory is free and the data is relatively free. You don't have to think too much about DB issues and it's pretty amazing what LINQ can do for you... just open the currect XML log file...

Answer 2

I would start with SQLite. SQLite is like a binary format library that you can query conveniently and quickly. It is not really like a database, in that you can really run it on any machine, with no installation whatsoever.

I strongly recommend against XML, given the requirement of millions of steps, potentially with millions parameters.

EDIT: Given the sheer amount of data involved, SQLite may well end up being too slow for you. Don't get me wrong, SQLite is very fast, but it won't beat seeks & reads, and it looks like your use case is such that basic binary IO is rather appropriate.

If you go with the binary IO method you should expect some moderately involved coding, and the absence of such niceties as your file staying in a consistent state if the application dies halfway through (unless you code this specifically that is).

Answer 3

KISS -- just write a logfile for each entity and at each time slice write out every parameter in a specified order (so you don't double the size of the logfile by adding parameter names). You can have a header in each logfile if you want to specify the parameter names of each column and the identify of the entity.

If there are many parameter values that will remain fixed or slowly changing during the course of the simulation, you can write these out to another file that encodes only changes to parameter values rather than every value at each time slice.

You should probably synchronize the logging so that each log entry is written out with the same time value. Rather than coordinate through a central file, just make the first value in each line of the file the time value.

Forget about database - too slow and too much overhead for simulation replay purposes. For replaying of a simulation, you will simply need sequential access to each time slice, which is most efficiently and fastest implemented by simply reading in the lines of the files one by one.

For the same reason - speed and space efficiency - forget XML.