Suggestions for passing large table between Python and C#

Question

I have a C# application that needs to be run several thousand times. Currently it precomputes a large table of constant values at the start of the run for reference. As these values will be the same from run to run I would like to compute them independently in a simple python script and then just have the C# app import the file at the start of each run.

The table consists of a sorted 2D array (500-3000+ rows/columns) of simple (int x, double y) tuples. I am looking for recommendations concerning the best/simplest way to store and then import this data. For example, I could store the data in a text file like this "(x1,y1)|(x2,y2)|(x3,y3)|...|(xn,yn)" This seems like a very ugly solution to a problem that seems to lend itself to a specific data structure or library I am currently unaware of. Any suggestions would be welcome.

Answer 1

I would go for a simplified csv file. Given that all your values are numbers, you can read them in C# using

File.ReadAllText(filename).Split(',')

You can find more C# options for csv here

On Python you can use the csv module to read and write them. Better explanation here, but the short of it is

import csv
writer = csv.writer(filename)
writer.writerows(data)

Using CSV also gives you flexibility for future improvements, as well as exporting and importing from other programs like Excel for further processing.

Answer 2

You may consider running IronPython - then you can pass values back and forth across C#/Python

Answer 3

Why not just have your C# program check for the existence of a file called something like "constants.bin". If the file does not exist, then have it generate the array and serialize it out to "constants.bin". If the file does exist then just use serialization to read it back in.

int[,] constants;

if(!File.Exists("constants.bin")) {
    GenerateConstants();

    Stream stream = new FileStream("constants.bin", FileMode.Create, FileAccess.Write, FileShare.None);
    new BinaryFormatter.Serialize(stream, constants);
    stream.Close();
}
else
{
    Stream stream = new FileStream("constants.bin", FileMode.Open, FileAccess.Read, FileShare.Read);
    constants = (int[,])(new BinaryFormatter.Deserialize(stream));
    stream.Close();
}

I haven't tested this, so you may need to tweak it a little bit.

The first time you run the C# app the "constants.bin" won't exist so it will generate the 2D array and then serialize it out to the file. Each subsequent run of the program will find the "constants.bin" file and deserialize it into the local 2D array.

Answer 4

CSV is fine suggestion, but may be clumsy with values being int and double. Generally tab or semicomma are best separators.

Answer 5

Python standard library includes the sqlite3 module - a lightweight disk-based database. For C# there are a few libraries providing sqlite support. For example, System.Data.SQLite - a complete ADO.NET 2.0/3.5 provider.

For your application, use datatypes REAL (stored as an 8-byte IEEE floating point number) and INTEGER (stored in 1, 2, 3, 4, 6, or 8 bytes depending on the magnitude of the value).

Answer 6

Have a look at NetCDF and/or HDF5 file formats. HDF5 in particular seems to have a .NET implementation, and PyTables is handy on the Python side of things.