I am new to PyTables, and am looking at using it to process data generated from an agent-based modeling simulation and stored in HDF5. I'm working with a 39 MB test file, and am experiencing some strangeness. Here's the layout of the table: /example/agt_coords (Table(2000000,)) '' description := { "agent": Int32Col(shape=(), d...
Hello all, Question in nutshell What is the best way to get Python and Java to play nice with each other? More detailed explanation I have a somewhat complicated situation. I'll try my best to explain both in pictures and words. Here's the current system architecture: We have an agent-based modeling simulation written in Java. ...
Hi guys, I have HDF5 files with multiple groups, where each group contains a data set with >= 25 million rows. At each time step of simulation, each agent outputs the other agents he/she sensed at that time step. There are ~2000 agents in the scenario and thousands of time steps; the O(n^2) nature of the output explains the huge numbe...
viTables only seems to work with python 2.5. I have downloaded HDFView, but when I try to open a table I created following this tutorial, I get the following error message: Failed to open file C:\tutorial.h5 java.io.IOException: Unsupported fileformat - C:\tutorial.h5 Any ideas for getting HDFView to work? Is there something I need t...
Does anyone out there have any great ideas to achieve a massively scalable hierarchical datastore? It needs rapid add and ability to have many users of site requesting reports on the number of nodes below a certain node in hierarchy. This is the scenario.... I will have a very large number of nodes getting added per hour. Lets say I w...
The following code provides my needed functionality; but, takes a bit more than 10 seconds for a table that includes 200 variables with 64000 rows. Is there a more efficient way to create a variable namespace that matches the column names? strExec = "a = table[:]" for colobj in table.description._f_walk(type="Col"): colName = co...
Is there a way to create a PyTable with a specific column order? By default, the columns are alphabetically ordered when using both dictionary or class for schema definition for the call to createTable(). My need is to establish a specific order and then use numpy.genfromtxt() to read and store my data from text. Unfortunately, my text ...
Hello All, I recently came across Pytables and find it to be very cool. It is clear that they are superior to a csv format for very large data sets. I am running some simulations using python. The output is not so large, say 200 columns and 2000 rows. If someone has experience with both, can you suggest which format would be more conv...