I need to exchange data with different programs in text format what do you suggest ?

Question

Hello, Our program needs to import/export data from/to different programs, each from specific independant proprieties. Most of these companies would send 2 text files: the first consisting of the data content along with a second one holding descriptions. Suppose this example:

content file: TheArtofDeceptionKevin11.53

Description file: offset[0..16] -> Book name 
                  offset[17..21] -> Author
                  offset[22..27] -> Price

In order to deal with this problem, I used an Jason based configuration file for each company and in order to extract data I would parse the jason, extract offset informations and import... But recently I was thinking of using a database for this, let's say definning a table with primary key as (company, docType, fieldname) and as columns as (start, end)

What do you think of that last design ? I wish someone has already run through this and help me come up with a good solution.

Thank you,

well I actually meant by using a database, the fact of parsing those configuration files for each company and store description in the DB. This way I think, I won't be needing to parse the config file each time I wanna import data from a specific known company. So all the exchange files are parsed only once and stored into the database, and whenever I need to import data from a new content file I will just query the db and extract the offsets.

Answer 1

It depends;) (I assume you mean json and not jason.

Data interchange is one of the main use cases of XML of which json is a subset. This allows a common interchange format that can be manipulated with common tools.

However if the data is regularly structure then you can use a database like interchange format but there is no well defied format for this you cannot just copy a MySql database file. You can transfer tab delimited or CSV files but the data they contain is not as well defined as XML - e.g. non ASCII characters how are they represented? aslo this file will not contain primary key information. If you can agree with all participants on the exact format then this transfer method will use smaller files than XML and be quicker to process but then again this only matters I think if you have a very large amount of data.

Thus I would stick to the XML/json format unless the speed of transfer or processing makes a noticeable difference to the process.

Answer 2

Using a database system is mainly useful if you want to have permanent storage, which I gather is not your goal. What you may be interested in is some kind of message queuing system that can arrange for the reliable transport between your various applications. As for the data format, the decision between XML and JSON is mostly a matter of taste and a question what your applications and transport mechanisms can handle best, unless you are interested in using some kind of schema language to validate the XML, something that is currently not widely practices with JSON.