tags:

views:

39

answers:

1

I've used JavaScript to get the data through the internet (I'm interfacing with a brokerage firm's API functions), but unlike most of the rest of their API's, this one returns the data in a 'binary' like format. Here is the layout of the file I get back:

Field --------     Type ------------     Length(8 bit bytes) ---------------       Description

Symbol Count----Integer-------------           4---------------------------                    Number of symbols for which data is being returned. The subsequent sections are repeated this many times

REPEATING SYMBOL DATA

Symbol Length    Short           2                     Length of the Symbol field

Symbol          String        Variable                 The symbol for which the historical data is returned

Error Code        Byte           1                      0=OK, 1=ERROR

Error Length      Short          2                     Only returned if Error Code=1. Length of the Error string

Error Text        String       Variable                Only returned if Error Code=1. The string describing the error

Bar Count        Integer          4                   # of chart bars; only if error code=0

REPEATING PRICE DATA

close             Float            4  

high              Float            4

Low               Float            4  

open              Float            4

volume            Float            4                   in 100's

timestamp         Long             8                   time in milliseconds from 00:00:00 UTC on January 1, 1970

END OF REPEATING PRICE DATA

Terminator        Bytes            2                   0xFF, 0XFF

END OF REPEATING SYMBOL DATA

As you can see, this file is a mixture of different types of fields. My requirement is to convert this file from the way it is into a fixed field text file (or CSV file). I'm not very good at JavaScript, but I know enough to get by. My main language is MAPPER from Unisys (it is actually called "Business Information Server"). Currently I get all HTTP responses as text files, but this one is a 'binary' file, and MAPPER can not process it because it is a text-based language (a 4GL). I've spent days trying to find a snippet of JavaScript code that I could use, but to no avail. I think this is really simple stuff for a guy that knows JavaScript.

A: 

Hi, I'm a fellow UNISYS programmer. 25 years of FORTRAN 77 on a 2200 mainframe. Happily, I rarely had anything to do with MAPPER.

I'd like to help, but you're not providing enough information.

  • Where is this JavaScript code running? In a browser, or is it an extension to whatever you're using to access MAPPER?
  • Are you using some kind of terminal emulator? AttachMate?
  • Is your data really arriving in a file, or is it in memory? How are you receiving it, how are you passing on the contents?
  • Is it vital that your processing happen in JavaScript? There are dozens of languages that would make very short work of the task if the data were lying around as a file and the output should be a file too.

One problem I see is that, AFAIK, JavaScript doesn't know about file IO. That's why I'm asking where it's running.


EDIT:

OK, somehow you have a browser-like environment and JavaScript running in it.

First, the problem of getting binary data out of your response. Here's a bit of help:

https://developer.mozilla.org/en/using_xmlhttprequest

This is Mozilla documentation, under "Receiving binary data," but I'm hoping there will be enough overlap for it to be useful:

function load_binary_resource(url) {  
  var req = new XMLHttpRequest();  
  req.open('GET', url, false);  
  //XHR binary charset opt by Marcus Granado 2006 [http://mgran.blogspot.com]  
  req.overrideMimeType('text/plain; charset=x-user-defined');  
  req.send(null);  
  if (req.status != 200) return '';  
  return req.responseText;  
}  

The above lets you fiddle with the connection a bit to hopefully obtain binary data.

That function is called like so:

var filestream = load_binary_resource(url);  
var abyte = filestream.charCodeAt(x) & 0xff; 

...and if I understand this correctly, your responseText is a JavaScript string (as usual) but thanks to the fiddling and the binary content, it's not containing printable text but binary data. Heh, as long as you don't try to interpret it, it's just a series of bytes just like any old text.

The second line lets you extract a single byte from any position in the string. That byte will be a value between 0 and 255; or if you're unlucky, between -128 and 127. Not sure how JavaScript deals with signed bytes.

This may look like it's doing you a lot of no good. Let's see how you could get to your data:

Your data starts off with a short called symbolLength. I'm guessing a short is 2 bytes, and I'm guessing that offsets for charCodeAt() begin at 0. So you'll be wanting the first two bytes, or bytes 0 and 1. I'm not sure if your data will be coming in high-endian or low-endian, but you should be able to reconstruct that short from either

var symbolLength = fileStream.charCodeAt(0) + 256 * fileStream.charCodeAt(1);

or

var symbolLength = 256 * fileStream.charCodeAt(0) + fileStream.charCodeAt(1);

In other words, using multiplication to re-assemble bytes into integers.

Integers are presumably 4 bytes, so you'll be multiplying by 4 powers of 256: 16777216, 65536, 256 and 1 - again, either in that order or reversed.

The String data is, of course, just that, and once you've taken into account the number of bytes taken up by the preceding fields, you should be able to dig it out of your response string simply using substring operators.

Now for the yucky part - conversion of floats. The internal structure of those numbers is defined by IEEE 754. float probably corresponds to binary32 and double (if you have any) to binary64. The links from the Wikipedia article I linked explain these formats well enough that you could write your own conversion routine if you were desparate, but in your shoes I'd go looking for ready-built coding for this. Surely you're not the first person faced with converting a handful of bytes into a floating point number. Maybe you can find some C or Java code you could hand-convert, or you could even find a routine already written in JavaScript.

Finally, once you have in hand methods to convert all the data types you mentioned, all you need to do is to format that data in whatever format you want to see downstream in MAPPER. Loop through the structures, incrementing the pointers for the offsets... probably nothing new for you.


Admittedly, I've done a lot of guessing and handwaving here. This could be the beginning of a solution but you'll probably want to do a bit of experimenting and hit SO with some detail questions. Don't mention UNISYS, phrase your question as if you wanted to do this in IE :)

As a first step, I'd try dumping out your incoming binary string, byte-wise, preferrably in hex, to some medium where you can read it and compare the bytes you see with the bytes you're expecting from the input data.

Carl Smotricz
Hi Carl, I'm sorry you never had the pleasure. MAPPER is a great business programming language. They recently added JavaScript to the MAPPER system. I only use it when I have to, and I have to here. The interface is actually straight from MAPPER to the website. Here's a little code that may explain things better:
Gary Erdman
function XMLPRCHST (prcstring,ssid) {try {var FromUrl="https://.../PriceHistory;sessionid=" +ssid +"?var Http = COMCreateObject("Msxml2.XMLHTTP.4.0");var params = prcstring;var HttpResponse;Http.open('POST',FromUrl,false);Http.setRequestHeader("Content-type", "application/x-www-form-urlencoded");Http.send(params);HttpResponse=Http.responseText;COMReleaseObject(Http);var ds = new Dataset();ds.create('I0');var sa = HttpResponse.split('\n');for (var a in sa) {var len = sa[a].length;var i =0;while (len-i > ds.column[1].size) {
Gary Erdman
ds.addRecord(sa[a].substr(i, ds.column[1].size));i += ds.column[1].size;}ds.addRecord(sa[a].substr(i, ds.column[1].size));}ReturnDataset(ds);}
Gary Erdman
That didn't help much, did it? Basically, after submitting the request, the lower portion of code parses the response into max of 998 chars per line and then passes the report back to the calling run in MAPPER where the data is actually interpreted and posted or whatever needs to be done.
Gary Erdman
Ah OK, that *does* help. UNISYS cobbled some functionality on by providing a Dataset class that you can write to... I'm going to expand on my answer.
Carl Smotricz
My answer updated with some pointers that with luck should be enough to solve the problem. Alas, I need to hit the hay but I'll be happy to talk about details if need be, later. My (full, dotted) name will reach me at GMail.com if you feel cramped discussing here. You should be able to edit your question to provide further info to me or others if need be. Good luck!
Carl Smotricz