Alternative to innerhtml that includes header?

Question

I'm trying to extract data from the following page:

http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#

Which, conveniently and inefficiently enough, includes all the data embedded as a csv file in the header, set as a variable called gs_csv.

How do I extract this? Document.body.innerhtml skips the header where the data is, what is the alternative that includes the header (or better yet, the value associated with gs_csv)?

(Sorry, new to all this, I've been searching through loads of documentation, and trying a lot of them, but nothing so far has worked).

---

Thanks to Sinan (this is mostly his solution transcribed into Python).

import win32com.client 

import time 

import os 

import os.path

ie = Dispatch("InternetExplorer.Application") 

ie.Visible=False 

ie.Navigate("http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#")

time.sleep(20)

webpage=ie.document.body.innerHTML

s1=ie.document.scripts(1).text 

s1=s1[s1.find("gs_csv")+8:-11]

scriptfilepath="c:\FO Share\bmreports\script.txt" 

scriptfile = open(scriptfilepath, 'wb') 

scriptfile.write(s1.replace('\n','\n')) 

scriptfile.close()

ie.quit

Answer 1

Untested: Did you try looking at what Document.scripts contains?

UPDATE:

For some reason, I am having immense difficulty getting this to work using the Windows Scripting Host (but then, I don't use it very often, apologies). Anyway, here is the Perl source that works:

use strict;
use warnings;

use Win32::OLE;
$Win32::OLE::Warn = 3;

my $ie = get_ie();

$ie->{Visible} = 1;

$ie->Navigate(
    'http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?'
    .'param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#'
);

sleep 1 until is_ready( $ie );

my $scripts = $ie->Document->{scripts};

for my $script (in $scripts ) {
    print $script->text;
}

sub is_ready { $_[0]->{ReadyState} == 4 }

sub get_ie {
    Win32::OLE->new('InternetExplorer.Application', 
        sub { $_[0] and $_[0]->Quit },
    );
}

__END__

C:\Temp> ie > output

output now contains everything within the script tags.

Answer 2

fetch the source of that page using ajax, and parse the response text like XML using jquery. It should be simple enought to get the text of the first tag you encounter inside the

I'm out of touch with jquery, or I would have posted code examples.

EDIT: I assume you are talking about fetching the csv on the client side.

Answer 3

If this is just a one off script then exctracting this csv data is as simple as this:

import urllib2

response = urllib2.urlopen('http://www.bmreports.com/foo?bar?')
html = response.read()
csv = data.split('gs_csv=')[1].split('</SCRIPT>')[0]

#process csv data here

Answer 4

Thanks to Sinan (this is mostly his solution transcribed into Python).

import win32com.client

import time import os

import os.path

ie = Dispatch("InternetExplorer.Application") ie.Visible=False

ie.Navigate("http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#")

time.sleep(20)

webpage=ie.document.body.innerHTML

s1=ie.document.scripts(1).text s1=s1[s1.find("gs_csv")+8:-11]

scriptfilepath="c:\FO Share\bmreports\script.txt"

scriptfile = open(scriptfilepath, 'wb')

scriptfile.write(s1.replace('\n','\n'))

scriptfile.close()

ie.quit