Python code to download a webpage using JavaScript

Question

Hi,

Im trying to download share data from a stock exchange using python. The problem is that there is no direct download link, but rather a javascript to export the data.

The data page url:

http://tase.co.il/TASE/Templates/Company/CompanyHistory.aspx?NRMODE=Published&NRORIGINALURL=%2fTASEEng%2fGeneral%2fCompany%2fcompanyHistoryData.htm%3fcompanyID%3d001216%26ShareID%3d01091248%26subDataType%3d0%26&NRNODEGUID={045D6005-5C86-4A8E-ADD4-C151A77EC14B}&NRCACHEHINT=Guest&shareID=01820083&companyID=000182&subDataType=0

When I open the data page in a browser and then the download page, it works like a charm. When I just open the download page, it doen't download anything. I guess this is because the data page injects the actual data to the variables Columns, Titles etc.

I've tried to mimic this behaviour in a python script, but without success.

def download_CSV (shareID, compID):
 data_url ="http://tase.co.il/TASE/Templates/Company/CompanyHistory.aspx?NRMODE=Published&NRORIGINALURL=%2fTASEEng%2fGeneral%2fCompany%2fcompanyHistoryData.htm%3fsubDataType%3d0%26shareID%3d00759019&NRNODEGUID={045D6005-5C86-4A8E-ADD4-C151A77EC14B}&NRCACHEHINT=Guest&shareID="+shareID+"&companyID="+compID+"&subDataType=0"  
 import urllib2
 response = urllib2.urlopen(data_url)
 html = response.read()

 down_url ="http://tase.co.il/TASE/Pages/Export.aspx?tbl=0&Columns=AddColColumnsHistory&Titles=AddColTitlesHistory&sn=dsHistory&enumTblType=GridHistorydaily&ExportType=3"  
 import urllib
 urllib.urlretrieve (down_url, "test.csv")

Thanks a lot

Answer 1

You could use Selenium or other ways to automate a browser to take advantage of the browser's built-in Javascript interpreter -- to control Selenium with Python, see e.g. here.