Have a look at this - Php exec and return binary
Are you sending proper headers??
something like
header("Content-Type: application/octet-stream");
header("Content-Disposition: attachment; filename=\"yourfile.xlsx\"");
UPDATE
header('Content-Type: application/xls');
header('Content-Disposition: attachment; filename=example.xlsx');
header('Pragma: no-cache');
echo file_get_contents("/path/to/yourfile.xlsx");
UPDATE 2
Spread sheet mime types
application/vnd.ms-excel [official]
application/msexcel
application/x-msexcel
application/x-ms-excel
application/vnd.ms-excel
application/x-excel
application/x-dos_ms_excel
application/xls
UPDATE 3
Regarding your javascript problem did you try using
location.href instead of window.open ??