How to get data with JavaScript from another server?

Question

How can I make requests to other server(s) (i.e. get a page from any desired server) with a JavaScript within the user's browser? There are limitations in place to prevent this for methods like XMLHttpRequest, are there ways to bypass them or other methods?

That is a general question, specifically I want to check a series of random websites and see if they contain a certain element, so I need the HTML content of a website without downloading any additional files; all that in a JavaScript file, without any forwarding or proxy mechanism on a server.

(Note: one way is using Greasemonkey and its GM_xmlhttpRequest.)

Answer 1

You should check out jQuery. It has a rich base of AJAX functionality that can give you the power to do all of this. You can load in an external page, and parse it's HTML content with intuitive CSS-like selectors.

An example using $.get();

$.get("anotherPage.html", {}, function(results){
  alert(results); // will show the HTML from anotherPage.html
  alert($(results).find("div.scores").html()); // show "scores" div in results
});

For external domains I've had to author a local PHP script that will act as a middle-man. jQuery will call the local PHP script passing in another server's URL as an argument, the local PHP script will gather the data, and jQuery will read the data from the local PHP script.

$.get("middleman.php", {"site":"http://www.google.com"}, function(results){
  alert(results); // middleman gives Google's HTML to jQuery
});

Giving middleman.php something along the lines of

<?php

  // Do not use as-is, this is only an example.
  // $_GET["site"] set by jQuery as "http://www.google.com"
  print file_get_contents($_GET["site"]);

?>

Answer 2

You will need to write a proxy on the server to do this. And all requests will be to your server and then your server will load html and send it back to client. And there are no good way to implement this via javascript only.
jQuery contains functionality to load JSON data or external scripts using XmlHttpRequest but this functionality can not be used for html pages. Also you may check this thread of jQuery mailing list.

Answer 3

Write a proxy script that forwards along the http request from your domain, this will bypass the XMLHttpRequest restrictions.

If your using PHP, simply use cURL to request and read the page, then simply spit out the html as if it was from you domain.

Answer 4

You can also use a iframe to emulate an ajax request. This saves you the mess of having to code a Backend solution for a Frontend problem. Here is an example:

function setUploadEvent(typeComponet){
       var eventType = "";
       var iframe = document.getElementById("iframeId");
       try{
           /* for Mozilla / Opera9 */
           if (/(?!.*?compatible|.*?webkit)^mozilla|opera/i.test(navigator.userAgent)) {
                eventType = "onload";
           }else{
           /* IE  */
                eventType = "onreadystatechange";
           }

           iframe[eventType] = function(){
                var doc = iframe.contentDocument || iframe.contentWindow.document;
                var response = doc.body.innerHTML; /* or what ever content you are looking for */
             }
           }
       catch(e){
           alert("Error loading content")}
       }

That should do the trick. Please note that the Browser detection line is not the cleanest, you should absolutely use the ones provided in all the most common JS frameworks (Prototype, JQuery, etc)

Answer 5

<script language="JavaScript" type="text/javascript" src="http://www.example.com/hello.js">&lt;/script>

You add the data into hello.js in as an array, JSON or similar. Example: var daysInMonth = new Array(31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31);

Getting a JavaScript from another server doesn't much simpler.. :-)