ansaurus

Question

Can Javascript read the source of any web page?

Answer 1

+1 A:

As a security measure, Javascript can't read files from different domains. Though there might be some strange workaround for it, I'd consider a different language for this task.

nickf 2009-03-25 07:37:25

Answer 2

+1 A:

If you absolutely need to use javascript, you could load the page source with an ajax request.

Note that with javascript, you can only retrieve pages that are located under the same domain with the requesting page.

kkyy 2009-03-25 07:39:20

Answer 3

+2 A:

You could simply use XmlHttp (AJAX) to hit the required URL and the HTML response from the URL will be available in the responseText property. If it's not the same domain, your users will receive a browser alert saying something like "This page is trying to access a different domain. Do you want to allow this?"

Cerebrus 2009-03-25 07:40:46

Answer 4

A:

Using jquery

<html>
<head>
<script src="http://jqueryjs.googlecode.com/files/jquery-1.3.2.js" ></script>
</head>
<body>
<script>
$.get("www.google.com", function(response) { alert(response) });
</script>
</body>

Sergej Andrejev 2009-03-25 07:49:44

You can't request a page outside of your domain in this way, you have to do it via proxy, e.g. $.get('http://mydomain.com/?url=www.google.com')

karim79 2009-03-25 08:03:30

Answer 5

+9 A:

Javascript can be used, as long as you grab whatever page you're after via a proxy on your domain:

<html>
<head>
<script src="/js/jquery-1.3.2.js"></script>
</head>
<body>
<script>
$.get("www.mydomain.com/?url=www.google.com", function(response) { 
    alert(response) 
});
</script>
</body>

karim79 2009-03-25 08:06:08

Answer 6

+5 A:

Simple way to start, try jQuery

$("#links").load("/Main_Page #jq-p-Getting-Started li");

More at jQuery Docs

Another way to do screen scraping in a much more structured way is to use YQL or Yahoo Query Language. It will return the scraped data structured as JSON or xml.
e.g.
Lets scrap stackoverflow.com

select * from html where url=http://stackoverflow.com

will give you a JSON array(I chose that) like this

 "results": {
   "body": {
    "noscript": [
     {
      "div": {
       "id": "noscript-padding"
      }
     },
     {
      "div": {
       "id": "noscript-warning",
       "p": "Stack Overflow works best with JavaScript enabled"
      }
     }
    ],
    "div": [
     {
      "id": "notify-container"
     },
     {
      "div": [
       {
        "id": "header",
        "div": [
         {
          "id": "hlogo",
          "a": {
           "href": "/",
           "img": {
            "alt": "logo homepage",
            "height": "70",
            "src": "http://i.stackoverflow.com/Content/Img/stackoverflow-logo-250.png",
            "width": "250"
           }
……..

The beauty of this is that you can do projections and where clauses which ultimately gets you the scraped data structured and only the data what you need (much less bandwidth over the wire ultimately)
e.g

select * from html where url="http://stackoverflow.com" and
      xpath='//div/h3/a'

will get you

 "results": {
   "a": [
    {
     "href": "/questions/414690/iphone-simulator-port-for-windows-closed",
     "title": "Duplicate: Is any Windows simulator available to test iPhone application? as a hobbyist who cannot afford a mac, i set up a toolchain kit locally on cygwin to compile objecti … ",
     "content": "iphone\n                simulator port for windows [closed]"
    },
    {
     "href": "/questions/680867/how-to-redirect-the-web-page-in-flex-application",
     "title": "I have a button control ....i need another web page to be redirected while clicking that button .... how to do that ? Thanks ",
     "content": "How\n                to redirect the web page in flex application ?"
    },
…..

Now to get only the questions we do a

select title from html where url="http://stackoverflow.com" and
      xpath='//div/h3/a'

Note the title in projections

 "results": {
   "a": [
    {
     "title": "I don't want the function to be entered simultaneously by multiple threads, neither do I want it to be entered again when it has not returned yet. Is there any approach to achieve … "
    },
    {
     "title": "I'm certain I'm doing something really obviously stupid, but I've been trying to figure it out for a few hours now and nothing is jumping out at me. I'm using a ModelForm so I can … "
    },
    {
     "title": "when i am going through my project in IE only its showing errors A runtime error has occurred Do you wish to debug? Line 768 Error:Expected')' Is this is regarding any script er … "
    },
    {
     "title": "I have a java batch file consisting of 4 execution steps written for analyzing any Java application. In one of the steps, I'm adding few libs in classpath that are needed for my co … "
    },
    {
……

Once you write your query it generates a url for you

http://query.yahooapis.com/v1/public/yql?q=select%20title%20from%20html%20where%20url%3D%22http%3A%2F%2Fstackoverflow.com%22%20and%0A%20%20%20%20%20%20xpath%3D'2F%2Fdiv%2Fh3%2Fa'%0A%20%20%20%20&format=json&callback=cbfunc

in our case.

So ultimately you end up doing something like this

var titleList = $.getJSON(theAboveUrl);

and play with it.

Beautiful, isn’t it?

Cherian 2009-03-25 08:09:08

Answer 7

A:

when using many of the methods listed above, you will find that sites like youtube, for example, will block your code, either by returning error statements instead of the page's html, or else preventing you from accessing the html entirely if you try to access a series of pages or the same page multiple times. Recently i tried to scrape the html from my "view friends" page, "http://www.youtube.com/profile?user=myUserName&view=friends", and got, instead of the html containing my list of friends, error messages. yet the code worked perfectly for other web pages/sites.

doonrothmani 2010-06-29 15:02:05

ansaurus

tags:

views:

answers:

Can Javascript read the source of any web page?

related questions