Fetch data from Other Websites

Question

Hi,

Even before telling my question, will tell you that this is a very vague question. But please let me know if you have any similar ideas.

I am actually trying to write a website of my own locally using ASP.Net. I actually wanted to try and simulate a website with trading and stock details. I wanted to actually fetch the details from some were. Please let me know if it is possible to fetch data from another website or is there any site where i can fetch such details. Also please let me know of what all parameters i need to keep in mind before and while designing a website. Please let me know any such ideas which i can make use of. Thank you one and all very very much. Any comment from you is appreciated.

Answer 1

Yes, of course you can fetch data using System.Net classes such as System.Net.WebClient.

Answer 2

Yes, you can use ASP.Net to fetch data from other websites, though generally speaking if you're planning on making a large site the scraping off other sites is done "offline", and the ASP.Net application serves off this cache of crawled pages. Keep in mind that heavy traffic coming from a single IP (ie, your server) is likely to raise flags with a sysadmin unless you have a prior agreement with the site, and you might just get your IP banned to cool it down.

Answer 3

Generally I'd grab data like that from a webservice provided by the data source. Although I doubt you could find such a service for free.

Answer 4

These articles may come in handy for you to see how data is retrieved and also manipulated:

http://aspnetlibrary.com/articledetails.aspx?article=Retrieve-data-from-a-web-page

and:

http://aspnetlibrary.com/articledetails.aspx?article=Convert-HTML-tables-to-a-DataSet

Answer 5

Some of the basics you can begin looking into is the HttpWebRequest & HttpWebResponse classes, below is a simple example grabbed from MSDN

    // Create an HttpWebRequest using WebRequest.Create (see .NET docs)!
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);


    // Execute the request and obtain the response
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();