I have read previous posts and used suggestions about firebug and selenium etc. I am still a newbie and am still pretty confused as how to implement this. I am trying to access URLs that appear when I click on a link. I have accessed the JavaScript code and the function just has a couple of parameters which I have no idea how to intercept using vb.net or C# etc. Can this be done using firebug?
A:
If you goal is the get hold of the URLs that are generated by Javascript code when you click on hyperlinks, then using Firebug's Net tab is one solution. HTTP proxies like Fiddler, HttpWatch or Charles are other alternatives. I don't understand where vb.net is in the picture though.
Murali
2009-12-28 16:43:45
Thanks I will take a look into both suggestions. VB.net was just the lang I used for the crawler.
vbNewbie
2009-12-28 16:53:31
A:
Sounds like you want to use Firebug to monitor the AJAX calls:
http://code.google.com/gme/articles/gmefirebug.html#section8
DA
2009-12-28 16:43:58
I monitored the Net tab and I am able to view the href tags with the urls I need. Is there a way to automate the process of extracting these urls via .NET? Can I access firebugs view from a program in the .NET IDE? Reason I ask is because there are over 10000 urls I need.
vbNewbie
2009-12-28 18:25:18
can you show us the page you are trying to parse? I have a hunch you're going to want to do either XML or Regex text parsing of the source of the page in question.
DA
2009-12-29 13:36:19
http://caps.fool.com/stats.aspxlook at the links for the top ten lists. I cannot use regex parsing for this.
vbNewbie
2009-12-29 15:04:42
the approach I have taken now is rather tedious. I get the script view with firebug for each list and copy and paste into a text document which I will later parse.
vbNewbie
2009-12-29 15:10:34