How to fix my crawler in C#?

Question

Regex hrefs = new Regex("<a href.*?>");
Regex http = new Regex("http:.*?>");  
StringBuilder sb = new StringBuilder();
WebClient client = new WebClient();
string source = client.DownloadString("http://google.com");
foreach (Match m in hrefs.Matches(source)){
sb.Append(http.Match(m.ToString()));
Console.WriteLine(http.Match(m.ToString()));
}

The codes works fine, but just once problem Look at the output.

http://images.google.se/imghp?hl=sv&tab=wi" onclick=gbar.qs(this) class=gb1>
http://video.google.se/?hl=sv&tab=wv" onclick=gbar.qs(this) class=gb1>
http://maps.google.se/maps?hl=sv&tab=wl" onclick=gbar.qs(this) class=gb1>
http://news.google.se/nwshp?hl=sv&tab=wn" onclick=gbar.qs(this) class=gb1>
http://translate.google.se/?hl=sv&tab=wT" onclick=gbar.qs(this) class=gb1>
http://mail.google.com/mail/?hl=sv&tab=wm" class=gb1>
http://www.google.se/intl/sv/options/" onclick="this.blur();gbar.tg(event);return !1" aria-haspopup=true class=gb3>
http://blogsearch.google.se/?hl=sv&tab=wb" onclick=gbar.qs(this) class=gb2>
http://www.youtube.com/?hl=sv&tab=w1&gl=SE" onclick=gbar.qs(this) class=gb2>
http://www.google.com/calendar/render?hl=sv&tab=wc" class=gb2>
http://picasaweb.google.se/home?hl=sv&tab=wq" onclick=gbar.qs(this) class=gb2>
http://docs.google.com/?hl=sv&tab=wo" class=gb2>
http://www.google.se/reader/view/?hl=sv&tab=wy" class=gb2>
http://sites.google.com/?hl=sv&tab=w3" class=gb2>
http://groups.google.se/grphp?hl=sv&tab=wg" onclick=gbar.qs(this) class=gb2>
http://www.google.se/ig%3Fhl%3Dsv%26source%3Diglk&usg=AFQjCNEsLWK4azJkUc3KrW46JTUSjK4vhA" class=gb4>
http://www.google.se/" class=gb4>
http://www.google.com/intl/sv/landing/games10/index.html">
http://www.google.com/ncr">

How can i remove the html tags?

Answer 1

Substring the following line http.Match(m.ToString()) to http.Match(m.ToString().remove(m.ToString().IndexOf("\"")))

Not the cleanest way to do it but it works

Answer 2

Change the closing tag to be "

Answer 3

Change your regex to:

Regex http = new Regex("http:.*?\"");

Or even better, parse all links using HtmlAgilityPack and Xpath:

var web = new HtmlWeb();
var doc = web.Load("http://www.stackoverflow.com");

var nodes = doc.DocumentNode.SelectNodes("//a[@href]"); // Will find all links

foreach (var node in nodes)
{
    Console.WriteLine(node.InnerHtml);
}

Answer 4

The quick solution is to change this:

Regex http = new Regex("http:.*?>");

To this:

Regex http = new Regex("http:.*?\"");

The better solution is to use a library to parse the html - the HTML Agility Pack can be used for that and will make your life easier.

Answer 5

Regex hrefs = new Regex("<a href.*?>");
Regex http = new Regex("(http:.*?)\"");  
StringBuilder sb = new StringBuilder();
WebClient client = new WebClient();
string source = client.DownloadString("http://google.com");
foreach (Match m in hrefs.Matches(source))
{
    var value = http.Match(m.ToString()).Groups[1].Value;
    sb.Append(value);
    Console.WriteLine(value);
}

Answer 6

A nice and simple solution. Match any character after http: except " character

"http:[^\"]*"