What's a good book about web crawling?

Question

I'm searching for a good book which discusses the problems of developing a web-crawler in java. Not this typical "search engine optimization" books.

What books can you recommend?

Answer 1

Maybe Lucene in Action

Answer 2

I don't know of any certain book about it but it has a very wide range of topics where you might want to take a look at:

• Evaluating existing Open Source Crawlers in Java
• HTTP based connections e.g. working with Apache Commons net
• HTML/XML processing in Java
• Indexing (e.g. Lucene)
• Processing algorithms (e.g. Googles Map Reduce) for working with the data

Answer 3

I've referenced the O'Reilly book Spidering Hacks on several occasions when I've needed to do something along these lines :)

Answer 4

First, a good base in HTTP & its surrondings (URI, URL, cookies, etc...) would be good, so:

HTTP: The Definitive guide

HTTP Developer's handbook

And two books that seems spot on:

HTTP Programming Recipes for Java Bots

Programming spiders, bots and aggregators in Java