Web request returns "DOS"

Question

I am getting a "DOS" instead of the html string ....

let getHtmlBasic (uri :System.Uri ) =
    use client = new WebClient()
    client.DownloadString( uri)


let uri = System.Uri( "http://www.b-a-r-f.com/" )
getHtmlBasic uri

This gives a string, "DOS"

Lol what the ?

All other websites seems to work ...

Answer 1

It's based on the user agent. If you use a browser user agent, the request will work. That said, it seems to be a pretty clear Keep Out message.

Answer 2

They probably detect automated crawling and send you that response.

Answer 3

I've been testing myself (not in F but it doesn't really matter) and I can confirm that the site reads the User Agent string and depending on its value it either returns the site contents or the "DOS" text.

Curiously, they provide a feed service (both RSS and Atom) and they also filter out requests to it.

Although the User Agent information can be easily faked, my advice is that you try to get their permission to grab contents; at least from the XML feed!

Answer 4

let req =  (WebRequest.Create(uri)) :?> HttpWebRequest
// 'use' is equivalent to ‘using’ in C# for an IDisposable
req.UserAgent<-"Mozilla"

Thanks to all !!!