HttpWebRequest long URI workaround?

Question

I've encountered an issue with HttpWebRequest that if the URI is over 2048 characters long the request fails and returns a 404 error even though the server is perfectly capable of servicing a request with a URI that long. I know this since the same URI that causes an error if submitted via HttpWebRequest works fine when pasted directly into a browser address bar.

My current workaround is to allow users to set a compatability flag to say that it's safe to send the parameters as a POST request instead in the case where the URI would be too long but this is not ideal since the protocol I'm using is RESTful and GET should be used for queries. Plus there is no guarentee that other implementors of the protocol will accept POSTed queries

Is there another class in .Net that has equivalent functionality to HttpWebRequest that doesn't suffer from the URI length limit that I could use?
I'm aware of WebClient but I don't really want to use that as I need to be able to fully control the HTTP Headers which WebClient restricts the ability to do.

Edit

Because Shoban asked for it:

http://localhost/BBCDemo/sparql/?query=PREFIX+rdf%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F1999%2F02%2F22-rdf-syntax-ns%23%3E%0D%0APREFIX+rdfs%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23%3E%0D%0APREFIX+xsd%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema%23%3E%0D%0APREFIX+skos%3A+%3Chttp%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23%3E%0D%0APREFIX+dc%3A+%3Chttp%3A%2F%2Fpurl.org%2Fdc%2Felements%2F1.1%2F%3E%0D%0APREFIX+po%3A+%3Chttp%3A%2F%2Fpurl.org%2Fontology%2Fpo%2F%3E%0D%0APREFIX+timeline%3A+%3Chttp%3A%2F%2Fpurl.org%2FNET%2Fc4dm%2Ftimeline.owl%23%3E%0D%0ASELECT+*+WHERE+{%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+dc%3Atitle+%3Ftitle+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Ashort_synopsis+%3Fsynopsis-short+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Amedium_synopsis+%3Fsynopsis-med+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Along_synopsis+%3Fsynopsis-long+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Amasterbrand+%3Fchannel+.%0D%0A++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Agenre+%3Fgenre+.%0D%0A++++%3Fchannel+dc%3Atitle+%3Fchanneltitle+.%0D%0A++++OPTIONAL+{%0D%0A++++++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Abrand+%3Fbrand+.%0D%0A++++++++%3Fbrand+dc%3Atitle+%3Fbrandtitle+.%0D%0A++++}%0D%0A++++OPTIONAL+{%0D%0A++++++++%3Chttp%3A%2F%2Fwww.bbc.co.uk%2Fprogrammes%2Fb00n4d6y%23programme%3E+po%3Aversion+%3Fver+.%0D%0A++++++++%3Fver+po%3Atime+%3Finterval+.%0D%0A++++++++%3Finterval+timeline%3Astart+%3Fstart+.%0D%0A++++++++%3Finterval+timeline%3Aend+%3Fend+.%0D%0A++++}%0D%0A}&default-graph-uri=&timeout=30000

Which is the following encoded onto the querystring:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#&gt;
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#&gt;
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#&gt;
PREFIX skos: <http://www.w3.org/2004/02/skos/core#&gt;
PREFIX dc: <http://purl.org/dc/elements/1.1/&gt;
PREFIX po: <http://purl.org/ontology/po/&gt;
PREFIX timeline: <http://purl.org/NET/c4dm/timeline.owl#&gt;
SELECT * WHERE {
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; dc:title ?title .
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:short_synopsis ?synopsis-short .
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:medium_synopsis ?synopsis-med .
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:long_synopsis ?synopsis-long .
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:masterbrand ?channel .
  <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:genre ?genre .
  ?channel dc:title ?channeltitle .
  OPTIONAL {
    <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:brand ?brand .
    ?brand dc:title ?brandtitle .
  }
  OPTIONAL {
    <http://www.bbc.co.uk/programmes/b00n4d6y#programme&gt; po:version ?ver .
    ?ver po:time ?interval .
    ?interval timeline:start ?start .
    ?interval timeline:end ?end .
  }

}

Answer 1

the protocol I'm using is RESTful and GET should be used for queries.

There's no reason POST can't also be used for queries; for really long request data you have to, as very-long-URIs aren't globally supported, and have never been. This is one area where HTTP does not live up to the REST ideal.

The reason POST generally isn't used on a plain-HTML level is to stop the browser prompting for reloads, and promote eg. bookmarking. But for HttpWebRequest you don't have either of those concerns, so go ahead and POST it. Web applications should use a parameter or a URI path part to distinguish write requests from queries, not merely the request method. (Of course a write request from a GET method should still be denied.)

Answer 2

The interesting thing is that the same Url works from the browser address bar, but doesnt through HttpWebRequest.

Can you try sniffing the actual Url sent over the wire using Wireshark? maybe it is being encoded differently by .net, and in that process the length is being increased to an unsupported value on the server.

Answer 3

I don't think HttpWebRequest is actually incompatible with GET URLs of the size you are talking about. I say this based on two things:

1. In my own work I use HttpWebRequest to send HTTP GET requests longer than 2048 characters without trouble. I'm not sure what my longest ones are, but we're talking 10,000+ characters. (This is primarily between a web application and an instance of Solr running under Tomcat.)

2. .NET does have some limits on GET URL lengths, but the ones I'm aware of are much higher than 2048 characters. For example, I learned today from my profiler that WebRequest.Create(string url) calls the Uri class constructor, and that is documented to throw a UriFormatException if "the length of uriString exceeds 65534 characters."

I'm not sure where your problem might be, if it's not HttpWebRequest itself. Do you know under what conditions your web service will return HTTP 404 (i.e. "not found")? (I assume the 404 is coming from your web service, rather than being faked inside the depths of .NET.) I'd also want to double-check that the address you're pasting into the browser is actually the same one that's being sent by .NET; as feroze suggested, you should use a network sniffing tool for this. If the two addresses are the same, then maybe next compare how the HTTP headers vary between the .NET case and the browser case. (Incidentally, I personally find Fiddler a bit handier than wireshark for HTTP debugging tasks along these lines.)

See also this somewhat related question: http://stackoverflow.com/questions/1106814/how-does-httpwebrequest-differ-functional-from-pasteing-a-url-into-an-address-b

Answer 4

Your query string is wrong according to RFC3986. '{' and '}' characters are not allowed in a URI.