views:

856

answers:

7

I have a small (or perhaps not so small) issue with some ASP.NET/Castle Monorail sites for a product I work on. This is a fairly legacy system (written well before my time), and it uses GET requests with a considerable amount of information in the query string. We have recently run into query string length limitations, and for the amount of data we need to transfer to the server, storing the data temporarily in a cookie is also not plausible (we are already well past the 4096 byte limit per cookie, and we set a lot of cookies, so we are likely near or at the limit of cookies per domain as well.)

I am wondering if there are any alternatives, besides POST (in some cases changing to a POST request may be possible, but likely not in all), that might solve this problem. I'm hoping someone else here on StackOverflow has run into similar issues, and has some wizardly solution (i.e. compress data with javascript, encode as base64, pass to a single query string item? Just not sure if there are any libraries that can compress data with javascript in a way compatible with built-in compression classes in .NET 3.5.)

UPDATE:

Solution I ended up choosing was to POST to a temporary controller. This temp controller pulled the large list of data, stuck it in a shared Session (the production servers are in a large multi-bank server farm that does not use sticky sessions/IPs), and performed a GET to the actual controller, which pulled the data from the shared Session. Not the most performant solution, but it solved the problem.

+1  A: 

Beyond switching to POST your options are going to be limited. Base64 is going to increase the size of the data not decrease it. You mention compressing the data into a single query string item... maybe instead you could split the data into 2 separate requests using Ajax? I'm not sure if that's feasible for your situation, though.

John Rasch
+1 for AJAXing some data over the pipe
Janie
Ajax may be a possibility in some cases. Problem is there are a quite a number of pages that have the problem, and each have different requirements. Thanks for the reminder about base64...I always forget that it doubles the data size. *sigh*
jrista
A: 

I would not rely too much on javascript. Compacting the query string would be the first natural thing to do; but if you're way ahead those limits, you should try mentaining sessions. You start up fresh with default state info, then gradually reflect the state on the server into the user's session with POST data provided, so you partition all that data into many handlers (read pages).

+3  A: 

There are several Base64 encode plugins for jQuery, but that doesn't help because Base64 generally makes the data longer, not shorter.

Depending on the data, I'd look at some other text compression techniques. Wikipedia lists a few:

  • Context Tree Weighting method (CTW)
  • Burrows-Wheeler transform (block sorting preprocessing that makes compression more efficient)
  • LZ77 (used by DEFLATE)
  • LZW

Here's an LZW implementation for Javascript.

Huffman encoding is based on letter frequency, so it might not be appropriate for your data. You'll probably have to escape the result to make it URL-safe.

Jon Galloway
Do you know of any compression algorithms for javascript or jQuery? I can find some other encoding, its the compression I am really interested in...
jrista
There's http://rumkin.com/tools/compression/compress_huff.php. You can see the code if you view source.
John Fisher
John, I'd already linked to that (see the link for Huffman encoding). The one thing to keep in mind is that Huffman works from standard character frequencies in the English language, so if your form data isn't English text it might not be very efficient.
Jon Galloway
A: 

The textbook answer is to use POST. Why would this not be possible? The reason POST was invented was pretty much to get around length limits in GET.

Another possibility depending on the details is to have multiple screens, each with a subset of the data, and store the data from each on the server and connect them up at the end.

Jay
The reason POST was invented was not to get around the length limitations in GET.
Duncan Beevers
"The reason POST was invented was pretty much to get around length limits in GET." Sorry but I laughed my ass off when I read this.
Josh Stodola
Okay, I'm willing to be educated. Why was POST invented then? Yes, I understand that technically GET is supposed to be used for idempotent transactions and POST for non-idempotent transactions. But in real life, the practical difference between them is that GET has a sharp length limit and POST doesn't. In JSP, GET and POST are processed by exactly the same logic. You have to go out of your way to check and distinguish them.
Jay
A: 

Is the query string limitation on the client or the server?

If the issue is just that the client will not send the long GET string, perhaps you can POST to a proxy server which then forwards the request as a GET.

Duncan Beevers
for query string limitation reference:http://en.wikipedia.org/wiki/Query_string#Compatibility_issues
smoothdeveloper
A: 

Is this to hold state information between pages? If so, an option could be to use a cookie with it's value a GUID as such:

_SESSIONID=3F2504E0-4F89-11D3-9A0C-0305E82C3301

And store the session information server-side. The session is uniquely identified with a GUID, so it is then easy to retrieve the large amount of information directly on the server.

All you then need to send/retrieve client-side is that session id, as well as client generated information (GET/POST fields).

Andrew Moore
Its a massive set of ID's that are processed server-side.
jrista
Well, one SESSIONID by client connected. That's the way PHP handles sessions by default with no real performance lost. Your application will actually behave faster since there is less transfer.
Andrew Moore
+1  A: 

I would suggest using JSON as the data transfer format if an array like query string is generated as follows:

params[]=sth&params[]=sth2&params[]=sth3

This would be

{params:['sth1', 'sth2', 'sth3']}

You can sssign this JSON string to a single letter variable like below

p={params:['sth1', 'sth2', 'sth3']}

Something even better would be compressing and Base64 encoding the whole query string. Since you generate the query string on the server side(I assume this but same thing can be done in JS by using the same comp/decomp algorithms) you can use any builtin compression algorithm with your programming language such as gzip. After Base64 encoding the compressed data, yes it will expand a bit but not as much as url encoding expands it.

BYK
Thanks for the out of the box idea (that's what I was looking for...out of the box ideas.) Transforming the data to JSON saves a small amount of space...but that lead me to just simplifying it as much as humanly possible...so I ended up with basically this: p=123234,123235,123237,288721,... Still not a long term solution (I think I still need some kind of compression/encoding), but it might suffice until this system is completely rewritten next year.
jrista
BYK
What kind of compression did you use? I've been messing with the huffman algorithms posted by others, but for the data I have, it isn't efficient enough to offset the encoding expansion...
jrista
Well we neded that functionality in JS so I have used U-LZSS library which can be found @ Google Code. I had to adapt its decompressor to PHP but it was not a big deal. I suggest using gzip if it is already implemented in ASP.NET and you do not need JS. LZSS is a variant of LZW and it may not be as efficient as DEFLATE based algorithms but worth a try ;)
BYK
The problem isn't the speed, its the fact that the URL that is created is too long. The gzip used on the wire won't help me with a URL that is too long...I was hoping to do something that would shrink the length of the URL down before issuing the request.
jrista
You have misunderstood me. I was referring to the query string compression, not the embedded gzip of the browsers. Think exactly as the json example I have given. Compress the original query string, base64 encode it so that it wont double or triple its size when you url encode it and then assign it to a single letter variable. Now you have a much shorter URL and query string which holds the exact same data.
BYK