How to get the correct Content-Length for a POST request.

Question

I am using a perl script to POST to Google Appengine application. I post a text file containing some XML using the -F option.

http://www.cpan.org/authors/id/E/EL/ELIJAH/bget-1.1

There is a version 1.2, already tested and get the same issue. The post looks something like this.

Host: foo.appspot.com
User-Agent: lwp-request/1.38
Content-Type: text/plain
Content-Length: 202

<XML>
   <BLAH>Hello World</BLAH>
</XML>

I have modified the example so the 202 isn't right, don't worry about that. On to the problem. The Content-Length matches the number of bytes on the file, however unless I manually increase the Content-Length it does not send all of the file, a few bytes get truncated. The number of bytes truncated is not the same for files of different sizes. I used the -r option on the script and I can see what it is sending and it is sending all of the file, but Google Appengine self.request.body shows that not everything is received. I think the solution is to get the right number for Content-Length and apparently it isn't as simple as number of bytes on the file or the perl script is mangling it somehow.

Update: Thanks to Erickson for the right answer. I used printf to append characters to the end of the file and it always truncated exactly the number of lines in the file. I suppose I could figure out what is being added by iterating through every character on the server side but not worth it. This wasn't even answered over on the google groups set up for app engine!

Answer 1

Can you post your code here? We really can't say much about your code if we don't know what you're doing.

Answer 2

Is the number of extra bytes you need equal to the number of lines in the file? I ask because perhaps its possible that somehow carriage-returns are being introduced but not counted.

Answer 3

Erickson,

That might be it, I will test.

Answer 4

How are you getting the number of bytes? .. By looking at the size of the file on the filesystem?

You can use "-s" to get the size of the file.

Or, if you want to do more, you may use File::Stat

Answer 5

I've run into similar problems before.

I assume you're using the length() function to determine the size of the file? If so, it;s likely the data that you're posting is UTF-8 encoded, instead of ASCII.

To get the correct count you may need to add a "use bytes;" pragma to the top of your script, or wrap the length call in a block:

my $size; do {use bytes; $size = length($file_data)}

From the perlfunc man page:

"Note the characters: if the EXPR is in Unicode, you will get the number of characters, not the number of bytes."