Google App Engine - throttle_code=4 when under high load in load testing

Question

I'm running some load tests using JMeter to get an idea of how my application performs under load on Google App Engine (Java).

When I hit around 100 virtual users my response time starts dropping, and I notice "throttle_code=4" appearing in the GAE log files (see below).

Does anyone know what this means? Is the fact that I'm making multiple requests from the same IP hitting some kind of automatic DDOS protection?

Thanks

---- request with"throttle_code=4"; when connecting from JMeter
07-05 05:11AM 58.917 /market/1234/history?pricebars=240 200 3651ms 19cpu_ms 35kb Java/1.5.0_16,gzip(gfe)
75.101.226.4 - - [05/Jul/2010:05:12:02 -0700] "GET /market/1234/history?pricebars=240 HTTP/1.1" 200 35566 - "Java/1.5.0_16,gzip(gfe)" "ci-pricehistory.appspot.com" ms=3652 cpu_ms=19 api_cpu_ms=0 cpm_usd=0.004527 pending_ms=3570 throttle_code=4
I 07-05 05:12AM 02.532
uk.co.cityindex.CandleServlet fetch: Loading 240 bars from cache...
I 07-05 05:12AM 02.558
uk.co.cityindex.CandleServlet fetch: time:47

---- request without "throttle" log; when connecting from browser
07-05 06:28AM 10.993 /market/1234/history?pricebars=240 200 69ms 19cpu_ms 7kb Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4,gzip(gfe)
80.169.172.178 - - [05/Jul/2010:06:28:11 -0700] "GET /market/1234/history?pricebars=240 HTTP/1.1" 200 7572 - "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4,gzip(gfe)" "ci-pricehistory.appspot.com" ms=69 cpu_ms=19 api_cpu_ms=0 cpm_usd=0.001423
I 07-05 06:28AM 11.031
uk.co.cityindex.CandleServlet fetch: Loading 240 bars from cache...
I 07-05 06:28AM 11.055
uk.co.cityindex.CandleServlet fetch: time:45

Answer 1

See (apparently) official Google answer here, though I'm not sure it totally answers the question.

It looks like your request is taking 3500ms. We autoscale up the number of instances servicing your requests as long as you maintain average milliseconds/request of 1000ms or under (we recommend 800ms, sub 400ms is optimal). If you run lots of long running requests, we will not spin up new instances for you and you will hit a scaling ceiling.

http://osdir.com/ml/GoogleAppEngine/2010-07/msg00165.html

Apparently, they throttle based on your 'efficiency,' but it looks like your original request was only 69ms. Was your app's average ms/req higher because of other requests, possibly? It seems like a weird tactic NOT spin up because your app is slow if your app is slow because it hasn't yet spun up...

The other thing I notice is that your test request through JMeter is 35kb vs the 7kb of the other request - does your app's data increase under testing load? Headers piling up?