How to improve the throughput of request_logs on Google App Engine

Question

Downloading logs from App Engine is nontrivial. Requests are batched; appcfg.py does not use normal file IO but rather a temporary file (in reverse chronological order) which it ultimately appends to the local log file; when appending, the need to find the "sentinel" makes log rotation difficult since one must leave enough old logs for appcfg.py to remember where it left off. Finally, Google deletes old logs after some time (20 minutes for the app I use).

As an app scales, and the log generation rate grows, how can one increase the speed of fetching the logs so that appcfg.py does not fall behind?

Answer 1

You can increase the per-request batch size of logs. In the latest SDK (1.3.1), check out google_appengine/google/appengine/tools/appcfg.py around like 861 (RequestLogLines method of LogsRequester class). You can modify the "limit" parameter.

I am using 1000 and it works pretty well.