What should I log in AJAXy web application and how long to store the logs?

Question

After some time of developing AJAXy application I found out, that my log files are growing drasticaly fast. Even when I am the only user testing the application, I get about 90MB of log ever few hours.

I'm logging every request, parameters and how long did it take, so single log isn't very large (not bigger than 4 lines), but since there are pretty high number of requests, it grows instanely fast.

I don't even try to image what would happen when this would go in production and app will be used not by one, but by thousands of users at the same time, which would cause logs to grow to enormous size.

So my questions are

• What kind of data should I store in logs?
• What shouldn't be logged?
• How long should I store logs? (hour, day, week, month, 10 000 years or so ..)

How does big sites like Facebook or Twitter handle logging anyway? Do they log only errors and exceptions?

Answer 1

I would log only php errors and notices, as well as all the search queries, otherwise the app spends more time on writing to log than on serving your user. Once you read your logs and fix issues you will have less notices and errors from front end. Just save them till you think that the issue is not fixed yet.