Do you think analyzing generate logs during testing cycles can increase quality?
A:
I don't think that there is a good general answer to that question: It depends.
Analyzing logs can be very valuable if: a) there is good information to be mined b) the information is analyzed by people or machines that understand the bottom line, business logic.
without those two ingredients, they can be a resource (time and $) sinkhole
MexDev
2010-03-22 19:14:24
You are right, we had to add some more descriptive application oriented info to the logs to improve testing visibility and data.
bobyg
2010-05-05 13:44:57
A:
Are you thinking integration testing? or unit testing? I would think that logging anomalous events during integration testing, i.e., potential errors that your code logs and recovers from rather than crashing, might be useful. Run your integration tests, then check to see if there are any unexpected anomalies even though your tests pass -- or if they fail, use the logs to help trace the failures.
I can't really see where they would be very useful in unit testing. Your unit tests should be prescriptive. You might want to check that proper logging is taking place in your unit test for anomalous events, but I don't see how the logs should be able to tell you anything that the test output doesn't
tvanfosson
2010-03-22 19:16:42
We did intergation with JUnit and analyzed logs generated by the tested apps to identify new error messages amon unique ones, we alos created reports and compared logs generated during different execution and test cycles. Also usuefull in load testing
bobyg
2010-05-05 13:44:10
+1
A:
My standard opinion is that logs are extremely useful every time they bring to light a problem and not useful at all if they don't tell me anything valuable. It will completely depend upon what they contain. I don't use them for unit testing but at the integration testing time, they can be quite valuable.
I've found logs particularly useful when the testing being done by someone at a different location. On more than one occasion, I was unable to duplicate a problem but after a look at the log data, it was clear what was happening.
itsmatt
2010-03-22 19:23:37
After some evaluations we are using XpoLog(http://www.xpolog.com) for Testing labs. We compare logs, index unique messages, integrated to our Jira issue tracking and more
bobyg
2010-05-05 13:41:31
A:
Absolutely; not so much for unit tests as mentioned above, but if you set up a log analysis tool such as logwatch or Splunk that can send you a summary of errors found during the test, you can definitely increase quality.
gareth_bowles
2010-03-22 20:15:44
Those two dont give good features for testing, just for manual limited simple log browsing.
bobyg
2010-04-06 08:18:30
Incorrect; especially using Splunk, you can automate the heck out of log analysis and get custom alerts when your test criteria pass or fail, including all the info you need to find the cause of a problem (no, I don't work for Splunk). The only manual part of the process is reading the alerts. This certainly doesn't replace formal testing of your application, but I've found it a huge benefit for finding the root cause of regression errors.
gareth_bowles
2010-04-06 14:53:30
We evaluated number of solutions, splunk is tool search for low level searching but we were looking for a more holistic approach that we will give Testing labs oriented solution.
bobyg
2010-05-05 13:42:53