I'd like to have a tool in Perl to gather useful statistics for page loads (eg, download time/speed, CDN information, headers, dns lookups, compressions)
Does anyone know if one exists or if there's a place to learn about how to make one?
A:
You could have the perl CGI (or any perl program) run a few times under the profiler, and scan for commonalities. I haven't seen a web-based interface like this, but if you have control over the perl side of things, the documentation is here:
http://www.perl.com/pub/a/2004/06/25/profiling.html
It basically boils down to running your perl program with -d:DProf and then, after it finishes, running dprofpp in the same directory:
# perl -d:DProf ./foo.pl
# dprofpp
Update:
Yes, this is not the same thing as protocol profiling, as duly noted below, but there aren't may alternatives for perl. If you are trying to find where the perl portion of the slowness is coming from, profiling perl is a good place to start. Products like YSlow will track the pure protocol aspects of it, whether the CGI is perl or php or python.
Personally, I use it to profile my django site, which is in python and flash, and I profile those separately from the protocol portion of the system, which I also use YSlow for.
Also, there are perl plugins for "ddd" which will at least make it graphical:
http://www.gnu.org/software/ddd/
Sorry if this doesn't solve the exact problem, I would like to know if there's a perl interface to collate this as well, but I know this is where I would start looking...
eruciform
2010-07-09 22:13:45
Answer is barely useful: Runtime profiling is not the same as protocol profiling.
daxim
2010-07-10 08:23:01
@daxim: it's not the same, but it is something useful, and there isn't much else out there for perl that i know of. if you want to be critical, at least find something applicable to add.
eruciform
2010-07-10 16:12:49
ddd is a debugger; yet another tool that's totally not suitable for doing what YSlow does. What's next, a memory leak tracker?
daxim
2010-07-10 16:53:21
@daxim: you're welcome to make a suggested answer any time. i won't downvote yours.
eruciform
2010-07-10 18:02:53
While this isn't exactly what I was looking for, I appreciate the answer. I realize there aren't many solutions out there. Perhaps this will steer me in the right direction. +1 for attempt
vol7ron
2010-07-14 01:27:54
thank you! :-) good luck with the continued search. if you do find out from another source, I would love to hear about it.
eruciform
2010-07-14 04:09:03
+1
A:
You might want to try WWW::Mechanize::Timed, which extends the WWW::Mechanize module. The ::Timed features will allow you to collect information on how long your requests take. The underlying ::Mechanize module, which is itself a subclass of LWP::UserAgent, would give you access to your response, including headers, body content, and images. From these you could compute total page "weight", number of requests, etc. This doesn't cover everything YSlow does (exposing the DNS internals underlying gethostbyname would be a good trick!) but I hope it's a place to start, if I've understood your question properly.
Ryan M
2010-07-20 14:39:17
Does ::Timed also include loads from JavaScript operations? I thought Mechanize can't process JavaScript instructions, thus any post-load or on-load AJAX operations would be neglected.
vol7ron
2010-07-22 01:36:17
You're correct, it does not handle JavaScript. Mechanize would be getting pretty close to being a web browser if it did.Where you go from here depends on what you're trying to do. Are you trying to analyze server logs? Run automated benchmarks on your own sites? Do research on other sites?Rather than reinvent YSlow, maybe you just want to accumulate its data? You might try http://developer.yahoo.com/yslow/help/beacons.html#yslow_beacon for a method of extracting metrics from YSlow.
Ryan M
2010-07-28 18:45:52
Once you've got beaconing running, you could automate URL browsing using an extension such as iMacros or CronZilla. Firefox would hit the URLs you specify on the schedule you specify, and YSlow (in automatic mode) would evaluate each URL and beacon your data collection server. You could write a data collector in Perl with one of the many daemon packages, such as Net::Server, that would receive the beacons and process them.Without knowing more about your project, this is about all I can offer you. Good luck!
Ryan M
2010-07-28 18:49:48
I've accepted this answer, but still feel it's limited. It's mainly to do research on my site and other sites for comparison.
vol7ron
2010-09-06 14:06:08