views:

166

answers:

2

Hi

I am running a huge task [automated translation scripted with perl + database etc.] to run for about 2 weeks non-stop. While thinking how to speed it up I saw that the translator outputs everything (all translated sentences, all info on the way) to STDOUT all the time. This makes it work visibly slower when I get the output on the console.

I obviously piped the output to /dev/null, but then I thought "could there be something even faster?" It's so much output that it'd really make a difference.

And that's the question I'm asking You, because as far as I know there is nothing faster... (But I'm far from being a guru having used linux on a daily basis only last 3 years)

+5  A: 

Output to /dev/null is implemented in the kernel, which is pretty bloody fast. The output pipe isn't your problem now, it's the time it takes to build the strings that are getting sent to /dev/null. I would recommend you go through the program and comment out (or guard with if $be_verbose) all the lines that are useless print statements. I'm pretty sure that'll give you a noticeable speedup.

Benson
yeah, but the worst part is in C++ and I don't have the code.Still it's cool to know /dev/null is the best ;)
naugtur
Ahh, okay. You did say it was written in perl. ;-)Yes, /dev/null is as fast as an output file descriptor can possibly be without kernel hacking, and even then it'd be hard to beat.
Benson
I picked this answer for the additions in comment. I can't accept both answers. stackoverflow wouldn't let me;)
naugtur
+7  A: 

I'm able (via dd) to dump 20 gigabytes of data per second down /dev/null. This is not your bottleneck :-p

Pretty much the only way to make it faster is to not generate the data in the first place - remove the logging statements entirely. The cost of producing all the log messages likely exceeds the cost of throwing them away quite a bit.

Steven Schlansker
Thanx for the test. I didn't expect it to be a bottleneck, rather just wondering... it's 23:15 where I sit. Good time for such thoughts :P
naugtur