Named pipe is using 100% CPU

Question

I'm starting the script with ./file.py < pipe >> logfile and the script is:

while True:
    try:
            I = raw_input().strip().split()
    except EOFError:
            continue
    doSomething()

How could I better handle named pipe? This script always run at 100% CPU and it need to be real-time so I cannot use time.sleep.

Answer 1

At EOF you will loop forever getting yet another EOF. No more input will be done after EOF.

EOF does not mean a "gap" in the data. It means the named socket was disconnected and can no longer be used.

If you want "real-time" data, you have to read individual bytes from the socket until you get a complete "message". Perhaps a message ends with a '\n'. You can't use raw_input.

You have to use sys.stdin.read(1) to get bytes.

Pipes, BTW, are buffered. So you won't get real-time anything. If you want "real-time" you have to use UDP sockets, not TCP pipes.

Answer 2

"Real-time" (since it's obviously "soft" real-time given you have multiple processes going, not "hard" real-time!) does not mean "you cannot use time.sleep": even a tiny amount of sleep will make things a little bit better -- try adding a time.sleep(0.01) in your loop, just to give other processes a better chance to run. The lack of sleep may actually be making you take a longer time by giving other processes very little chance to fill the pipe!

Beyond this, @S.Lott has it just right: for "real-timeoid" behavior, you have to read from sys.stdin (though it probably need not be a byte at a time, depending on the platform: typically sys.stdin.read(1024) will read up to 1024 bytes, when sys.stdin is a pipe or other "raw" as opposed to "cooked" FD, returning however many bytes are in the pipe, if <100, rather than waiting -- you can set the FD to non-blocking to help assure that) directly, and perform string manipulation (e.g. to put lines together, strip them, etc) later in your code.