Python, redirecting the stream of Popen to a python function

Question

Dear all, I'm new to python programming. I have this problem: I have a list of text files (both compressed and not) and I need to : - connect to the server and open them - after the opening of the file, I need to take his content and pass it to another python function that I wrote

def readLogs (fileName):
f = open (fileName, 'r')
inStream = f.read()
counter = 0
inStream = re.split('\n', inStream) # Create a 'list of lines'
out = ""              # Will contain the output
logInConst = ""       # log In Construction
curLine = ""          # Line that I am working on

for nextLine in inStream:
    logInConst += curLine  
    curLine = nextLine
    #   check if it is a start of a new log && check if the previous log is 'ready'
    if newLogRegExp.match(curLine) and logInConst != "":

        counter = counter + 1

        out = logInConst
        logInConst = ""
        yield out

yield logInConst + curLine

def checkFile (regExp, fileName):
    generatore = readLogs(fileName)
    listOfMatches=[]

    for i in generatore: #I'm now cycling through the logs
        # regExp must be a COMPILE regular expression
        if regExp.search(i):
            listOfMatches.append(i)
    return listOfMatches

in order to elaborate the info contained in those files. The function has the aim of write in just 1 line the logs that are stored in those files using 3 lines ... The function is working fine on files read from my local machine but I cannot figure out how to connect to a remote server and create these one-line logs without storing the content of each file into a string and then working with the string ... The command that I use to connect to the remote machine is :

connection_out = Popen(['ssh', retList[0], 'cd '+retList[2]+'; cat'+fileName], stdout=PIPE).communicate()[0]

retList[0] and retList[2] are the user@remote and the folder name that I have to access

Thanks to all in advance !

Answer 1

You do not have to split the stream/file into lines yourself. Just iterate:

for ln in f:
    # work on line in ln

This should work equally well for files (using open() for file()) and pipes (using Popen). Use the stdout property of the popen object to access the pipe connected to stdout of the subprocess

Example

from subprocess import Popen, PIPE
pp = Popen('dir', shell=True, stdout=PIPE)

for ln in pp.stdout:
    print '#',ln

Answer 2

Remove InStream and just use the file object.

So that your code would read:

for nextLine in f.readlines():
    .
    .
    .

Ber has it right.

To clarify, the default iteration behavior of a file object is to return the next line. so "for nextLine in f" will give you the same results as "for nextLine in f.readlines()".

See the file object documentation for details: http://docs.python.org/library/stdtypes.html#bltin-file-objects

Answer 3

My problem is that I have to establish an ssh connection first :

pr1=Popen(['ssh', 'siatc@lgssp101', '*~/PRD/sialt/log_archive/00/MSG_090308_162648.gz*' ], stdout=PIPE).communicate()[0]

All the files that I need to open are stored in a list, fileList[], part of them are compressed (.gz) and part are just text files !! I have tried all the procedures that u showed before bot nothing worked ... I think that I mus modify the third argument of the Popen function but I cannot figure out how to do it ! Is there anyone that can help me ???

Answer 4

If you want to do something via ssh, why not use the Python SSH module?