Problem using os.system() with sed command

Question

Hi,

I'm writing a small method to replace some text in a file. The only argument I need is the new text, as it is always the same file and text to be replaced.

I'm having a problem using the os.system() call, when I try to use the argument of the method

If I use a string like below, everything runs ok:

stringId = "GRRRRRRRRR"
cmd="sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
os.system(cmd)

Now, if i try to give a string as a parameter like below, the command is not executed. I do a print to see if the command is correct, and it is. I can even execute it with success if I copy / paste to my shell

import os
def updateExportConfigId(id):
stringId = "%s" % id
cmd= "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
print "command is " + cmd
os.system(cmd)

Does anyone knows what is wrong?

Thanks

Answer 1

To help you debug it, try adding:

print repr(cmd)

It might be that some special characters slipped into the command that normal print is hiding when you copy and paste it.

Answer 2

Maybe some indentation problem?

The following works correctly:

import os

def updateExportConfigId(id):
    stringId = "%s" % id
    cmd= "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' test.dat > test.new"
    print "command is " + cmd
    os.system(cmd)


updateExportConfigId("adsf")

Also do not use reserved words (id) as variables.

Answer 3

What is wrong is that there is some difference. Yeah, I know that's not helpful, but you need to figure out the difference.

Try running this:

import os
def updateExportConfigId(id):
    stringId = "%s" % id
    cmd1 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"
    stringId = "GRRRRRRRRR"
    cmd2 = "sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=" + stringId + "/g' path/file.old > path/file.new"

    print "cmd1:" , cmd1
    print "cmd2:" , cmd2
    print cmd1 == cmd2

updateExportConfigId("GRRRRRRRRR")

The code should print:

sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
sed '1,$s/MANAGER_ID=[0-9]*/MANAGER_ID=GRRRRRRRRR/g' path/file.old > path/file.new
True

Thereby showing that they are exactly the same. If the last line is "False" then they are not the same, and you should be able to see the difference.

Answer 4

Maybe it helps to use only raw strings.

Answer 5

So from previous answers we now know that id is a Unicode string, which makes cmd1 a Unicode string, which os.system() is converting to a byte string for execution in the default encoding.

a) I suggest using subprocess rather than os.system()

b) I suggest not using the name of a built-in function as a variable (id).

c) I suggest explicitly encoding the string to a byte string before executing:

if isinstance(cmd,unicode):
    cmd = cmd.encode("UTF-8")

d) For Lennart Regebro's suggestion add:

assert type(cmd1) == type(cmd2)

after

print cmd1 == cmd2

Answer 6

Finally, I found a way to run the os.system(cmd)!

Simple trick, to "clean" the cmd string:

os.system(str(cmd))

Now, I'm able to build the cmd with all arguments I need and at the end I just "clean" it with str() call before run it with os.system() call.

Thanks a lot for your answers!

swon

Answer 7

Obligatory: don't use os.system - use the subprocess module:

import subprocess

def updateExportConfigId(m_id, source='path/file.old', 
                             destination='path/file.new'):
    if isinstance(m_id, unicode):
        m_id = m_id.encode('utf-8')
    cmd= [
          "sed",
          ",$s/MANAGER_ID=[0-9]*/MANAGER_ID=%s/g" % m_id,  
          source,
         ]
    subprocess.call(cmd, stdout=open(destination, 'w'))

with this code you can pass the manager id, it can have spaces, quote chars, etc. The file names can also be passed to the function, and can also contain spaces and some other special chars. That's because your shell is not unnecessarly invoked, so one less process is started on your OS, and you don't have to worry on escaping special shell characters.

Another option: Don't launch sed. Use python's re module.

import re
def updateExportConfigID(m_id, source, destination):
    if isinstance(m_id, unicode):
        m_id = m_id.encode('utf-8')
    for line in source:
        new_line = re.sub(r'MANAGER_ID=\d*', 
                          r'MANAGER_ID=' + re.escape(m_id), 
                          line)
        destination.write(new_line)

and call it like this:

updateExportConfigID('GRRRR', open('path/file.old'), open('path/file.new', 'w'))

No new processes needed.