What's the best way to transfer data from python to another application in windows?

Question

I'm developing an application with a team in .Net (C++) and provide a COM interface to interact with python and other languages.

What we've found is that pushing data through COM turns out to be pretty slow.

I've considered several alternatives:

• dumping data to a file and sending the file path through com
• Shared Memory via mmap?
• Stream data through a socket directly?

From your experience what's the best way to pass around data?

Answer 1

XML/JSON and a either a Web Service or directly through a socket. It is also language and platform independent so if you decide you want to host the python portion on UNIX you can, or if you want to suddenly use Java or PHP or pretty much any other language you can.

As a general rule proprietary protocols/architectures like COM offer more restrictions than they do benefits. This is why the open specifications appeared in the first place.

HTH

Answer 2

It shouldn't be too complicated to set up a test for each of your alternatives and do a benchmark. Noting beats context sensitive empirical data... :)

Oh, and if you do this I'm sure a lot of people would be interested in the results.

Answer 3

Staying within the Windows interprocess communication mechanisms, we had positive experience using windows named pipes. Using Windows overlapped IO and the win32pipe module from pywin32.

You can learn much about win32 and python in the Python Programming On Win32 book.

The sending part simply writes to r'\\.\pipe\mypipe'.

A listener (ovpipe) object holds an event handle, and waiting for a message with possible other events involves calling win32event.WaitForMultipleObjects.

rc = win32event.WaitForMultipleObjects(
    eventlist,    # Objects to wait for.
    0,            # Wait for one object
    timeout)      # timeout in milli-seconds.

Here is part of the python overlapped listener class:

import win32event
import pywintypes
import win32file
import win32pipe

class ovpipe:
"Overlapped I/O named pipe class"
def __init__(self):
    self.over=pywintypes.OVERLAPPED()
    evt=win32event.CreateEvent(None,1,0,None)
    self.over.hEvent=evt
    self.pname='mypipe'
    self.hpipe = win32pipe.CreateNamedPipe(
        r'\\.\pipe\mypipe',             # pipe name 
        win32pipe.PIPE_ACCESS_DUPLEX|   # read/write access
        win32file.FILE_FLAG_OVERLAPPED,
        win32pipe.PIPE_TYPE_MESSAGE|    # message-type pipe 
        win32pipe.PIPE_WAIT,            # blocking mode 
        1,                              # number of instances 
        512,                            # output buffer size 
        512,                            # input buffer size 
        2000,                           # client time-out
        None)                           # no security attributes
    self.buffer = win32file.AllocateReadBuffer(512)
    self.state='noconnected'
    self.chstate()

def execmsg(self):
    "Translate the received message"
    pass

def chstate(self):
    "Change the state of the pipe depending on current state"
    if self.state=='noconnected':
        win32pipe.ConnectNamedPipe(self.hpipe,self.over)
        self.state='connectwait'
        return -6

    elif self.state=='connectwait':
        j,self.strbuf=win32file.ReadFile(self.hpipe,self.buffer,self.over)
        self.state='readwait'
        return -6

    elif self.state=='readwait':
        size=win32file.GetOverlappedResult(self.hpipe,self.over,1)
        self.msg=self.strbuf[:size]
        ret=self.execmsg()
        self.state = 'noconnected'
        win32pipe.DisconnectNamedPipe(self.hpipe)
        return ret

Answer 4

+1 on the named pipes but I would also like to add that from your comments it seems that your application is very chatty. Every time you make a remote call no matter how fast the underlying transport is you have a fixed cost of marshaling the data and making a connection. You can save a huge amount of overhead if you change the addpoint(lat, long) method to a addpoints(point_array) method. The idea is similar to why we have database connection pools and http-keep-alive connections. The less actual calls you make the better. Your existing COM solution may even be good enough if you can just limit the number of calls you make over it.