Python differences between running as script and running via interactive shell

Question

I am attempting to debug a problem with a ctypes wrapper of a windows DLL and have noticed differences when I run tests via an interactive shell (python or ipython) and when I run the scripts non-interactively.

I was wondering if there is any explanation for the differences I am seeing here?

Specifically, when I interactively run a simple test, a DLL call will hang and never return, where as exactly the same code run as a script will not exhibit this problem.

To be more explicit with what I mean here, imagine you had the following code

from foobar import bar, foo
bar(foo(1,2,3))

When put in a file, say "myfoo.py", and excecuted via "python myfoo.py", the above code executes as expected. However, if you type in the above into a python/ipython shell, the code behaves differently (in my case, hangs when calling a ctypes.WinDLL function)

Some additional details:

I am using the same interpreter and the same PYTHONPATH in both cases. The DLL being wrapped is the Canon EDSDKv2.9, a SDK to remotely control cameras. It is always hangs in the DLL, not in python code.

When initialised, my EDSDK wrapper launches a thread whose run method looks like this:

  def run(self):
    sys.coinit_flags = 0 #use multithreaded mode
    from pythoncom import PumpWaitingMessages
    #^^ done here so this thread is correctly initialised
    error(EDSDK.EdsInitializeSDK())
    self.EDSDK_initialised = True
    while self.active:
      PumpWaitingMessages()
      sleep(self.msg_sleep_time)
    error(EDSDK.EdsTerminateSDK())

This threads purpose is basically to initialise the SDK, pump messages, and allow other threads to call wrapped methods.

Note: this has worked, both interactively and non-interactively, in previous EDSDK versions. My current problem only happens in the latest version of EDSDK.

I suspect it may be something to do with threads (hence the snippet), but can't find any information online to back up my suspicion.

So, is anyone aware of any differences when running python interactively and non-interactively? Possibly related to windows threads? Any help, even wild guesses, would be appreciated at this point, because I am completely stumped! :)

Answer 1

The Python interactive interpreter is not thread safe. So, if you try to send a blocking command, the whole interpreter will hang.

See this article as to why this happens (tl; dr is that the IDLE and threads don't mix). As for how to fix this, use the console rather than the IDLE GUI. Or, you can just use a script.

Answer 2

This is something to keep in mind in the context of asynchronous or concurrent operations (e.g. multithreading, multiprocessing). For all intents and purposes when you're running Python interactively you do not have a main loop. Even though __name__ is set to __main__, anything assigned or executed there might not be available. This is especially the case with multiprocessing and in some cases multithreading, where the state of objects may not shared between processes/threads.

This behavior can be confusing at best and dangerous at worst. A good habit to get into when debugging is to always name your threads and pair that with the use of Python's logging module to track what is going on behind the scenes.