Embedded IronPython Memory Leak

Question

I need some help finding a solution to a memory leak I'm having. I have a C# application (.NET v3.5) that allows a user to run IronPython scripts for testing purposes. The scripts may load different modules from the Python standard library (as included with IronPython binaries). However, when the script is completed, the memory allocated to the imported modules is not garbage collected. Looping through multiple runs of one script (done for stress testing) causes the system to run out of memory during long term use.

Here is a simplified version of what I'm doing.

Script class main function:

public void Run()
{
    // set up iron python runtime engine
    this.engine = Python.CreateEngine(pyOpts);
    this.runtime = this.engine.Runtime;
    this.scope = this.engine.CreateScope();

    // compile from file
    PythonCompilerOptions pco = (PythonCompilerOptions)this.engine.GetCompilerOptions();
    pco.Module &= ~ModuleOptions.Optimized;
    this.script = this.engine.CreateScriptSourceFromFile(this.path).Compile(pco);

    // run script
    this.script.Execute(this.scope);

    // shutdown runtime (run atexit functions that exist)
    this.runtime.Shutdown();
}

An example 'test.py' script that loads the random module (adds ~1500 KB of memory):

import random
print "Random number: %i" % random.randint(1,10)

A looping mechanism that will cause the system to run out of memory:

while(1)
{
    Script s = new Script("test.py");
    s.Run();
    s.Dispose();
}

I added the section to not optimize the compilation based on what I found in this thread, but the memory leak occurs either way. Adding the explicit call to s.Dispose() also makes no difference (as expected). I'm currently using IronPython 2.0, but I've also tried upgrading to IronPython 2.6 RC2 without any success.

How do I get the imported modules in the embedded IronPython script to be garbage collected like normal .NET objects when the scripting engine/runtime goes out of scope?

Answer 1

Have you tried running the Iron python in its own appDomain? Python.CreateEngine allows you to pass it an AppDomain, which can then be unloaded when the script completes.

Or, based on this discussion, use LightweightScopes option, like so

Dictionary<String, Object> options = new Dictionary<string, object>();
options["LightweightScopes"] = true;
ScriptEngine engine = Python.CreateEngine(options);
ScriptRuntime runtime = engine.Runtime;

Answer 2

using Iron Python 2.6 RC 2, and C# 3.5

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Scripting.Hosting;
using IronPython.Hosting;

namespace IPmemTest {
    class IPy {
     private string script = "import random; random.randint(1,10)";

     public IPy() {
     }

     public void run() {
      //set up script environment
      Dictionary<String, Object> options = new Dictionary<string, object>();
      options["LightweightScopes"] = true;
      ScriptEngine engine = Python.CreateEngine(options);
      ScriptRuntime runtime = engine.Runtime;
      ScriptScope scope = runtime.CreateScope();
      var source = engine.CreateScriptSourceFromString(this.script);
      var comped = source.Compile();
      comped.Execute(scope);
      runtime.Shutdown();
            }
    }
}

and my loop is

class Program {
     static void Main(string[] args) {
      while (true) {
       var ipy = new IPy();
       ipy.run();
      }
     }
    }

memory usage increases to about 70,000K, but then levels off.