Matplotlib runs out of memory when plotting in a loop - EDITED

Question

I have a fairly simple plotting routine that looks like this:

from __future__ import division
import datetime
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import figure, plot, show, legend, close, savefig, rcParams
import numpy
from globalconstants import *

    def plotColumns(columnNumbers, t, out, showFig=False, filenamePrefix=None, saveFig=True, saveThumb=True):
        lineProps = ['b', 'r', 'g', 'c', 'm', 'y', 'k', 'b--', 'r--', 'g--', 'c--', 'm--', 'y--', 'k--', 'g--', 'b.-', 'r.-', 'g.-', 'c.-', 'm.-', 'y.-', 'k.-']

        rcParams['figure.figsize'] = (13,11)
        for i in columnNumbers:
            plot(t, out[:,i], lineProps[i])

        legendStrings = list(numpy.zeros(NUMCOMPONENTS)) 
        legendStrings[GLUCOSE] = 'GLUCOSE'
        legendStrings[CELLULOSE] = 'CELLULOSE'
        legendStrings[STARCH] = 'STARCH'
        legendStrings[ACETATE] = 'ACETATE'
        legendStrings[BUTYRATE] = 'BUTYRATE'
        legendStrings[SUCCINATE] = 'SUCCINATE'
        legendStrings[HYDROGEN] = 'HYDROGEN'
        legendStrings[PROPIONATE] = 'PROPIONATE'
        legendStrings[METHANE] = "METHANE"

        legendStrings[RUMINOCOCCUS] = 'RUMINOCOCCUS'
        legendStrings[METHANOBACTERIUM] = "METHANOBACTERIUM"
        legendStrings[BACTEROIDES] = 'BACTEROIDES'
        legendStrings[SELENOMONAS] = 'SELENOMONAS'
        legendStrings[CLOSTRIDIUM] = 'CLOSTRIDIUM'

        legendStrings = [legendStrings[i] for i in columnNumbers]
        legend(legendStrings, loc='best')

        dt = datetime.datetime.now()
        dtAsString = dt.strftime('%d-%m-%Y_%H-%M-%S')

        if filenamePrefix is None:
            filenamePrefix = ''

        if filenamePrefix != '' and filenamePrefix[-1] != '_':
            filenamePrefix += '_'

        if saveFig: 
            savefig(filenamePrefix+dtAsString+'.eps')

        if saveThumb:
            savefig(filenamePrefix+dtAsString+'.png', dpi=300)


        if showFig: f.show()

        close('all')

When I plot this in single iterations, it works fine. However, the moment I put it in a loop, matplotlib throws a hissy fit...

Traceback (most recent call last):
  File "c4hm_param_variation_h2_conc.py", line 148, in <module>
    plotColumns(columnNumbers, timeVector, out, showFig=False, filenamePrefix='c
4hm_param_variation_h2_conc_'+str(hydrogen_conc), saveFig=False, saveThumb=True)

  File "D:\phdproject\alexander paper\python\v3\plotcolumns.py", line 48, in plo
tColumns
    savefig(filenamePrefix+dtAsString+'.png', dpi=300)
  File "C:\Python25\lib\site-packages\matplotlib\pyplot.py", line 356, in savefi
g
    return fig.savefig(*args, **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 1032, in savef
ig
    self.canvas.print_figure(*args, **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\backend_bases.py", line 1476, i
n print_figure
    **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
358, in print_png
    FigureCanvasAgg.draw(self)
  File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
314, in draw
    self.figure.draw(self.renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 773, in draw
    for a in self.axes: a.draw(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\axes.py", line 1735, in draw
    a.draw(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 374, in draw
    bbox = self._legend_box.get_window_extent(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 209, in get
_window_extent
    px, py = self.get_offset(w, h, xd, yd)
  File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 162, in get
_offset
    return self._offset(width, height, xdescent, ydescent)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 360, in findof
fset
    return _findoffset(width, height, xdescent, ydescent, renderer)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 325, in _findo
ffset_best
    ox, oy = self._find_best_position(width, height, renderer)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 817, in _find_
best_position
    verts, bboxes, lines = self._auto_legend_data()
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 669, in _auto_
legend_data
    tpath = trans.transform_path(path)
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1911, in t
ransform_path
    self._a.transform_path(path))
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1122, in t
ransform_path
    return Path(self.transform(path.vertices), path.codes,
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1402, in t
ransform
    return affine_transform(points, mtx)
MemoryError: Could not allocate memory for path

This happens on iteration 2 (counting from 1), if that makes a difference. The code is running on Windows XP 32-bit with python 2.5 and matplotlib 0.99.1, numpy 1.3.0 and scipy 0.7.1.

EDIT: The code has now been updated to reflect the fact that the crash actually occurs at the call to legend(). Commenting that call out solves the problem, though obviously, I would still like to be able to put a legend on my graphs...

Answer 1

Although this is not a direct answer, I see that you intend to save the figure as a PNG using savefig. You can actually save a figure without displaying it first. That may avoid memory problems. For example, consider this code:

f = pylab.figure()
ax = f.add_axes([0.1,0.1,0.8,0.8])
for i in columnNumbers:
    ax.plot(t, out[:,i], lineProps[i])
f.savefig(dtAsString+'.png', dpi=300)

I think this should work.

Answer 2

Is each loop supposed to generate a new figure? I don't see you closing it or creating a new figure instance from loop to loop.

This call will clear the current figure after you save it at the end of the loop:

pyplot.clf()

I'd refactor, though, and make your code more OO and create a new figure instance on each loop:

from matplotlib import pyplot

while True:
  fig = pyplot.figure()
  ax = fig.add_subplot(111)
  ax.plot(x,y)
  ax.legend(legendStrings, loc = 'best')
  fig.savefig('himom.png')
  # etc....

Answer 3

I have run into the same problem, but the suggested fixes do not work for me.

In the process of writing some scientific code I have a number of cases which I plot within a loop. As the loop produces more and more plots (on the order of 200 plots, about 10MB) I can watch the memory usage grow in top. I have tried clf(), I don't bother displaying the figure, but the above suggestions don't work--the memory usage steadily grows until it crashes, giving me an error message.

I'm using matplotlib version 0.99.1.2-3 with python version 2.6.5 (standard, up to date Ubuntu 10.04 installation, 2GB RAM). The crash comes when I have roughly 8MB's worth of figures completed, so it would appear that Agg is storing the uncompressed images without discarding them.

Answer 4

We have run into the exact same problem: Plotting a sequence of seismic signals with matplotlib within wxPython. We can see memory growing monotonously as more signals are plotted.

Deleting handles or calling clf() only frees some of the memory. Calling pyplot.close('all') helps, but doesn't solve the problem either.

Is there a way to force collecting the plot objects?

Many thanks Jono and Ole, Australia-Indonesia Facility for Disaster Reduction.