How do I control memory usage with Omni Thread Library's thread pool?

Question

I've been using gabr's OmniThreadLibrary to build a ThreadPool.

I'm a novice when it comes to multi-threading, which is why I've found OTL to be so interesting.

Following the demo application "app_11_ThreadPool.exe", I setup a thread pool in my application. The purpose of my app is to take a list of URL's, and attempt to connect to them and download any file that is automatically returned (I'm testing potential malware sites for payloads so I can block them).

Typically I'll have a list of 500 - 1000 sites. I have a TMemo that holds the URL's on my main form. Here is my relevant code. Note that this code is executed from my "Begin" buttons click event.

  iNumTasks := memoSiteList.Lines.Count - 1;
  GlobalOmniThreadPool.MaxQueued := 16;
  GlobalOmniThreadPool.MonitorWith(OmniEventMonitor1);
  GlobalOmniThreadPool.MaxExecuting := 16;
  GlobalOmniThreadPool.MaxQueued := 0;

    for iTask := 1 to iNumTasks + 1 do
    begin

      if iTask mod 4 = 0 then
        Application.ProcessMessages;

      CreateTask(
        TSiteQuery.Create(
        url,
        full_url,
        sProxyServer,
        sProxyPort,
        sSaveLocation,
        bVerboseHeaders)
        ).MonitorWith(OmniEventMonitor1).Schedule;
     end;

And here is TSiteQuery:

type
  TSiteQuery = class(TOmniWorker)
  strict private
    { Private declarations }
    FTaskID: int64;
    furl: string;
    f_full_url: string;
    fsProxyServer: string;
    fsProxyPort: string;
    fbVerboseHeaders: boolean;
    fsSaveLocation: string;
  private // [..]

Right now, everything works great. Running 1000 URL's increases my application's memory usage from 10mb, to ~130mb. That's not a big deal and I understand that.

But the problem is that every single time I click "Begin", the app's memory usage increases by ~130mb. Maybe I just don't know what I'm doing, but I was under the impression that using a thread pool would mean I would not have to create new threads every run.

My hope is that the previously created threads in the pool would be reused on subsequent executions. I was expecting that my app's memory usage would stay around ~130 mb whether I click "begin" 1 time or 10 subsequent times.

Any suggestions?

Regards

Answer 1

Is this a threading issue or a memory leak issue? Try using the debugger's Thread Status window to determine whether or not your number of allocated threads is actually increasing. If not, then you're probably creating objects inside the threads and not freeing them.

Answer 2

Looks more like an memory leak issue than a threading issue...