How are data shared between fastCGI processes?

Question

I've written a simple perl script that I'm running via fastCGI on Apache. The application loads a set of XML data files which are used to lookup values based upon the the query parameters of an incoming request. As I understand it, if I want to increase the amount of concurrent requests my application can handle I need to allow fastCGI to spawn multiple processes. Will each of these processes have to hold duplicate copies of the XML data in memory? Is there a way to set things up so that I can have one copy of the XML data loaded in memory while increasing the capacity to handle concurrent requests?

Answer 1

Memory is shared between separate FastCGI processes just as it is between ordinary, separate processes, which is to say that, for our purposes, the data is not shared.

(FastCGI lets a single process handle multiple requests serially, avoiding the need to re-initialize, re-read config and XML data, for example, with each request after the first served by that process.)

On the upside, any technique that would work to reduce the in-memory footprint of your XML between separate processes should work here. You might read the files into shared memory (which can be tricky to synchronize and update), select a lower-memory XML parser, or access the information indirectly, say through a "compiled" GDBM of your XML data or a custom server you write to answer queries.

Answer 2

As pilcrow correctly answered, FastCGI provides no special way to share data between processes and lists traditional ways to reduce memory usage.

One additional possibility is to have a persistent, non-FastCGI process read the XML file and act as a data server for the FastCGI processes. The efficiency of this depends on how complicated the queries are and how much data needs to be passed in and out, but it would allow a single copy of the data to remain in memory.