So really I have 3 questions, but they are all pretty straightforward (for once):

  1. Can an add-on in Firefox see if there are multiple instances of Firefox/the firefox process running?

  2. Since the only way (in Windows, at least) to have multiple instances of Firefox running is by having them run under different profiles, is it possible for an add-on to determine which profiles the other instances are running under?

  3. Could an add-on interact with the other process? Specifically, could it do things like close the other process or tell it to open a new window with a URL specified within the add-on's process?

+1  A: 
  1. Indirectly: Firefox doesn't allow to have multiple instances running on the same profile. Since your plugin is part of a profile, "your" Firefox (the one running the plugin) is unique.

  2. Probably not. Profile information is security related and Firefox doesn't hand that out easily. That's one reason why the profile names are created with random prefixes.

  3. Firefox listens to remote commands (that's how a new process tells an existing FF to open a new URL), so opening more windows is possible. I don't know whether there is a remote command to terminate FF.

Note: If you start closing other windows and annoy people by loading ads, they will hate you. It takes only a few seconds to disable and uninstall an unwanted plugin. And if you try anything to prevent users from uninstalling your plugin, they will really hate you. Pitchforks or tar and feathers might be involved. ;)

Aaron Digulla
Ha! I was afraid that #3 might sound like I'm trying to make a nag-on. Not at all, actually. What I want is an add-on that will extend the privacy mode so that certain pages are always loaded in a "privacy-profile" process. But I don't want to open a new instance of the process every time.
Cool idea. But I think that this is beyond of the scope that people here can answer. You should join the FF developer list and repeat your question there. I think your plugin is a really cool addition but you'll probably need a couple of changes in the FF code to make it work.
Aaron Digulla
Wow, thanks! I will most definitely give them a holler. It's sad to think anything is outside of the scope of SO.
There is just so much you can achieve with one page of text. :)
Aaron Digulla

An add-on (or a "Mozilla app") can do anything, since it can include binary components. If you're asking for existing APIs usable from JS, I don't think there are any right now.

If you're open to installing an add-on into all profiles, your add-on in instance #1 could communicate with your add-on in instance #2 using any IPC mechanism, such as sockets (which are usable from JS-only add-on).