Why don't you e-mail [email protected] and ask them? The Free Software Foundation (who are the stewards of the GPL) would be the best authority on this, since they wrote the license. You can also try contacting the SFLC, but I'd try the FSF first.
Some things to include in your e-mail:
- Is your engine going to be licensed with a license that is compatible with the GPL? If the answer to this is no, don't bother contacting them. If yes, indicate which license.
- Is the license for the library/API to interface with your engine going to be GPL compatible? Will you require static linking, or other things that result in a combined work?
Try to be as descriptive as possible without writing a book about everything that your program will do.
Again, if you wish to make your program non-free, you'll be better off talking to a lawyer. If your core engine uses an OSI approved license, the FSF will likely help you to understand any possible hassles / entanglements.
If plug-ins can be distributed independently of your core engine and the library used to interact with your core engine is GPL compatible, you'll probably be OK. However, IANAL, they are, try contacting them.
EDIT:
Make sure you remain the SOLE copyright holder to the engine. This allows you to use a dual licensing scheme. Projects such as the Cherokee web server have employed this with great success, not to mention MySQL AB. A quick Google search on copyright sign-off should turn up all you need to know.
Your business model appears smart, asking questions this early makes it seem smarter. If your product is good, you'll do well :)
There are some fundamentalists that argue about dual licensing, most of them have rather high paying jobs and aren't supported by the code that they produce. Using an OSI approved license for your core helps your neighbor as well as your wallet. In this day and age, that's as good as its going to get.