C++ multithreading: explicit locks in domain model classes

Question

Guys, I'm developing a multiplayer game application with C++ and currently in the process of choosing an appropriate multithreading architecture for it.

The core of the application is the endless loop which essentially updates each frame all entities of the game World. Currently this World loop is singlethreaded. It's working just fine but I'd really like to make it more scalable on multicores.

Since all World entities exist in Locations and updated in each frame as follows:

- World::update(dt) //dt is delta time since the last frame
  - Location::update(dt)
    - WorldEntity::update(dt)
    - WorldEntity::update(dt)
    - ...
  - Location::update(dt)
    - WorldEntity::update(dt)

...I was thinking about running each Location(and its updating logic) in a separate thread. This means I need to synchronize properly the World entities. And this is what I really don't want to to do since, I believe, explicit locking in domain classes methods is wrong and it makes the development, maintaining and debugging much-much more difficult.

At first I was thinking about isolating Location entities from entities in different Locations by forbidding any calls between them. What are possible ways to achieve this? Store entities of each Location in a thread local storage so that they are not accessible from outside? Or maybe instead of a thread per Location use processes instead?(but that's going to complicate everything a lot).

However even if Location entities are nicely isolated there another problem - persistence. I already have some sort of a simple generic persistence service which is running in a separate thread. It can be used in async mode, it accepts an object to be saved and returns a special future object which can be used to track the persistence process. I would love to use this service, however since it's running in a separate thread I again need to properly synchronize access to domain classes. In this case the possible option could be to implement proper cloning of domain objects so that persistence service would accept a copy of the object to be saved and no explicit locking would be needed...

Hence the question, is all said above worth it? Or maybe I should simply add explicit synchronizing logic into all domain classes and be done with it? Or maybe there is some better option I'm not aware of?

Thanks in advance

Update added world structure scheme thanks to Jed Smith

Answer 1

Well, when I was making MMO game servers, I used Staged, a Highly-concurrent Programming Model. You may see different Highly-concurrent Programming Models before.

Below is a part of a Staged Model:

                 ...
          +-------+-------+         
          | Process Msg & |          
          |  Send AckReq  |           
          +---------------+         
          |App.MsgStage() |         
          +-------+-------+           
                  | Pop()                  
 ^              +-V-+                     
 | Events       | Q |    Msg Stage |      
 | Go Up        | 0 |   Logic-Half |        
-+------------- |   | -------------+-- ... 
 | Requests     |   |    I/O-Half  |             
 | Move Down    +-^-+              |               
 V                | Push()                            
   +--------------+-------------+                 
   |   Push OnRecv Event,       |          
   | 1 Event per message        |     
   |                            |   
   |  Epoll I/O thread for      |   
   |multi-messaging connections |  
   +------^-------^--------^----+   
          |       |        |                          
Incoming msg1    msg2     msg3

As shown in above figure, it is a Network/Messaging Stage include Logic-half and I/O-half. The 2 halfs make communication using lock-free queues and/or lock-free ring buffers such as Event Queue and Request Queue, so no lock/mutex would be needed.

While making a complete MMO, other stages should be involved too besides Messaging at server side, for example Database Stage for loading/store players, AI/Timer Stage for refresh monsters or resources, and Logger Stage for logging, and so on.

In general I/O-half or bottom-half of a stage is responsible for I/O or other time-consuming tasks such as submiting "select" query to database, writing data to a disk file, or sending a message out of a network interface etc, which may be blocked; while Logic-half is pure logic computing, no any I/O operations and will never be blocked.

Since Logic-half could be executed by CPU very quickly, all Stage() (logic-half) such as MsgStage() or TimerStage() can be executed in just one StagedModel.Stage() call sequentially, say the Main Thread. And each bottom-half could have one or 2 threads, say Bottom-half Threads. For example as we had tested, on a Linux 2.6 machine, just one EPOLL thread should be enough for multi-liseners plus thousands of messaging clients. If Win/MSVC you will use Completion Port instead of EPOLL.

This way for a complete heavyweight MMO game server you only have just a few threads in total, and they are optimized for multi-core computer architecture because each core will run two or three threads per a 2-core processor, or one or two threads per a 4-core processor. Again, you can use lock-free queues and/or lock-free ring buffers, and you'll know in the Staged Model that most queues or ring buffers have just single producer and single consumer.

So upon your concerns, the world could be associated with a Stage (e.g. Scene Stage, AI or Timer or whatever), and make it in one seperate Bottom-half Thread, note only one thread for all of your locations and should be enough. Thus you need not to trigger updating of all locations at the same time, though you still could do it if you like. In the bottom-half, e.g. SceneStageThread, Update Event (with LocationID+WorldEntityID) will be generated when need to update, and your logic-half, e.g. SceneStage(), the OnUpdate(&UpdateEvent) will handle it to update the Location and WorldEntity when the MainThread is calling SceneStage(). If you like, the SceneStageThread could generate other events such as MonsterRelive Event etc.

See the document EffoNetMsg.pdf at http://code.google.com/p/effonetmsg/downloads/list or EffoAddons.pdf at http://code.google.com/p/effoaddon/downloads/list to learn more about Highly-concurrent Programming Models (including a complete Staged Model) and network messaging; see EffoDesign_LockFree.pdf at http://code.google.com/p/effocore/downloads/list to learn more about lock-free facilities such as lock-free queue and lock-free ring buffer.

Effo EDIT@2009nov09, adding Staged references C++ interface and implementation code URL:

Interface:
http://code.google.com/p/effoaddon/source/browse/trunk/devel/effo/codebase/addons/staged/include/staged_i.h
Implementation:
http://code.google.com/p/effoaddon/source/browse/trunk/devel/effo/codebase/addons/staged/src/staged.cpp