Is Project Darkstar Realistic?

Question

Project Darkstar was the topic of the monthly JavaSIG meeting down at the Google offices in NYC last night. For those that don't know (probably everyone), Project Darkstar is a framework for massively multiplayer online games that attempts to take care of all of the "hard stuff." The basic idea is that you write your game server logic in such a way that all operations are broken up into tiny tasks. You pass these tasks to the Project Darkstar framework which handles distributing them to a specific node in the cluster, any concurrency issues, and finally persisting the data.

Apparently doing this kind of thing is a much different problem for video games than it is for enterprise applications. Jim Waldo, who gave the lecture, claims that MMO games have a DB read/write ratio of 50/50, whereas enterprise apps are more like 90% read, 10% write. He also claims that most existing MMOs keep everything in memory exlcusively, and only dump to a DB every 6 hours of so. This means if a server goes down, you would lose all of the work since the last DB dump.

Now, the project itself sounds really cool, but I don't think the industry will accept it. First, you have to write your server code in Java. The client code can be written in anything (Jim claims ActionScript 3 is the most popular, follow by C++), but the server stuff has to be Java. Sounds good to me, but I really get the impression that everyone in the games industry hates Java.

Second, unlike other industries where developers prefer to use existing frameworks and libraries, the guys in the games industry seem to like to write everything themselves. Not only that, they like to rewrite everything for every new game they produce. Things are starting to change where developers are using Havok for physics, Unreal Engine 3 as their platform, etc., but for the most part it looks like everything is still proprietary.

So, are the guys at Project Darkstar just wasting their time? Can a general framework like this really work for complex games with the performance that is required? Even if it does work, are game companies willing to use it?

Answer 1

I don't work in the games industry, but it sounds to me like this will do the same thing for video games as the Quake and Half-Life engines did. That is they will promote getting young developers interested in the industry and promote development of indie games.

From what I can tell, video game companies do not reuse most of their code, because if they do it implies that their new game is just a rehash of an old one. Everyone wants a cool new physics engine, better graphics, new ways to play the game. Most video game engines and frameworks are made for a specific scenario and thus are not very bendable to other situations.

Maybe Darkstar will get it right though, but I kinda doubt it, since generalizing only works for so much.

Answer 2

From what I can tell, video game companies do not reuse most of their code, because if they do it implies that their new game is just a rehash of an old one.

Um... if you're referring to the long tail of video game companies, maybe. Within a company that has had a series of successful games, there is usually some modicum of reuse. Major hardware changes can result in ditching a lot of work, but it really depends on the company.

MSN

Answer 3

Sounds like useless tech to me. The MMO world is controlled by a few big game companies that already have their own tech in place. Indie game developers love trying to build MMO's and sometimes they do, but those games rarely gain traction. Larger companies breaking into the MMO world would probably license "proven" technology, or extend their own.

Game companies reuse vast quantities of code from game to game. Most/many game companies have developed their own tech internally, and use it on every game they produce. Occasionally, they will do something like replace their physics code with a 3rd party physics engine. If their internal code base (game engine, design tools, internal pipeline) starts to age too much, or become unwieldy, they might switch to one of the big game engines like Unreal. Even then, major chunks of code will continue to be re-used from game to game.

Answer 4

It sounds like fun to design and code, but I think it ultimately comes down to useless abstractions (to steal from Joel).

Answer 5

It's very common for games to reuse "game engines," even those from third-parties. This sounds like another step in that direction.

Answer 6

Edit: This was written before Oracle bought Sun and started a rampage to kill everything that does not make them a billion $ per day. See the comments for an OSS Fork. I still stand by my opinion that stuff like that (MMO Middleware) is realistic, you just need a company that doesn't suck behind it.

The Market may be dominated by few large games, but that does not mean that there is not a lot of room for more niche games. Lets face it: If you want to reach 100.000+ players, you're ending up building your own technology stack, at least for the critical core. That's what CCP did for EVE Online (StacklessIO), that's what Blizzard did for World of Warcraft (although they do use many third-party libraries), that's what Mythic did for Warhammer Online (although they are based on Gamebryo).

However, if you aim to be a small, niche MMO (like the dozens of Free-to-Play/Itemshop MMOs), then getting the Network stuff right is just insanely hard, data consistency is even harder and scalability is the biggest b*tch.

But game technology is not your only problem - you also need to tackle Billing. Credit Card only? Have fun selling in Germany then, people there want ELV. That's where you need a reliable billing provider, but you still need to wire in the billing application with your accounts to make sure that accounts are blocked/reactivated when the billing fails.

There are some companies already offering "MMO Infratructure Services" (i.e. Arvato's EEIS), but the bottom line is: Stuff like Project Darkstar IS realistic, but assuming that you can build a Multi-Billion-MMO entirely on a Third Party Stack is optimistic, possibly idealistic.

But then again, entirely inventing all of the technology is even more stupid - use the Third Party stuff that you need (i.e. Billing, Font Rendering, Audio Output...), but write the stuff that really makes or breaks your business (i.e. Network stack, User interface etc.) on your own. (Note: Jeff's posting may be a bit flawed, but the overall direction is correct IMHO.)

Addendum: Also, the game industry does license and reuse engines a lot. The most prominent game Engines are the Unreal Engine, Source Engine and id Tech, which fuel dozens, if not hundreds of games. But there are some lesser-known (outside of the industry) engines. There is Gamebryo, the Middleware behind games like Civilization 4 and Fallout 3, there was RenderWare that is now only EA-in-House, but used in games like Battlefield 2 or The Sims 3. There is the open source Ogre3d, which was used in some commercial titles. If you're just looking for Sound, there's stuff like FMOD or if you want to do font-rendering, why not give FreeType a spin?

What I'm saying is: Third-Party Engines/Middleware do exist, and they ARE being successfully used since more than a decade (I know for sure that id's Wolfenstein Engine was licensed to other companies, and that was 1992), even by big companies in multi-million-dollar titles. The important thing is the support, because a good engine with no help in case of an issue is pretty much worthless or at least very expensive if the developer has to spend their game-development-time with unneccessary debugging of the Engine.

If the Darkstar folks manage to get the support side right and 2 or 3 higher profile titles out, I do believe it could succeed in opening the MMO market to a lot more smaller developers and indies.

Answer 7

I think it's a great thing to do. Developers not having to worry about all these things that project darkstar takes care of, and it's very easy to use. But it's not all about just getting it to work and not having to learn everything about internet-communication, It's also about performance. Project darkstar has been under development for over 2 years and it keeps getting better,faster and more robust.

I think it will be hard and probably not worth the time to write these things when aiming at a specific game, when technologies like this can be used instead. And you also get nice information during runtime telling you where in an application there's a cause of slowdown or deadlocks so you can improve that.

Answer 8

i dont really think what you guys are saying here is right, indeed we need new game engines for client side but server side... come on! the network layers didn't change since they were invented the requirements of a mmo from 10 years ago are pretty much the same as those from today games, but you must consider that the hardware is lot better and cheaper now so indie games really have a chance.

There was someone stating that one need 1mil $ to start a mmo that is pretty stupid since there are many types of mmo from wow and eve to travian and tribal-wars, you need resources to invest in cool latest gen graphics but a game is not just about graphics there are many successful text based games with thousands of users at any point in time also there are the board games that gather huge amounts of users, mmo means massive multiplayer online this can be anything from dungeons games to poker or chess games, or even social platforms. All of them share pretty much same requirements, latency management, connections management and parallel programming and i think darkstar if is supported well can increase the quality of games through competition, lot of small companies try to build games some of them probably with good ideas but they are scared when they hit the server infrastructure wall