Hi.
I'm looking for tips on books that goes through what is needed/recommended in a game engine for 3D. I don't need books about math, physics or OpenGL/XNA/DirectX, but only about the inner goings of a standard game engine. I also just want tips and tricks for the game engine, and not a full source game engine, since I really want to write it myselv, I just don't know exactly where to start.
I'm marking this as community wiki, since more people are probably looking for something like this.
You might find that studying the source for Id's Quake III Arena more beneficial than any book you're likely to find. The engine is open source under the GPL and available for download from Id's site.
Ogre3d is another OSS engine that might be worth exploring. It even has support for contemporary fanciness like SpeedTree.
You're venturing into some scary territory, just a warning - here be dragons. I'm not sure that any book alone is going to be sufficient to help you build an engine, although you might find one that leads you in the right direction.
3D Game Engine Design, Second Edition: A Practical Approach to Real-Time Computer Graphics is nice...be warned this endeavour is not for beginners.
If you're familiar with Foley/Van Damm then you'll be comfortable with
3D Game Engine Design by Eberly.
Also, there are several game engines, with source code, that you can choose from to get an idea how they work. I'll list just a few.
That should get you started.
For a comprehensive list go to the DevMaster 3D Game Engine Database.
3D Game Engine Programming is a nice book focusing on Direct X under C++ working through initial class design, 3D maths classes, shaders and culminating with a huge chapter dedicated to making a level editor for the first person game that has been built up throughout the book.
As other responders have posted this is no small task, but I consider the actual task of understanding this book (let alone writing the code) to be one of the major milestones in my programming development.
Eberly's book is excellent, and LaMothe is supposed to be well known in this area. It really depends on what area you're interested in. Do you need a physics engine? Do you need real time collision detection? Books on game physics and collision detection are quite interesting and the following three are quite readable...
Game Physics Engine Development http://www.amazon.com/Physics-Development-Kaufmann-Interactive-Technology/dp/012369471X/ref=pd_bbs_sr_3?ie=UTF8&s=books&qid=1241031284&sr=8-3
Real Time Collision Detection http://www.amazon.com/Real-Time-Collision-Detection-Interactive-Technology/dp/1558607323/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1241031400&sr=8-1
Cheers!
I have been looking for similar books for a long time. The problem here is that when it comes to literature, when it says "game engine" there is almost certainly "render engine" inside. My current thesis acutally covers this topic a bit, since I cannot understand how come there are no good books out there about the high-level game architecture.
The best thing to do is check out other engines, especially open source ones since you obviously can see the source code and understand what they did.
Studying the Quake III source is a bad idea. Although quite interesting, we are talking pure C here. Typedefs and structs, hand-written assembler and whatnot. You surely will learn a lot when going through it, but probably not what you asked for here. See an interesting blog post on id Tech 3 here.
One engine that mobilizes lots of modern patterns and rather clean OOP is Delta3d (delta3d.org). The website contains good documentation, especially on the architecture, entity framework and messaging infrastructure. I like to mention this engine since it seems to be rather unknown. Although it was devleoped for military simulations, the general concepts of the actual high-level game engine are probably applicable to your situation.
In my opinion it's probably best to learn and develop in 2D before jumping into 3D. I'm assuming you haven't here...
Working in 2D first gives you time to learn about the other components of a game engine without the pain of a complex render engine - sound, file management, input - without needing to know too much in the way of graphics. Once you have it all figured out, you've learnt a lot about developing an engine, and 3D graphics are just a logical step (rather than learning everything in one go).