Considering getting into reverse engineering/disassembly

Question

Assuming a decent understanding of assembly on common CPU architectures (eg: x86), how can one explore a potential path (career, fun and profit, etc) into the field of reverse engineering? There is so little educational guides out there so it is difficult to understand what potential uses this has today (eg: is searching for buffer overflow exploits still common, or do stack monitoring programs make this obselete?). I am not looking for any step by step program, just some relevant information such as tips on how to efficiently find a specific area of a program. Basic things in the trade. As well as what it is currently being used for today.

So to recap, what current uses does reverse engineering yield today? And how can one find some basic information on how to learn the trade (again it doesn't have to be step-by-step, just anything which can through a clue would be helpful).

Answer 1

One of the biggest potential usage I see is project like samba (http://www.samba.org/)

There are always going to be closed proprietary systems like these and we always need engineers to reverse these so others can use it too.

Answer 2

The main one that I know of, as mentioned previously, is malware related. One of the primary tasks of researchers that work for scanner companies is to take a sample and debug it in a lab or virtual environment.

Along those same lines, there are plenty of security related areas that use reverse engineering/disassembly. Computer forensics is an area you might want to look into. A confiscated computer might have contain command and control programs (but no source) for various activities (command and control botnet, DoS attack programs, etc). It is usually much easier to circumvent protected data schemes by reverse engineering the program that protects rather than figuring out the password or key.

DRM/security protection in both hardware and software is a big reverse engineering area. Note this could be on either "side" of the issue (and law). Consider DVD copy programs, protection removal, the ability to play iTunes music on other devices, the ability to run homebrew programs on Wii, parallelizing a PS3 grid, unlocking an iPhone, etc, etc. Obviously there are many nonlegal only options too (reverse engineer a slot machine timer, ATM machine authentication, etc).

Legacy program conversion is a huge opportunity in many areas, especially government, finance, manufacturing, etc. There are mission critical programs that have run for 30 years on an ancient mainframe or mini that no one has source code to. Teams have to reverse engineer the program to convert it to something newer.

The other suggestions about learning Win32 tutorials are great. Also, sadly, some of the best published work is going to be on cracking (games). Search on that and there are a few tutorials out there which show the basics. A class I took used the book "The Art of Computer Virus Research and Defense" by Peter Szor but it was more heavy on the malware ideas and not the exact disassembly part.

Depending on which route you take you will need a background in other things but knowing assembly is going to be your most critical skill. Not just from an "I understand what that code does mostly" standpoint - you should be able to write stuff from scratch and understand exactly what a given bit of code does and understand other ways that same code could be written. Assembly (coding) involves figuring out one solution to a problem and coding it. Disassembly involves figuring out which of many many diverse solutions was used initially to solve the problem - MUCH harder ;)

Answer 3

The current maintainer of FFMpeg got his start reverse engineering video codecs so he could get videos to work on Linux. His blog is pretty interesting and he had a recent post talking about a book that he wish he had when he started.

Answer 4

There is a book Reversing: Secrets of Reverse Engineering written by Eldad Eilam;Elliot Chikofsky mybe helpfull four you