Getting started reverse-engineering OS X?

Question

Hi,

What is a good place to learn reverse engineering, specifically as it applies to Mac OS X? Two apps that I admire in terms of this subject:

Hyperspaces – http://thecocoabots.com/hyperspaces/

and

Orbit – http://www.steventroughtonsmith.com/orbit/

Thanks guys.

Answer 1

You should grab a copy of Mac OS X Internals which is an awesome book about everything that Apple does not tell you. Not only is this great if you are interested in reverse engineering, it will also make you a better OS X programmer in general.

Answer 2

Apple releases a ton of the foundation of OS X as open source. See here.

In addition, F-Script Anywhere will help a ton with dissecting the Finder and/or any other closed source application.

Answer 3

Use class-dump-x/-z to get the private Objective-C headers for OS X/iPhone OS system frameworks. There are a lot of classes/methods hidden from the public (some rightly so)

Answer 4

For iPhoneOS specifically, class-dump-z is a great way to dump headers. The only problem, of course, is that you can't actually see what is going on inside of each method. IDA Pro and a few scripts make it possible to see the assembly instructions for these system frameworks. (example picture: http://grab.by/1Vn6).

The most handy IDC scripts are fixobjc2 and dyldinfo. You can find each of these linked from this blog post: http://networkpx.blogspot.com/2010/01/two-ida-pro-5x-scripts-for-iphoneos.html

But, what good is this information if you can't use it? iPhone developer saurik has written something called MobileSubstrate that enables hooking onto any method. http://svn.saurik.com/repos/menes/trunk/mobilesubstrate/

Answer 5

Others have already mentioned class-dump, which is an excellent tool for retrieving the class definitions from a compiled executable. On a related note, you should also take a look at otx, which is provides very nice (readable), disassembled output.

If you need a way to quickly test snippets of code, use F-Script (mentioned by others), Nu or MacRuby. Of these, I've mainly used Nu. It has the capability to define bridged functions on the fly, and can handle pointers, both of which are pretty handy if you need to call arbitrary C functions.

Since you mentioned being interesting in Spaces and other screen managers, you should also read A brief tutorial on reverse engineering OS X. It's an old article by Rich Wareham (author of the pre-Spaces multi-desktop app: 'Desktop Manager') on how he figured out the call syntax for few private CoreGraphics methods in order to do nice desktop transitions. The source code for Desktop Manager is also available, which might be useful to you.

Answer 6

This site shows how to patch an existing Objective C program: http://www.culater.net/wiki/moin.cgi/CocoaReverseEngineering

Namely posing:

[[B class] poseAsClass:[A class]];

and method swizzling:

 /**
 * Renames the selector for a given method.
 * Searches for a method with _oldSelector and reassigned _newSelector to that
 * implementation.
 * @return NO on an error and the methods were not swizzled
 */
BOOL DTRenameSelector(Class _class, SEL _oldSelector, SEL _newSelector)
{
        Method method = nil;

        // First, look for the methods
        method = class_getInstanceMethod(_class, _oldSelector);
        if (method == nil)
                return NO;

        method->method_name = _newSelector;
        return YES;
}

// *** Example ***


// never implemented, just here to silence a compiler warning
@interface WebInternalImage (PHWebInternalImageSwizzle)
- (void) _webkit_scheduleFrame;
@end

@implementation WebInternalImage (PHWebInternalImage)

+ (void) initialize
{
        DTRenameSelector([self class], @selector(scheduleFrame), @selector (_webkit_scheduleFrame));
        DTRenameSelector([self class], @selector(_ph_scheduleFrame), @selector(scheduleFrame));
}

- (void) _ph_scheduleFrame
{
        // do something crazy...
        ...
        // call the "super" method - this method doesn't exist until runtime
        [self _webkit_scheduleFrame];
}

@end

(code copied from http://www.culater.net/wiki/moin.cgi/CocoaReverseEngineering)

Answer 7

As an addition to the other answers, you are going to want to check out DYLD_INSERT_LIBRARIES to inject your code into a Cocoa program.

Answer 8

You should definitely consider using DTrace. There is an excellent BlackHat presentation on using DTrace for reverse engineering on OS X entitled, "DTRACE: The Reverse Engineer's Unexpected Swiss Army Knife".

You can get a copy and view the video presentation here.

There are also some excellent papers at www.uninformed.org on reverse engineering OS X.