views:

3434

answers:

10

* UPDATE * I've reinstalled with Snow Leopard, clean install. Completely wiped my existing Leopard install. Same problem persists.

I've tried numbers of versions of symbolicatecrash to resolve symbols in my crash reports. From the version provided by Apple, to Alan's Quatermain's version posted on GitHub and finally from http://openradar.appspot.com/6438643.

For whatever reason, the best results I can get is for symbols on my own libraries to get resolved. Normally, this is enough data to point me in the right direction -- other times it is not. With 2.x I had no problems getting the symbols for my code + Apple provided libraries from within the stack traces in each thread.

Most likely an issue with my environment here, I'm not at all doubting the work that Apple or Alan have done. Yes I'm certain the dSYM I have stashed away is the same exact one that's generating the crash report.

Although 'Foo' is me, and getting symbols from it is wonderful, I need to see symbols from the other functions in the stack to truly understand my reports.

Note: For devices that crash running the app on iPhone OS 2.2.1, I have no problem getting all symbols. This is an iPhone OS 3.0 issue it appears.

Also, while running symbolicatecrash in verbose mode here's a few of the things that struck me as wrong:

- NO MATCH
NOT searching in Spotlight for dsym with UUID of /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
## Warning: Can't find any unstripped binary that matches version of /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation

..........fetching symbol file for libobjc.A.dylib--[undef] 
Searching [/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A345)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A347)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.1 (5B108)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.2 (5C1)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/3.0 (7A341)/Symbols/usr/lib/libobjc.A.dylib]...--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A345)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A347)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.1 (5B108)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.2 (5C1)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/3.0 (7A341)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
NOT searching in Spotlight for dsym with UUID of /usr/lib/libobjc.A.dylib
## Warning: Can't find any unstripped binary that matches version of /usr/lib/libobjc.A.dylib

Here's an example of the crash report after running it through symbolicatecrash:

Thread 0 Crashed:
0   libSystem.B.dylib                   0x31dc476c 0x31d46000 + 517996
1   libSystem.B.dylib                   0x31dc4755 0x31d46000 + 517973
2   Foo                            0x00053075 uncaught_exception_handler + 21
3   CoreFoundation                      0x3028f65f 0x301fd000 + 599647
4   libobjc.A.dylib                     0x30013693 0x3000c000 + 30355
5   libstdc++.6.dylib                   0x374ccc2d 0x3748a000 + 273453
6   libstdc++.6.dylib                   0x374ccc81 0x3748a000 + 273537
7   libstdc++.6.dylib                   0x374ccd4d 0x3748a000 + 273741
8   libobjc.A.dylib                     0x300135ff 0x3000c000 + 30207
9   CoreFoundation                      0x30222f2d 0x301fd000 + 155437
10  CoreFoundation                      0x30222ecb 0x301fd000 + 155339
11  Foundation                          0x30521e33 0x30501000 + 134707
12  Foundation                          0x30570d47 0x30501000 + 458055
13  Foo                            0x0000a1db -[Bar barfoo] (Bar.m:1617)
14  Foo                            0x00032f73 -[MyViewController foobar] (MyViewController.m:727)
15  Foo                            0x000329b9 -[MyViewController foobar] (MyViewController.m:666)
16  Foo                            0x00031fab -[MyViewController tabBar:tabSelected:] (MyViewController.m:440)
17  Foo                            0x00068d41 -[TTTabBar setSelectedTabIndex:] (TTTabBar.m:160)
18  Foo                            0x00068ca3 -[TTTabBar setSelectedTabView:] (TTTabBar.m:142)
19  Foo                            0x000689cf -[TTTabBar tabTouchedUp:] (TTTabBar.m:83)
20  CoreFoundation                      0x302552f9 0x301fd000 + 361209
21  UIKit                               0x3094d101 0x308ed000 + 393473
22  UIKit                               0x3094d0a1 0x308ed000 + 393377
23  UIKit                               0x3094d073 0x308ed000 + 393331
24  UIKit                               0x3094cdcd 0x308ed000 + 392653
25  UIKit                               0x309779c1 0x308ed000 + 567745
26  UIKit                               0x30977011 0x308ed000 + 565265
27  UIKit                               0x309767d9 0x308ed000 + 563161
28  UIKit                               0x30923613 0x308ed000 + 222739
29  UIKit                               0x30923163 0x308ed000 + 221539
30  GraphicsServices                    0x32045a4d 0x32041000 + 19021
31  CoreFoundation                      0x30253041 0x301fd000 + 352321
32  CoreFoundation                      0x30252771 0x301fd000 + 350065
33  GraphicsServices                    0x32044b0f 0x32041000 + 15119
34  GraphicsServices                    0x32044bbb 0x32041000 + 15291
35  UIKit                               0x308f0363 0x308ed000 + 13155
36  UIKit                               0x308ef121 0x308ed000 + 8481
37  Foo                            0x00002097 main (main.m:13)
A: 

I know that for the 3.0 SDK they moved symbolicatecrash into a new location and I had to get this new version in order to get the symbol mappings to work again.

For 3.0 it is found at: /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources

So it might be worth making sure you have the most recent version. I am seeing proper symbol mapping for both my source and API functions.

paulthenerd
Unfortunately I've already copied that version of symbolicatecrash and am using an absolute path to make sure that's what I'm hitting. Still, no symbols for Apple libraries. Am able to get my own though..
A: 

Hi Steve, did you find any solution to this?

I'm facing the same problem now...

Alan's Quatermain suggested a tips (not the fix that everyone's talking about):

http://alanquatermain.net/post/144411701/if-youre-having-trouble-with-symbolicatecrash-not

but stil no luck to me....

Joseph Lin
A: 

Edit symbolicatecrash. Go to about line 506, and change this:

    ARM      =>  "armv6",

to this:

    ARM      =>  "armv7",
leftspin
+1  A: 

I experienced the exact same problem and was able to fix it by altering the symbolicatecrash script. The problem for me was that both 'otool', 'atos' and 'size' didn't behave well with modules compiled for armv7. There are versions of these tools that work well with such modules which can be found at /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin. For otool, the symbolicatecrash script found at /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources already utilizes a mechanism to find the relevant version:

# Find otool from the latest iphoneos
my $otool = `xcrun -sdk iphoneos -find otool`;
chomp($otool);
my $atos = `xcrun -sdk iphoneos -find atos`;
chomp($atos);

However, 'size' is used directly:

if (-e '/usr/bin/size') {
    open my($ph),"-|",'size','-m','-l','-x',$symbol or die $!;

Altering this piece of code to the ugly:

if (-e '/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/size') {
    open my($ph),"-|",'/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/size','-m','-l','-x',$symbol or die $!;

seems to make it work.

yonilevy
+1  A: 

I was experiencing exactly the same issue - symbols for system modules were being displayed but not from my application. Turns out it was because my .dSym files could not be found by spotlight as I had "hidden" all the build products by specifying in the preferences that all build files go in the ".xcode-build" directory. Changing this to "xcode-build" fixed the problem!

Gabriel
+1  A: 

one thing to check. if you know the dsym file that you want to symbolicate against, then right click on the dsym file and select "Get Info". In the more info section is the UUID for the dsym file. In order for symbolication to work, the UUID for your application (enclosed in angle brackets in the binary images section of the crash log), needs to match the UUID of the DSYM file.

I found a problem in my usage of PLCrashReporter where the UUID in the crash log was not matching the UUID of the generated dsym file, resulting in symbolication failing.

Hassan Abdel-Rahman
thanks for the great tip, this helps tracing down the (many) issues with symbolicatecrash...
William Denniss
+4  A: 

If you are experiencing the situation where, when you run symbolicate crash only your own libraries are symbolicated, but the system libraries are not symbolicated and you have 3.0+ iPhone SDK installed most likely the issue is that the report format that you are symbolicating is an older format. One of the header fields in the crash report is "Report Version", this is the version of your crash report. The most current version (at the time of this writing iPhone OS 3.2.1) is "104". However, symbolicatecrash handles version 102 and 103 as well. Version 104 introduced new a new chip architecture (armv7) and the binary images have the ability to specify the chip architecture that is used. Version 102 and 103 of the crash report do not specify the chip architecture to use. The most current version of symbolicatecrash assume to use armv6 for version 102 and 103 of the report. This is probably wrong.

Symbolicate crash needs to be written to be more compatible with the older report formats. In general the UUID's for the binary images are really all that you need to correctly match up the system library symbols for the correct OS version as well as architecture--the architecture is probably redundant. However, in order for this to occur correctly, the symbolicate crash script needs to be modified, such that in addition to iterating through all the system library versions installed on the host machine to find the correct library, it also needs to iterate through all the chip architectures as well. This, fortunately is a pretty easy change to make.

To handle this I made the following changes:

in the getSymbolPathFor subroutine:

if ( defined($temp_path) && matchesUUID($temp_path, $uuid, $arch) ) {
                        $out_path = $temp_path;
                        @out_path_arr = {};
                    } else {
                        undef $temp_path;
                        print STDERR "-- NO MATCH\n"  if $opt{v};
                    }

change to:

if ( defined($temp_path) && matchesUUID($temp_path, $uuid, $arch) ) {
                        $out_path = $temp_path;
                        @out_path_arr = {};
                    } else {
                        if ( defined($temp_path) && matchesUUID($temp_path, $uuid, 'armv7') ) {
                            $out_path = $temp_path;
                            @out_path_arr = {};
                        } else {
                            undef $temp_path;
                            print STDERR "-- NO MATCH\n"  if $opt{v};
                        }
                    }

This will test out different chip architectures's UUID on the host system to find a match with the UUID in the binary image section in the crash report (and if the upcoming 2010 iPhone has a different chip architecture that should be added to this if-else condition).

For the next change, the UUID searching piece of code is brittle. If otool does not return something that symbolicatecrash can understand, symbolicatecrash will exit. In the case where you test the UUID of an architecture that is not recognized by the host machine, otool will return nothing, and symbolicatecrash dies. You need to make symbolicate crash a little more lenient by just ignore the otool result so that it can move on to the next architecture you are checking.

Change the matchesUUID subroutine:

if ( $test eq $uuid ) {
        ## See that it isn't stripped.  Even fully stripped apps have one symbol, so ensure that there is more than one.
        my ($nlocalsym) = $TEST_uuid =~ /nlocalsym\s+([0-9A-Fa-f]+)/;
        my ($nextdefsym) = $TEST_uuid =~ /nextdefsym\s+([0-9A-Fa-f]+)/;
        my $totalsym = $nextdefsym + $nlocalsym;
        print STDERR "\nNumber of symbols in $path: $nextdefsym + $nlocalsym = $totalsym\n" if $opt{v};
        return 1 if ( $totalsym > 1 );

        print STDERR "## $path appears to be stripped, skipping.\n" if $opt{v};
    } else {
                    print STDERR "Given UUID $uuid for '$path' is really UUID $test\n" if $opt{v};
            }
} else {
            die "Can't understand the output from otool ($TEST_uuid)";
    }

To something more lenient like this:

if ( $test eq $uuid ) {
        ## See that it isn't stripped.  Even fully stripped apps have one symbol, so ensure that there is more than one.
        my ($nlocalsym) = $TEST_uuid =~ /nlocalsym\s+([0-9A-Fa-f]+)/;
        my ($nextdefsym) = $TEST_uuid =~ /nextdefsym\s+([0-9A-Fa-f]+)/;
        my $totalsym = $nextdefsym + $nlocalsym;
        print STDERR "\nNumber of symbols in $path: $nextdefsym + $nlocalsym = $totalsym\n" if $opt{v};
        return 1 if ( $totalsym > 1 );

        print STDERR "## $path appears to be stripped, skipping.\n" if $opt{v};
    } else {
                    print STDERR "Given UUID $uuid for '$path' is really UUID $test\n" if $opt{v};
            }
} else {
            #die "Can't understand the output from otool ($TEST_uuid)";
            print "Can't understand the output from otool ($TEST_uuid)";
    }

After I made these changes I was able to see the system objects get symbolicated correctly (as well as my own code) when I use crash report versions 102 and 103 with the current iPhone SDK (3.1.2).

Hassan Abdel-Rahman
A: 

Hi Hasan,

do you still know how you resolved the problem with PLCrashReporter. I also seem to get wrong UUID from PLCrashReporter so that no images is able to be matched.

Greetings,

Mathias

Mathias Gich
A: 
Greg
+3  A: 

Greg's suggestions, didn't work for me but helped to point the right direction to get plcrashreporter logs (reporter version 103) to symbolize again with sdk 4.0.

BTW: this is an issue you will definitly have with reports created on newer devices (armv7) because symbolicate assumes reports version 103 are armv6.

So this is what i did ...

1)

make a copy from the newest symbolicatecrash script (sdk 4.0) for the changes you make. Note: depending on the editor u use you might have to set the executable bit after editing. chmod 755 symbolicatecrash


2) issue: finding symbols -- NO MATCH

this is because the default is set to armv6.

Resolution: in sub parse_images set the default architecture my $default_arch = 'armv7';


3) issue: "Use of uninitialized value $bundle ..."

Resolution: (hack: removing some restrictions that don't seem to work on 103 reports)

replace:

$app = $bundlename if (!defined $app && defined $image{plus} && length $image{plus});

by:

$app = $bundlename if (!defined $app);

4) issue: "can't understand output from otool"

Resolution: Greg's fix for sub matchesUUID.

replace:

} else { die "Can't understand the output from otool ($TEST_uuid)"; }

by:

} else { if ($arch eq "armv7") { return matchesUUID($path, $uuid, "armv6"); } else { die "Can't understand the output from otool ($TEST_uuid)"; } }

5) issue: "atos can not find symbols"

here Gregs script changes didn't work,

  • it seems that <$ph> in rindex( got lost in the code highlighting.
  • the "atos cannot load symbols" wasn't beeing passed into the script - so the check wouldn't work
  • the <$ph> in rindex( check reads the first line ... so one frame gets lost in case it was successfull. As a "hack" i simply faked a frame with adreass 1... which is "used" during the atos check.

So this is what i changed in sub symbolize_frames:

replace:

my $cmd = "$atos -arch $arch -o '$escapedSymbol' @{[ keys %$frames ]} | ";

by:

my $cmd = "$atos -arch $arch -o '$escapedSymbol' 1 @{[ keys %$frames ]}  2>&1 | ";

replace:

open my($ph),$cmd or die $!;

by:

open my($ph),$cmd or die $!;
if (rindex(<$ph>, "atos cannot load symbols") != -1 && $arch eq "armv7") {
    my $arch = "armv6"; my $cmd = "$atos -arch $arch -o '$escapedSymbol' @{[ keys %$frames ]}  2>&1 | ";
    print STDERR "Running $cmd\n" if $opt{v};
    open $ph,$cmd or die $!;
}

Note: with this changes the script runs fine for me - the "fixes" are probably not very elegant but i'm not fluent in perl :) ...

Philip
These changes work. You sir, are a scholar and a genius. Thank you.Whilst trying to understand your changes (I don't speak Perl either) I realised this is about fat binaries which contain both arm7 and arm6 code.
Roger Nolan