Howto debug kernel oops on embedded system

Question

I've got a problem with one of the serial ports on an embedded development. /dev/ttyS0, /dev/ttyS2, and /dev/ttyS3 all work fine with no problems. But in some cases accessing /dev/ttyS1 throws the following two 'oops' messages.

I've no idea where to even start tracking down the cause of this, can you help?

1st

Unable to handle kernel NULL pointer dereference at virtual address 0000013c
pgd = c0004000
[0000013c] *pgd=00000000
stopped custom tracer.
Internal error: Oops: 17 [#1] PREEMPT
Modules linked in: macb
CPU: 0    Not tainted  (2.6.24-rc5-rt1 #2)
pc : [<c01a9e60>]    lr : [<c01a9e90>]    psr: 60000093
sp : c3c25f10  ip : 0000012c  fp : c3c25f1c
r10: 00000000  r9 : 00000000  r8 : 00000000
r7 : 000000ac  r6 : 00000000  r5 : 0000012c  r4 : 00000000
r3 : 60000093  r2 : 60000013  r1 : c3c1a340  r0 : 0000012c
Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 0005317f  Table: 23dcc000  DAC: 00000017
Process softirq-high/0 (pid: 4, stack limit = 0xc3c24258)
Stack: (0xc3c25f10 to 0xc3c26000)
5f00:                                     c3c25f2c c3c25f20 c01a9e90 c01a9e20
5f20: c3c25f44 c3c25f30 c0101b84 c01a9e90 c020ab48 c020abf4 c3c25f74 c3c25f48
5f40: c0119960 c0101b78 c3c1a340 c020ac2c 000f4240 00000000 00000004 00000000
5f60: c0205b78 c02059c8 c3c25f98 c3c25f78 c003cf5c c0119560 00000001 fffffffe
5f80: c02059d4 c3c24000 00000001 c3c25fa8 c3c25f9c c003d038 c003cee4 c3c25fd8
5fa0: c3c25fac c003d9ec c003d010 00000032 c3c24000 c02059c8 c003d8a0 00000000
5fc0: 00000000 00000000 00000000 c3c25ff4 c3c25fdc c004cc94 c003d8b0 00000000
5fe0: 00000000 00000000 00000000 c3c25ff8 c003b13c c004cc4c 00000000 00000000
Backtrace:
Function entered at [<c01a9e10>] from [<c01a9e90>]
Function entered at [<c01a9e80>] from [<c0101b84>]
Function entered at [<c0101b68>] from [<c0119960>]
 r5:c020abf4 r4:c020ab48
Function entered at [<c0119550>] from [<c003cf5c>]
Function entered at [<c003ced4>] from [<c003d038>]
 r8:00000001 r7:c3c24000 r6:c02059d4 r5:fffffffe r4:00000001
Function entered at [<c003d000>] from [<c003d9ec>]
Function entered at [<c003d8a0>] from [<c004cc94>]
Function entered at [<c004cc3c>] from [<c003b13c>]
 r6:00000000 r5:00000000 r4:00000000
Code: e592100c e10f2000 e3823080 e121f003 (e59c3010)

2nd

Unable to handle kernel NULL pointer dereference at virtual address 000000bc
pgd = c0004000
[000000bc] *pgd=00000000
Internal error: Oops: 17 [#2] PREEMPT
Modules linked in: macb
CPU: 0    Tainted: G      D  (2.6.24-rc5-rt1 #2)
pc : [<c0101868>]    lr : [<c01161c0>]    psr: 60000013
sp : c3c33f50  ip : c3c33f68  fp : c3c33f64
r10: c0205ab8  r9 : c0205b78  r8 : 00000000
r7 : 00000004  r6 : 00000000  r5 : 000f4240  r4 : c3e3c378
r3 : c3e3c360  r2 : 60000013  r1 : a0000013  r0 : 00000000
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
Control: 0005317f  Table: 23dcc000  DAC: 00000017
Process softirq-tasklet (pid: 9, stack limit = 0xc3c32258)
Stack: (0xc3c33f50 to 0xc3c34000)
3f40:                                     c3e3c378 000f4240 c3c33f74 c3c33f68
3f60: c01161c0 c010186c c3c33f98 c3c33f78 c003cf5c c01161b8 00000020 ffffffdf
3f80: c0205ac4 c3c32000 00000020 c3c33fa8 c3c33f9c c003d078 c003cee4 c3c33fd8
3fa0: c3c33fac c003d9ec c003d050 00000032 c3c32000 c0205ab8 c003d8a0 00000000
3fc0: 00000000 00000000 00000000 c3c33ff4 c3c33fdc c004cc94 c003d8b0 00000000
3fe0: 00000000 00000000 00000000 c3c33ff8 c003b13c c004cc4c ffffffff ffffffff
Backtrace:
Function entered at [<c010185c>] from [<c01161c0>]
 r5:000f4240 r4:c3e3c378
Function entered at [<c01161a8>] from [<c003cf5c>]
Function entered at [<c003ced4>] from [<c003d078>]
 r8:00000020 r7:c3c32000 r6:c0205ac4 r5:ffffffdf r4:00000020
Function entered at [<c003d040>] from [<c003d9ec>]
Function entered at [<c003d8a0>] from [<c004cc94>]
Function entered at [<c004cc3c>] from [<c003b13c>]
 r6:00000000 r5:00000000 r4:00000000
Code: c01f63a4 e1a0c00d e92dd830 e24cb004 (e59030bc)

Answer 1

The best thing you can do is to take a look at linux-src/Documentation/oops-tracing.txt It explains how you can start debugging this.

The first things you can try is decoding the backtrace. The system.map file should contain the addresses of all functions in your kernel. You should be able to produce a readable backtrace this way.

Answer 2

Have a look at http://lwn.net/Kernel/LDD3/ and download chapter 4 scroll down to page 94.

The easiest way would be to compile your kernel with debug info (CONFIG_DEBUG_INFO) and execute the kernel from gdb.

For an older kernel (2.4) you will need ksymoops.

Answer 3

Make sure that ksymoops is enabled Or look backtrace values in the System.map Or get a JTAG :)

Answer 4

You should take a look at the system.map that is generated with your kernel. Thanks to this file you'll be enable to decode your oops backtrace easily.

Answer 5

Hey Simon, did you find an answer for the actual problem? I have a very similar problem - also OOps #17, also doing some communication over a serial device (via usb). The problem occurs in random processes but always in the proc_flush_call function:

[   58.460000] Unable to handle kernel NULL pointer dereference at virtual address 00000020
[   58.470000] pgd = c0004000
[   58.470000] [00000020] *pgd=00000000
[   58.470000] Internal error: Oops: 17 [#1] PREEMPT
[   58.470000] Modules linked in: ppp_async ppp_generic slhc crc_ccitt snd_pcm_oss snd_mixer_oss chumby_timer snd_soc_stmp3780_devb snd_soc_stmp3xxx_dai snd_soc_stmp378x_codec snd_soc_stmp3t
[   58.470000] CPU: 0    Not tainted  (2.6.28-mno #1)
[   58.470000] PC is at proc_flush_task+0x50/0x2b4
[   58.470000] LR is at release_task+0x44/0x360
[   58.470000] pc : [<c00e13cc>]    lr : [<c0043860>]    psr: a0000013
[   58.470000] sp : c1421ef0  ip : c1421f50  fp : c1421f4c
[   58.470000] r10: c38565f8  r9 : 00000000  r8 : c3ceca00
[   58.470000] r7 : c3ceca00  r6 : c38566e0  r5 : c3856600  r4 : c1420000
[   58.470000] r3 : 00000000  r2 : c3ceca00  r1 : 60000013  r0 : c3856600
[   58.470000] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   58.470000] Control: 0005317f  Table: 41428000  DAC: 00000015
[   58.470000] Process httpd (pid: 2021, stack limit = 0xc1420268)
[   58.470000] Stack: (0xc1421ef0 to 0xc1422000)
[   58.470000] 1ee0:                                     0000000c c3ceca00 c1421f14 c3ceca00
[   58.470000] 1f00: c1421f2c c1421f10 c00b8fb8 c01ec5e0 c3efd6b0 c3856600 00126008 00000001
[   58.470000] 1f20: c1421f44 ffffffff c3856600 c38566e0 c1420000 c38566e0 c38565f8 c38565f8
[   58.470000] 1f40: c1421f74 c1421f50 c0043860 c00e138c ffffffff c3856600 c38566e0 c381a000
[   58.470000] 1f60: c38566e0 c38565f8 c1421f94 c1421f78 c00451d8 c004382c 00000001 c38566f4
[   58.470000] 1f80: c1421f80 c1421f80 c1421fa4 c1421f98 c00453bc c0044aa0 00000000 c1421fa8
[   58.470000] 1fa0: c0027e40 c00453b4 00000000 001260d8 00000000 00000001 00001000 00121250
[   58.470000] 1fc0: 00000000 001260d8 00126008 00000001 00000000 00000003 00000000 befe9a5c
[   58.470000] 1fe0: 00000000 befe98d8 0002031c 000c69e0 80000010 00000000 00000000 00000000
[   58.470000] Backtrace:
[   58.470000] [<c00e137c>] (proc_flush_task+0x0/0x2b4) from [<c0043860>] (release_task+0x44/0x360)
[   58.470000] [<c004381c>] (release_task+0x0/0x360) from [<c00451d8>] (do_exit+0x748/0x800)
[   58.470000]  r9:c38565f8 r8:c38566e0 r7:c381a000 r6:c38566e0 r5:c3856600
[   58.470000] r4:ffffffff
[   58.470000] [<c0044a90>] (do_exit+0x0/0x800) from [<c00453bc>] (sys_exit+0x18/0x1c)
[   58.470000] [<c00453a4>] (sys_exit+0x0/0x1c) from [<c0027e40>] (ret_fast_syscall+0x0/0x2c)
[   58.470000] Code: ea00008d e51b2058 e5973020 e3520000 (e593a020)
[   58.750000] ---[ end trace cfe7dadcb5653fd9 ]---
[   58.760000] Fixing recursive fault but reboot is needed!