Windows Server Crash Dump Analysis

I'm not certain that this is the right venue for this question, but a programmer friend of mine said I should try this here.

My company's main application is hosted on a terminal server running Windows Server 2008. Since last Thursday we have seen this server crash and reboot 3 times, and we just went live with this server on the previous Tuesday. I have used the the WinDbg program to analyze the crash dump file, but I'm a little outside by depth at this point and I'm hoping that someone out there can help me get this issue resolved.

The application that appears to me to be at fault is winoac.exe which is the executable for SmartWare 4.5 (www.smartware4.com). This is the platform that our application runs on. If this application is at fault, is there anything that I can do about it, other than complaining to SmartWare?

Thanks a million to anybody that can help.

Here are the results of the analysis.

Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP] Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible Product: Server, suite: TerminalServer Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612 Machine Name: Kernel base = 0x81c41000 PsLoadedModuleList = 0x81d4e930 Debug session time: Thu Jan 29 12:49:43.870 2009 (GMT-6) System Uptime: 0 days 11:18:08.929 Loading Kernel Symbols ............................................................... ................................................................ .............. Loading User Symbols PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details Loading unloaded module list .....

*
Bugcheck Analysis *
*

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}

Page bd1f2 not present in the dump file. Type ".hh dbgerr004" for details Page bc9c3 not present in the dump file. Type ".hh dbgerr004" for details PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details Probably caused by : RDPDD.dll ( RDPDD!OE2_TableEncodeOrderFields+11e )

Followup: MachineOwner

7: kd> !analyze -v

*
Bugcheck Analysis *
*

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e) This is a very common bugcheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Some common problems are exception code 0x80000003. This means a hard coded breakpoint or assertion was hit, but this system was booted /NODEBUG. This is not supposed to happen as developers should never have hardcoded breakpoints in retail code, but ... If this happens, make sure a debugger gets connected, and the system is booted /DEBUG. This will let us see why this breakpoint is happening. Arguments: Arg1: c0000005, The exception code that was not handled Arg2: 81c88043, The address that the exception occurred at Arg3: 9cef0840, Trap Frame Arg4: 00000000

Debugging Details:

PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: nt!RtlInitUnicodeString+1b 81c88043 f266af repne scas word ptr es:[edi]

TRAP_FRAME: 9cef0840 -- (.trap 0xffffffff9cef0840) ErrCode = 00000000 eax=00000000 ebx=fe414fd8 ecx=ffffffec edx=9cef0914 esi=fe40fcf0 edi=fe415000 eip=81c88043 esp=9cef08b4 ebp=9cef0924 iopl=0 nv up ei pl zr na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246 nt!RtlInitUnicodeString+0x1b: 81c88043 f266af repne scas word ptr es:[edi] Resetting default scope

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0x8E

PROCESS_NAME: WINOAC.EXE

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from 81c72fbe to 81cfc759

STACK_TEXT:
9cef0400 81c72fbe 0000008e c0000005 81c88043 nt!KeBugCheckEx+0x1e 9cef07d0 81c9953a 9cef07ec 00000000 9cef0840 nt!KiDispatchException+0x1a9 9cef0838 81c994ee 9cef0924 81c88043 badb0d00 nt!CommonDispatchException+0x4a 9cef085c 9976011a 99771680 997708e8 00000000 nt!Kei386EoiHelper+0x186 9cef0924 9959efab 5d0102bb 00000006 00000002 RDPDD!OE2_TableEncodeOrderFields+0x11e 9cef0a0c 995aeaf8 5d0102bb 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b 9cef0abc 9958455b 5d0102bb 0110007e 9cef0b04 win32k!xxxDrawState+0x1c9 9cef0b2c 995853e1 5d0102bb fe40fc78 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8 9cef0b98 9959f511 5d0102bb 00000000 fe414570 win32k!xxxMenuDraw+0x1f2 9cef0bf0 994ed1d6 00000017 5d0102bb 00000004 win32k!xxxMenuBarDraw+0x1bf 9cef0c38 9950c0f5 fe414570 5d0102bb 00000001 win32k!xxxDrawWindowFrame+0xf7 9cef0cb4 9950d73d fe414570 00000085 090402df win32k!xxxRealDefWindowProc+0x88b 9cef0ccc 994e673d fe414570 00000085 090402df win32k!xxxWrapRealDefWindowProc+0x2b 9cef0ce8 9950d6f4 fe414570 00000085 090402df win32k!NtUserfnNCDESTROY+0x27 9cef0d20 81c9897a 000200ba 00000085 090402df win32k!NtUserMessageCall+0xc6 9cef0d20 77089a94 000200ba 00000085 090402df nt!KiFastCallEntry+0x12a WARNING: Frame IP not in any known module. Following frames may be wrong. 0012d7cc 00000000 00000000 00000000 00000000 0x77089a94

STACK_COMMAND: kb

FOLLOWUP_IP: RDPDD!OE2_TableEncodeOrderFields+11e 9976011a 8b4518 mov eax,dword ptr [ebp+18h]

SYMBOL_STACK_INDEX: 4

SYMBOL_NAME: RDPDD!OE2_TableEncodeOrderFields+11e

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: RDPDD

IMAGE_NAME: RDPDD.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 4791923e

FAILURE_BUCKET_ID: 0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

BUCKET_ID: 0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

Followup: MachineOwner

Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP] Kernel Summary Dump File: Only kernel address space is available

*
Bugcheck Analysis *
*

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}

Followup: MachineOwner

7: kd> !analyze -v

*
Bugcheck Analysis *
*

Debugging Details:

PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 7ffd400c). Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: nt!RtlInitUnicodeString+1b 81c88043 f266af repne scas word ptr es:[edi]

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0x8E

PROCESS_NAME: WINOAC.EXE

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from 81c72fbe to 81cfc759

STACK_COMMAND: kb

FOLLOWUP_IP: RDPDD!OE2_TableEncodeOrderFields+11e 9976011a 8b4518 mov eax,dword ptr [ebp+18h]

SYMBOL_STACK_INDEX: 4

SYMBOL_NAME: RDPDD!OE2_TableEncodeOrderFields+11e

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: RDPDD

IMAGE_NAME: RDPDD.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 4791923e

FAILURE_BUCKET_ID: 0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

BUCKET_ID: 0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

Followup: MachineOwner

Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\2-3-09-2\MEMORY.DMP] Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols Executable search path is: Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible Product: Server, suite: TerminalServer Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612 Machine Name: Kernel base = 0x81c13000 PsLoadedModuleList = 0x81d20930 Debug session time: Tue Feb 3 14:20:03.117 2009 (GMT-6) System Uptime: 0 days 2:00:33.869 Loading Kernel Symbols ............................................................... ................................................................ ............. Loading User Symbols PEB is paged out (Peb.Ldr = 7ffdc00c). Type ".hh dbgerr001" for details Loading unloaded module list .....

*
Bugcheck Analysis *
*

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c5a043, d60a5840, 0}

Page bce51 not present in the dump file. Type ".hh dbgerr004" for details Page bce22 not present in the dump file. Type ".hh dbgerr004" for details Page bb16b not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details PEB is paged out (Peb.Ldr = 7ffdc00c). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 7ffdc00c). Type ".hh dbgerr001" for details Probably caused by : win32k.sys ( win32k!OffBitBlt+97 )

Followup: MachineOwner

0: kd> !analyze -v

*
Bugcheck Analysis *
*

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e) This is a very common bugcheck. Usually the exception address pinpoints the driver/function that caused the problem. Always note this address as well as the link date of the driver/image that contains this address. Some common problems are exception code 0x80000003. This means a hard coded breakpoint or assertion was hit, but this system was booted /NODEBUG. This is not supposed to happen as developers should never have hardcoded breakpoints in retail code, but ... If this happens, make sure a debugger gets connected, and the system is booted /DEBUG. This will let us see why this breakpoint is happening. Arguments: Arg1: c0000005, The exception code that was not handled Arg2: 81c5a043, The address that the exception occurred at Arg3: d60a5840, Trap Frame Arg4: 00000000

Debugging Details:

Page bb16b not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details Page bce5a not present in the dump file. Type ".hh dbgerr004" for details PEB is paged out (Peb.Ldr = 7ffdc00c). Type ".hh dbgerr001" for details PEB is paged out (Peb.Ldr = 7ffdc00c). Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: nt!RtlInitUnicodeString+1b 81c5a043 f266af repne scas word ptr es:[edi]

TRAP_FRAME: d60a5840 -- (.trap 0xffffffffd60a5840) ErrCode = 00000000 eax=00000000 ebx=fe41afd8 ecx=ffffffec edx=d60a5914 esi=fe40f5e0 edi=fe41b000 eip=81c5a043 esp=d60a58b4 ebp=d60a5924 iopl=0 nv up ei pl zr na pe nc cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010246 nt!RtlInitUnicodeString+0x1b: 81c5a043 f266af repne scas word ptr es:[edi] Resetting default scope

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0x8E

PROCESS_NAME: WINOAC.EXE

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from 81c44fbe to 81cce759

STACK_TEXT:
d60a5400 81c44fbe 0000008e c0000005 81c5a043 nt!KeBugCheckEx+0x1e d60a57d0 81c6b53a d60a57ec 00000000 d60a5840 nt!KiDispatchException+0x1a9 d60a5838 81c6b4ee d60a5924 81c5a043 badb0d00 nt!CommonDispatchException+0x4a d60a585c 999e2242 ff888010 00000000 00000000 nt!Kei386EoiHelper+0x186 d60a5924 999befab 1401009b 00000006 00000002 win32k!OffBitBlt+0x97 d60a5a0c 999ceaf8 1401009b 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b d60a5abc 999a455b 1401009b 0110007e d60a5b04 win32k!xxxDrawState+0x1c9 d60a5b2c 999a53e1 1401009b fe40d168 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8 d60a5b98 999bf511 1401009b 00000000 fe418398 win32k!xxxMenuDraw+0x1f2 d60a5bf0 9990d1d6 00000017 1401009b 00000004 win32k!xxxMenuBarDraw+0x1bf d60a5c38 9992c0f5 fe418398 1401009b 00000001 win32k!xxxDrawWindowFrame+0xf7 d60a5cb4 9992d73d fe418398 00000085 0904035f win32k!xxxRealDefWindowProc+0x88b d60a5ccc 9990673d fe418398 00000085 0904035f win32k!xxxWrapRealDefWindowProc+0x2b d60a5ce8 9992d6f4 fe418398 00000085 0904035f win32k!NtUserfnNCDESTROY+0x27 d60a5d20 81c6a97a 0003001c 00000085 0904035f win32k!NtUserMessageCall+0xc6 d60a5d20 77049a94 0003001c 00000085 0904035f nt!KiFastCallEntry+0x12a WARNING: Frame IP not in any known module. Following frames may be wrong. 0012d7cc 00000000 00000000 00000000 00000000 0x77049a94

STACK_COMMAND: kb

FOLLOWUP_IP: win32k!OffBitBlt+97 999e2242 8b4d20 mov ecx,dword ptr [ebp+20h]

SYMBOL_STACK_INDEX: 4

SYMBOL_NAME: win32k!OffBitBlt+97

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: win32k

IMAGE_NAME: win32k.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 48d1b9ef

FAILURE_BUCKET_ID: 0x8E_win32k!OffBitBlt+97

BUCKET_ID: 0x8E_win32k!OffBitBlt+97

ansaurus

tags:

views:

answers: