How's __RTC_CheckEsp implemented ?

Question

__RTC_CheckEsp is a call that verifies the correctness of the esp, stack, register. It is called to ensure that the value of the esp was saved across a function call.

Anyone knows how it's implemented?

Answer 1

If you're any good at asm, maybe this helps:

jne (Jump if Not Equal) - jumps if the ZERO flag is NZ (NotZero)

_RTC_CheckEsp:
004C8690  jne         esperror (4C8693h) 
004C8692  ret              
esperror:
004C8693  push        ebp  
004C8694  mov         ebp,esp 
004C8696  sub         esp,0 
004C8699  push        eax  
004C869A  push        edx  
004C869B  push        ebx  
004C869C  push        esi  
004C869D  push        edi  
004C869E  mov         eax,dword ptr [ebp+4] 
004C86A1  push        0    
004C86A3  push        eax  
004C86A4  call        _RTC_Failure (4550F8h) 
004C86A9  add         esp,8 
004C86AC  pop         edi  
004C86AD  pop         esi  
004C86AE  pop         ebx  
004C86AF  pop         edx  
004C86B0  pop         eax  
004C86B1  mov         esp,ebp 
004C86B3  pop         ebp  
004C86B4  ret              
004C86B5  int         3    
004C86B6  int         3    
004C86B7  int         3    
004C86B8  int         3    
004C86B9  int         3    
004C86BA  int         3    
004C86BB  int         3    
004C86BC  int         3    
004C86BD  int         3    
004C86BE  int         3    
004C86BF  int         3   

Answer 2

Well a little bit of inspection of the assembler gives it away

0044EE35  mov         esi,esp 
0044EE37  push        3039h 
0044EE3C  mov         ecx,dword ptr [ebp-18h] 
0044EE3F  add         ecx,70h 
0044EE42  mov         eax,dword ptr [ebp-18h] 
0044EE45  mov         edx,dword ptr [eax+70h] 
0044EE48  mov         eax,dword ptr [edx+0Ch] 
0044EE4B  call        eax  
0044EE4D  cmp         esi,esp 
0044EE4F  call        @ILT+6745(__RTC_CheckEsp) (42BA5Eh) 

There are 2 lines to note in this. First note at 0x44ee35 it stores the current value of esp to esi.

Then after the function call is completed it does a cmp between esp and esi. They should both be the same now. If they aren't then someone has either unwound the stack twice or not unwound it.

The _RTC_CheckEsp function looks like this:

_RTC_CheckEsp:
00475A60  jne         esperror (475A63h) 
00475A62  ret              
esperror:
00475A63  push        ebp  
00475A64  mov         ebp,esp 
00475A66  sub         esp,0 
00475A69  push        eax  
00475A6A  push        edx  
00475A6B  push        ebx  
00475A6C  push        esi  
00475A6D  push        edi  
00475A6E  mov         eax,dword ptr [ebp+4] 
00475A71  push        0    
00475A73  push        eax  
00475A74  call        _RTC_Failure (42C34Bh) 
00475A79  add         esp,8 
00475A7C  pop         edi  
00475A7D  pop         esi  
00475A7E  pop         ebx  
00475A7F  pop         edx  
00475A80  pop         eax  
00475A81  mov         esp,ebp 
00475A83  pop         ebp  
00475A84  ret              

As you can see the first thing it check is whether the result of the earlier comparison were "not equal" ie esi != esp. If thats the case then it jumps to the failure code. If they ARE the same then the function simply returns.