In RTOses like vxworks whenever we create a task the stacksize is specified . Can we write a routine in C which checks if the stack is overflowing or not for the task ?
If you know how big the stack is, and if you are careful, then yes (but not portably). If there is no other way to obtain the base address of stack, you need to record the address of a stack variable in the thread's main function; this gives you an approximation to the top of the stack. Then, in your checking function, you take the address of a local variable; this gives you the bottom of the stack. If the difference between the top and the bottom is about your stack size, it is time to worry; if the difference is bigger than the stack size, it is too late to worry - the damage has been done (but now you need to think about how to clean up).
Look at your compiler they often let you add prelude functions to do this or they they might even check it themself unless you manipulate the stack pointer register.
And check if the operating system allows you to install "guard-pages". Mark the last page in your threads stack as non-read/non-write and catch the SIGSEGV signal and use a OS/CPU specific way to find out if it is the guard-page that failed. For this to work you must be sure that the stackframe of a function (stack passed parameters, local variables and alloca allocated space) is always less then a page size otherwise you can skip over the "guard-page" This is the best way to handle it as it has no runtime overhead during normal processing.
You see this highly OS/CPU/Compiler dependent. But i'm pretty sure that google will find useable code and helpers for this technique for all systems as it is a pretty common technique for low level programmers (for example runtime or interpreter implementors).
There's a few techniques you can use - typically you have a low-priority task which sniffs the stack status of all other tasks every second or so.
a: Make sure the stack space is filled with a known pattern before your task starts. You can then find out how much 'uncorrupted' stack is left by checking for the pattern.
- Advantage: Lets you check the"high-watermark" of stack use.
- Disadvantage: If you allocate stack memory, but don't write to it for some reason, this technique MAY not detect the overflow.
b: You can simply sniff the stack pointer of all other threads.
- Disadvantage: This is just "sampling" the stack pointer, so a brief dip into overflow may not be noticed
- Advantage: Quick and easy.
I'd recommend a combination of both. Because you're doing lowlevel stuff using things like the VxWorks TaskInfoGet() functions, it's difficult to make this even remotely portable.
I don't know about VxWorks, but my recollection is that Green Hill's Velosity/uVelosity kernels provide code to do this. Even if they didn't, since they provide source that users can modify, and the infrastructure is there, it would be really easy to add.
Edit: For the sake of disclosure, I did a summer internship with them, porting uVelosity to a new architecture. This is how I became intimate with its handling of thread stacks.
If your particular application statically allocates its threads, then it's possible to place their stacks in statically defined areas, and use a linker map to place a symbol at the end of those areas. Then, you just need to get your current stack pointer (as described in other answers) and compare the "end of stack segment" pointer to that address. This can also work for dynamic allocation, if each thread has some place to store an address provided to it as the end of its stack.
FYI, you can do something like this from the shell in VxWorks using checkStack().