On linux, specifically, a user could do what you say. But linux will send sigkill to the process if the hard limit is reached(as opposed to the soft limit you've set), and that will terminate the process.
(Remember though, you really really need to run your stuff in a chrooted environment)
Wow. Having thought about this daunting undertaking for all of about 7 minutes I apologize in advance for anything stupid I am about to say.
Is this going to be something like UVA Judge?
If the goal is to allow relatively simple programs to run without allowing malicious users destroying your system then it seems you need to be more proactive than this or you will be patching holes until the end of time.
At a minimum I think you would need to strip out user header files and substitute one of your own that contains the minimum functionality. Disallow assembler. Use a modified stdlib and/or kernel that no-ops or kills the process on any attempted syscall(), etc.
There is an awful lot to consider here.