How can I limit memory acquired with `malloc()` without also limiting stack?

Answer 1

Can you force a macro on the unsuspecting students? :-)

#define malloc(bytes) limited_malloc(bytes)

and also a definition for limited_malloc that limits what can be done.

Answer 2

Building on the idea used by failmalloc, you could use the LD_PRELOAD^* environment variable and function interposition to build a wrapper around malloc() and impose any limitations there.

You would need to dynamically load a pointer to the original malloc() using dlsym(). You cannot directly call the original malloc() from the wrapper because it will be interpreted as a recursive call to the wrapper itself.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdint.h>
#include <dlfcn.h>

void * malloc(size_t size)
{
   static void * (*func)(size_t) = NULL;
   void * ret;

   if (!func)
   {
      /* get reference to original (libc provided) malloc */
      func = (void *(*)(size_t)) dlsym(RTLD_NEXT, "malloc");
   }

   /* impose any necessary restrictions before calling malloc */
   ...

   /* call original malloc */
   ret = func(size);

   /* impose any necessary restrictions after calling malloc */
   ...

   return ret;
}

* Note that LD_PRELOAD must specify the full path to the interposer library, and that library interposition is disabled for setuid programs in order to prevent security problems.

---

An alternative to using dlsym() would be to use the GNU linker --wrap symbol option.