How to force gcc use int for system calls, not sysenter?

Question

Hi, all

Is it possible to force gcc use int instruction for all the system calls, but not sysenter? This question may sound strange but I have to compile some projects like Python and Firefox this way.

Summary

Thanks to jbcreix, I've downloaded glibc 2.9 source code, and modified the lines in sysdeps/unix/sysv/linux/i386/sysdep.h, to disable use of sysenter by #undef I386_USE_SYSENTER, and it works.

Answer 1

You don't modify gcc; you modify libc (or more accurately, recompile it) and the kernel. gcc doesn't emit sysenter instructions; it generates calls to the generic syscall(2) interface, which presents a unified front end to system call entry and exit.

Or, you could use a Pentium; SYSENTER wasn't introduced until PII =]. Note the following KernelTrap link for the interesting methods used by Linux: http://kerneltrap.org/node/531

Answer 2

Recompile your C library after replacing sysenter by int 80 in syscall.s and link again.

This is not compiler generated code which means you are lucky.

The ultimate origin of the actual syscall is here, as the OP says:

http://cvs.savannah.gnu.org/viewvc/libc/sysdeps/unix/sysv/linux/i386/sysdep.h?root=libc&view=markup

And as I suspected there really was a syscall.S it's just that the glibc sources are a labyrinth.

http://cvs.savannah.gnu.org/viewvc/libc/sysdeps/unix/sysv/linux/i386/syscall.S?root=libc&view=markup

So I think he got it right, asveikau.