legacy linker (uses libc5) fails on linux kernel 2.6.25

Question

We have a legacy linker that uses libc5, and due to several factors we only have the binary and not the source. Yes, version control would have saved us from our current problem... that is now in use for our full tool chain and product line, but this particular horse is long gone.

This linker works on linux kernel 2.6.24, but on 2.6.25 (and 2.6.26) it fails with the message

    Virtual memory exceeded in `new'

We had a similar problem with the corresponding legacy compiler, but with some stackoverflow.com answers and much research have discovered that the compiler problem was caused by the "brk randomization" in linux kernel 2.6.25. The workaround for that is to set a sysctl vars and a environment var:

    /proc/sys/kernel/randomize_va_space = 0 or 1
    setenv MALLOC_TOP_PAD_ 536870912

This, however, does not help the linker.

I have found from using "ldd" that the linker has more shared library dependencies (the compiler only had the libc.so.5):

    libg++.so.27 => /usr/lib/libg++.so.27 (0xb7eca000)
    libstdc++.so.27 => /usr/lib/libstdc++.so.27 (0xb7e99000)
    libm.so.5 => /lib/libm.so.5 (0xb7e90000)
    libc.so.5 => /lib/libc.so.5 (0xb7dd3000)

And I have read that I may have to install the libc5 version of libg++.so.27. I hesitate to do that since I dont know if that will override the latest libg++.so.27 and cause problems for non-libc5 apps.

So, do I find and install the libc5 version of libg++.so.27, or is there some better way to disable brk randomization, or is there another difference between kernel 2.6.24 and 2.6.25 that is causing the linker problem?

Edit

See this for all of the details of this search, and my final resolution.

Answer 1

It does not answer your question exactly, but in your situation, I would create a chroot with a known-to-work libc + libstdc++ combination, or even kernel+libc+libstdc++ (in which case you need a virtual machine, obviously). This way, you can try things relatively easily, without disrupting anything else.

The best way to be compatible with an old library is to use this old library, after all, and since it is "just" a toolchain problem, using some sort of jail/chroot/virtual machine should not be too much of a problem ?