run out of system resource (execute many programs in a shell script)

Question

Hi guys

I'm running a shell script on the university's server. In this shell script, I will execute java, c, c++, python and perl programs. Because every program will be executed many many times(I'm a teaching assistant and will test the students' programs with many different inputs). The server always gives me an error: "running out of system resource". I guess this is due to I do not release the resource.

I heard that running a program in the shell script one time will active one process. So I think maybe there are so many processes that the system recourse allocated for me has been run out.

Is there any way to figure this problem out?

I pose part of my shell code as following:

# maxconnect4 is the compiled c code
for ((i = 1; i <= 21; i++))
do
    maxconnect4 input1.txt
done

Thanks

Zhong

Answer 1

You seem to be running maxconnect4, then waitng for it to finish before starting the next run, so I don't think your shell script itself is the isuue. The big question is what maxconnect4 is doing. It could be very hungry for resources, or it itself could start child processes and return to your script.

I would try a few experiments such as by hand start maxconnect4 a few times, do you se the resource error?

I would also use system tools to invetsigate. For example use ps to see whether there are lots of processes running. Use vmstat to look at CPU and memory usage.

Answer 2

To Djna:

I have run my shell one again. This time, I run the program that wrote by the student using java. It gives me this kind of error:

A fatal error has been detected by the Java Runtime Environment: java.lang.OutofMemoryError: Cannot create GC thread. Out of system resources.

I googled this kind of error. It seems that I run out of swap space.

The following is the result of using vmstat:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 122648 207740 2491412 42870000 0 0 27 7 0 2 0 0 100 0 0

Answer 3

Since you are automatically running students' programs then it may be that their programs are badly written and using more RAM than similar programs written by more skilled programmers would require. Even Java and Python programs can be written in such a way as to leak memory (think about a stack that never gets anything popped off of it, only more things pushed on).

You should test your setup with known good implementations of the assignments you are about to grade as a sanity check.

You should also look at the source code for the students' work. Especially if you get the error on their assignment.

You may also just have an overloaded system, and may need to run these tests on another machine. Using a machine that does not have other users is a good idea for this type of thing, since things outside of your and the program you are testing aren't likely to mess up your tests.

You may also want to keep top running on that machine on another terminal while you run the test to monitor resource usage.