Working around Java JIT bug

Question

We seem to be subject to a strange bug in our Java environment. We've now had two occurrences of the same "can't happen" exception; in one case the problem occurred 42,551 times over a period of 48 minutes in a running process and then spontaneously cleared itself.

The failing code is triggered by this line:

return String.format("%1d%XY%d", source, System.currentTimeMillis(), quoteID);

where int source = 0 and long quoteID = 44386874 (for example).

The exception is:

java.util.UnknownFormatConversionException: Conversion = 'd'
        at java.util.Formatter$FormatSpecifier.conversion(Formatter.java:2605)
        at java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2633)
        at java.util.Formatter.parse(Formatter.java:2479)
        at java.util.Formatter.format(Formatter.java:2413)
        at java.util.Formatter.format(Formatter.java:2366)
        at java.lang.String.format(String.java:2770)

Checking the code 'd' should never raise this exception.

The best explanation we've come up with is that the JIT compiler is generating bad bytecode, but on a subsequent re-JIT it writes good code.

Anyone have any experience of ways to work around / diagnose such a problem?

Roger.

Answer 1

I doubt this is a legit JIT issue.

Have you ruled out other possibilities such as memory corruption or runtime environment issues?

How did you conclude that this is a JIT problem?

Just to ease your mind, this is the code that is throwing the exception:

private char java.util.Formatter.FormatSpecifier.conversion(String s) {
    c = s.charAt(0);
    if (!dt) {
 if (!Conversion.isValid(c))
     throw new UnknownFormatConversionException(String.valueOf(c));

        ///////..........
}

with:

static boolean java.util.Formatter.Conversion.isValid(char c) {
    return (isGeneral(c) || isInteger(c) || isFloat(c) || isText(c)
     || c == 't' || c == 'c');
}

d is a legit integer identifier, and isValid() should return with True.

Debugging this is not the problem, and an educated guess will say that you will find nothing. This is clearly a memory corruption/environment issue. It sounds like this issue is easily reproduced. Try testing on a different machine, different OS, different JVM.

My hunch - your problem is not the JIT compiler.

Answer 2

It's a long shot, but have you checked the machine for memory or other errors? Those can definitely cause really weird "impossible" bugs. Running Prime95 worked well for me in the past (better than a "real" memory checker - probably because it stresses the CPU and perhaps other components too).

Answer 3

The first thing I would do is to check the source code of java.util.Formatter and see if there is any other checks performed resulting in an exception for the 'd' pattern character. Which Java version are you using?

Answer 4

Do:

final long time;

time = System.currentTimeMillis();

try
{
    return String.format("%1d%XY%d", source, time, quoteID);
}
catch(final UnknownFormatConversionExceptio ex)
{
    // log the source
    // log the time
    // log the quoteID
    throw ex;
}

If nothing looks odd there then look at the source for String.format and trace through it, by hand, with those values (sounds like you may have done the tracing, but doing it with the actual values might help).

And as others have said, random memory bugs are also possible here. If possible run a memory check on the machine and see (I have had issues go away by re-seating the memory...).

Answer 5

Are you sure the "d"s in that code are really ASCII d's and not some Unicode character that happens to look like a d?

(A long shot, but stranger things have happened.)

Answer 6

An easy way to check if it is a JIT bug is to put you code in a loop. If it is a JIT bug it will fail inside the loop but not outside :

for (int i = 0; i < 100000; i++) {
        String.format("%1d%XY%d", source, System.currentTimeMillis(), quoteID);
    }

Answer 7

Test the memory! E.g. with memtest86+. It is avaliable on the Ubuntu cd boot menu.