From The Java Language Specification:
The result of a floating-point
remainder operation as computed by the
% operator is not the same as that
produced by the remainder operation
defined by IEEE 754. The IEEE 754
remainder operation computes the
remainder from a rounding division,
not a truncating division, and so its
behavior is not analogous to that of
the usual integer remainder operator.
Instead, the Java programming language
defines % on floating-point operations
to behave in a manner analogous to
that of the integer remainder
operator; this may be compared with
the C library function fmod. The IEEE
754 remainder operation may be
computed by the library routine
Math.IEEEremainder.
In other words, this is due to the fact that Java rounds the result of the division involved in computing the remainder, while IEEE754 specifies truncating the answer of the division. This particular case seems to expose this difference very clearly.
You can get the answer you expect using Math.IEEEremainder:
System.out.println(3.9 - (3.9 % 0.1));
System.out.println(3.9 - Math.IEEEremainder(3.9, 0.1));