Replicating C struct padding in Java

Question

Hi,

According to here, the C compiler will pad out values when writing a structure to a binary file. As the example in the link says, when writing a struct like this:

struct {
 char c;
 int i;
} a;

to a binary file, the compiler will usually leave an unnamed, unused hole between the char and int fields, to ensure that the int field is properly aligned.

How could I to create an exact replica of the binary output file (generated in C), using a different language (in my case, Java)?

Is there an automatic way to apply C padding in Java output? Or do I have to go through compiler documentation to see how it works (the compiler is g++ by the way).

Answer 1

This hole is configurable, compiler has switches to align structs by 1/2/4/8 bytes.

So the first question is: Which alignment exactly do you want to simulate?

Answer 2

This is true not only when writing to files, but also in memory. It is the fact that the struct is padded in memory, that leads to the padding showing up in the file, if the struct is written out byte-by-byte.

It is in general very hard to replicate with certainty the exact padding scheme, although I guess some heuristics would get you quite far. It helps if you have the struct declaration, for analysis.

Typically, fields larger than one char will be aligned so that their starting offset inside the structure is a multiple of their size. This means shorts will generally be on even offsets (divisible by 2, assuming sizeof (short) == 2), while doubles will be on offsets divisible by 8, and so on.

UPDATE: It is for reasons like this (and also reasons having to do with endianness) that it is generally a bad idea to dump whole structs out to files. It's better to do it field-by-field, like so:

put_char(out, a.c);
put_int(out, a.i);

Assuming the put-functions only write the bytes needed for the value, this will emit a padding-less version of the struct to the file, solving the problem. It is also possible to ensure a proper, known, byte-ordering by writing these functions accordingly.

Answer 3

You can alter the packing on the c side to ensure that no padding is used, or alternatively you can look at the resultant file format in a hex editor to allow you to write a parser in Java that ignores bytes that are padding.

Answer 4

Don't do this, it is brittle and will lead to alignment and endianness bugs.

For external data it is much better to explicitly define the format in terms of bytes and write explicit functions to convert between internal and external format, using shift and masks (not union!).

Answer 5

With Java, the size of data types are defined by the language specification. For example, a byte type is 1 byte, short is 2 bytes, and so on. This is unlike C, where the size of each type is architecture-dependent.

Therefore, it would be important to know how the binary file is formatted in order to be able to read the file into Java.

It may be necessary to take steps in order to be certain that fields are a specific size, to account for differences in the compiler or architecture. The mention of alignment seem to suggest that the output file will depend on the architecture.

Answer 6

Is there an automatic way to apply C padding in Java output? Or do I have to go through compiler documentation to see how it works (the compiler is g++ by the way).

Neither. Instead, you explicitly specify a data/communication format and implement that specification, rather than relying on implementation details of the C compiler. You won't even get the same output from different C compilers.

Answer 7

For interoperability, look at the ByteBuffer class.

Essentially, you create a buffer of a certain size, put() variables of different types at different positions, and then call array() at the end to retrieve the "raw" data representation:

ByteBuffer bb = ByteBuffer.allocate(8);
bb.order(ByteOrder.LITTLE_ENDIAN);
bb.put(0, someChar);
bb.put(4, someInteger);
byte[] rawBytes = bb.array();

But it's up to you to work out where to put padding-- i.e. how many bytes to skip between positions.

For reading data written from C, then you generally wrap() a ByteBuffer around some byte array that you've read from a file.

In case it's helpful, I've written more on ByteBuffer.

Answer 8

you could try preon:

Preon is a java library for building codecs for bitstream-compressed data in a declarative (annotation based) way. Think JAXB or Hibernate, but then for binary encoded data.

it can handle Big/Little endian binary data, alignment (padding) and various numeric types along other features. It is a very nice library, I like it very much

my 0.02$

Answer 9

As I understand it, you're saying that you don't control the output of the C program. You have to take it as given.

So do you have to read this file for some specific set of structures, or do you have to solve this in a general case? I mean, is the problem that someone said, "Here's the file created by program X, you have to read it in Java"? Or do they expect your Java program to read the C source code, find the structure definition, and then read it in Java?

If you've got a specific file to read, the problem isn't really very difficult. Either by reviewing the C compiler specifications or by studying example files, figure out where the padding is. Then on the Java side, read the file as a stream of bytes, and build the values you know are coming. Basically I'd write a set of functions to read the required number of bytes from an InputStream and turn them into the appropriate data type. Like:

int readInt(InputStream is,int len)
  throws PrematureEndOfDataException
{
  int n=0;
  while (len-->0)
  {
    int i=is.read();
    if (i==-1)
      throw new PrematureEndOfDataException();
    byte b=(byte) i;
    n=(n<<8)+b;
  }
  return n;
}

Answer 10

A handy way of reading/writing C structs in Java is to use the javolution Struct class (see http://www.javolution.org). This won't help you with automatically padding/aligning your data, but it does make working with raw data held in a ByteBuffer much more convenient. If you're not familiar with javolution, it's well worth a look as there's lots of other cool stuff in there too.

Answer 11

I highly recommend protocol buffers for exactly this problem.