Parsing variable record lengths in Preon

Question

I'm trying to use Preon to parse binary files, which are structured as a sequence of variable length records. For each record, there's a number which specifies the record length (in bytes).

Here's a simplified version of what I'm trying to do:

package test.preon; 

import nl.flotsam.preon.annotation.BoundList; 
import nl.flotsam.preon.annotation.BoundNumber; 
import java.util.List; 

public class BinFile { 
    @BoundNumber(size="16") int numberOfRecords; 
    @BoundList(type=Record.class, size="numberOfRecords") List<Record> records; 

    public int getNumberOfRecords() { 
        return numberOfRecords;
    } 

    public List<Record> getRecords() { 
        return records;
    } 

    public class Record { 
        @BoundNumber(size="16") int recordLength; 
        @BoundList(size="recordLength") byte[] data; 

        public int getRecordLength() { 
            return recordLength; 
        } 

        public byte[] getData() { 
            return data; 
        } 
    } 
}

So, numberOfRecords specifies the number of records in the file and recordLength specifies the length of each record. The problem is that Preon isn't able to resolve recordLength in Record, although numberOfRecords works fine in BinFile.

Here's the exception I get:

nl.flotsam.limbo.BindingException: Failed to resolve recordLength on class test.preon.BinFile
at nl.flotsam.preon.codec.BindingsContext$BindingsResolver.get(BindingsContext.java:412)
at nl.flotsam.preon.codec.BindingsContext$BindingReference.resolve(BindingsContext.java:247)
at nl.flotsam.preon.codec.BindingsContext$BindingReference.resolve(BindingsContext.java:189)
at nl.flotsam.limbo.ast.ReferenceNode.eval(ReferenceNode.java:57)
at nl.flotsam.limbo.ast.ArithmeticNode$Operator$5.eval(ArithmeticNode.java:109)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:250)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:33)
at nl.flotsam.limbo.ast.ArithmeticNode$Operator$3.eval(ArithmeticNode.java:83)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:250)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:33)
at nl.flotsam.limbo.ast.ArithmeticNode$Operator$5.eval(ArithmeticNode.java:109)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:250)
at nl.flotsam.limbo.ast.ArithmeticNode.eval(ArithmeticNode.java:33)
at nl.flotsam.preon.codec.ListCodecFactory$SwitchingListCodec.decode(ListCodecFactory.java:458)
at nl.flotsam.preon.codec.ListCodecFactory$SwitchingListCodec.decode(ListCodecFactory.java:443)
at nl.flotsam.preon.binding.StandardBindingFactory$FieldBinding.load(StandardBindingFactory.java:128)
at nl.flotsam.preon.codec.ObjectCodecFactory$ObjectCodec.decode(ObjectCodecFactory.java:251)
at nl.flotsam.preon.DefaultCodecFactory$DefaultCodec.decode(DefaultCodecFactory.java:173)
at nl.flotsam.preon.Codecs.decode(Codecs.java:218)
at nl.flotsam.preon.Codecs.decode(Codecs.java:199)
    ...

If I change size="recordLength" to a constant, e.g. size="42", I don't get the exception (but of course, then the record length always has to be the same).

Is there some other way for me to make the record length variable, or should I have organised things differently?

If anybody's interested, here's the JUnit test I've used:

package test.preon;

import org.junit.Test;
import static org.junit.Assert.*;
import nl.flotsam.preon.Codecs;
import nl.flotsam.preon.Codec;
import nl.flotsam.preon.DecodingException;
import test.preon.BinFile;
import test.preon.BinFile.Record;
import java.util.List;

public class BinFileTest {

    @Test
    public void parseBinFile() throws DecodingException {
        Codec<BinFile> codec = Codecs.create(BinFile.class);
        byte[] buffer = new byte[] {
                2, 0, 
                3, 0, 
                'a', 'b', 'c',
                4, 0,
                '1', '2', '3', '4'
        };
        BinFile b = Codecs.decode(codec, buffer);

        assertEquals(b.getNumberOfRecords(), 2);

        List<Record> rL = b.getRecords();

        assertEquals(rL.size(), 2);

        Record r0 = rL.get(0);
        assertEquals(r0.getRecordLength(), 3);
        assertEquals(new String(r0.getData()), "abc");

        Record r1 = rL.get(1);
        assertEquals(r1.getRecordLength(), 4);
        assertEquals(new String(r1.getData()), "1234");
    }
}

Answer 1

It turns out you ran into a bug. The ListCodecFactory has a policy for deciding what type of Codec to generate in various circumstances, and it turns out it picks the wrong one in this case. I do have the patch for it, and I can send it to you if you're interested.

Answer 2

(This is rather a follow-up question than an answer)

I am doing something similar and having the same problem:

    @BoundNumber (size = "16", byteOrder = ByteOrder.LittleEndian)
    private int     length;

    @BoundString (size = "length")
    private String  name;

I have tried to apply the patch without success. Any plans of making a new release with the fix?

Thanks very much.

Answer 3

Created two bug reports for it: PREON-16 and PREON-17. The first one solves the problem outlined above. The second one solves a slightly related problem, which is worth mentioning here:

In the code below, the size of a Test2 element in the list called 'records' is defined by Test1 entirely. (The size of Test2 is basically the number of characters in Test2.value, which is determined by the 'nrCharacters' attribute of Test1.)

As a consequence of this, Preon is able to optimize reading the list of records. It doesn't have to read all records all at once; instead it is able to skip over it and read those elements only if needed. (The starting position of an element is basically a function of the elements index.)

The difficulty however is that the size of the element needs to be calculated before an instance of Test2 has been read. Since it contains references that are based on the context of Test2, those references need to be rewritten. In fact, the entire size expression (although in this case a simple one) needs to be rewritten. That is solved in PREON-17.

 public static class Test1 {

        @BoundNumber(size = "8")
        public int nrRecords;

        @BoundNumber(size = "8")
        public int nrCharacters;

        @BoundList(size = "nrRecords", type = Test2.class)
        public List<Test2> records;

        public static class Test2 {

            @BoundString(size = "outer.nrCharacters")
            public String value;

        }

    }