How to store binary data in a lua string

Question

I needed to create a custom file format with embedded meta information. Instead of whipping up my own format I decide to just use Lua.

texture
{
   format=GL_LUMINANCE_ALPHA;
   type=GL_UNSIGNED_BYTE;
   width=256;
   height=128;
   pixels=[[
<binary-data-here>]];
}

texture is a function that takes a table as it sole argument. It then looks up the various parameter by name in the table and forward the call on to my C++ routine. Nothing out of the ordinary I hope.

Occasionally the files fail to parse with the following error:

my_file.lua:8: unexpected symbol near ']'

What's going on here?
Is there a better way to store binary data in Lua?

---

Update

It turns out that storing binary data is a lua string is non-trivial. But it is possible when taking care with 3 sequences.

• Long string literals cannot have an embedded closing long bracket.
  This one is pretty obvious.

• Long string literal cannot end with ]== that matches the closing long bracket. This one is more subtle. Luckily the script will refuse to compile if you get this wrong.

• Cannot embed \n or \r.
  Lua's built in line-end processing messes these up. This problem is much more subtle. The script will compile fine but it will yield the wrong data. 0x13 replaced with 0x10, some 0x13's just missing.

To get around these limitation I split the binary data up then concat the various parts back together. I use a python script to generate output like this:

input:='XXXX\nXX]]XX\r\nXXXX]='

texture
{
   format = RGB;
   lg_width = 8;
   lg_height = 7;
   pixels= '' ..
      [[XXXX]] ..
      '\n' ..
      [=[XX]]XX]=] ..
      '\r\n' ..
      [==[XXXX]=]==];
}

Answer 1

The binary data needs to be encoded into printable characters. The simplest method for decoding purposes would be to use C-like escape sequences for all bytes. For example, hex bytes 13 41 42 1E would be encoded as '\19\65\66\30'. Of course, then the encoded data is three to four times larger than the source binary.

Alternatively, you could use something like Base64, but that would have to be decoded at runtime instead of relying on the Lua interpreter. Personally, I'd probably go the Base64 route. There are Lua examples of Base64 encoding and decoding.

Another alternative would be have two files. Use a well defined image format file (e.g. TGA) that is pointed to by a separate Lua script with the additional metadata. If you don't want two files to move around then they could be combined in an archive.

Answer 2

Lua is able to encode most characters in long bracket format including nulls. However, Lua opens the script file in text mode and this causes some problems. On my Windows system the following characters have problems:

Char code(s)      Problem
--------------    -------------------------------
13 (CR)           Is translated to 10 (LF)
13 10 (CR LF)     Is translated to 10 (LF)
26 (EOF)          Causes "unfinished long string near '<eof>'"

If you are not using windows than these may not cause problems, but there may be different text-mode based problems.

---

I was only able to produce the error you received by encoding multiple close brackets:

a=[[
]]] --> a.lua:2: unexpected symbol near ']'

But, this was easily fixed with the following:

a=[==[
]]==]