Alesis QS MIDI Sysex Data Conversion

Question

My aim is to convert a stream of byte code sent from an Alesis synthesizer to a human readable format. I need to be able to take a "Program Dump" and read the 10 character string that makes up the patch name.

In order to receive the "Program Dump" from the synth, I sent the synth the following command via MIDI-OX:

F0 00 00 0E 0E 01 73 F7

I requested that it send me the dump for program 73.

I received this:

F0 00 00 0E 0E 00 73 00 60 24 0B 27 27 01 64 1E 19 19 05 23 19 1E 2A 41 0D 23 46 19 1E 06 00 47 0D 23 30 6C 18 63 30 6C 18 40 3F 0A 67 1B 16 20 40 00 60 18 00 18 06 05 0C 2B 41 13 70 05 30 40 31 63 70 05 00 40 31 63 70 05 00 40 31 63 00 4C 2A 51 00 46 7F 78 18 40 0F 40 31 40 31 04 30 0C 00 30 6C 03 30 3C 0F 00 00 05 0A 0F 14 19 1E 23 28 2D 72 00 76 34 3C 54 42 19 46 0C 33 3C 0C 00 0E 1B 46 60 58 31 46 61 58 31 00 7F 14 4E 37 6C 74 13 00 40 31 00 30 0C 0A 18 56 02 27 60 0B 60 00 63 46 61 0B 00 00 63 46 61 0B 00 00 63 46 01 18 55 22 01 0C 7F 71 31 00 1F 00 63 00 63 08 60 18 00 60 58 07 60 18 1E 00 00 0A 14 1E 28 32 3C 46 50 5A 64 01 0C 2D 15 29 05 36 0C 19 66 78 18 00 1C 36 0C 41 31 63 0C 43 31 63 00 7E 29 1C 6F 58 00 01 02 00 63 00 60 18 14 30 2C 05 4E 40 17 40 01 46 0D 43 17 00 00 46 0D 43 17 00 00 46 0D 03 30 2A 45 02 18 7E 63 63 00 3E 00 46 01 46 11 40 31 00 40 31 0F 40 71 3D 00 00 14 28 3C 50 64 78 0C 21 35 49 03 58 4C 71 31 1C 6C 18 32 4C 71 31 00 38 6C 18 02 63 46 19 06 63 46 01 7C 53 00 60 18 53 37 6C 70 0D 03 40 31 28 60 58 0A 1C 01 2F 00 03 0C 1B 06 2F 00 00 0C 1B 06 2F 00 00 0C 1B 06 60 54 0A 05 30 7C 47 47 01 7C 00 0C 03 0C 23 00 63 00 00 63 1E 3C 63 18 00 00 28 50 78 20 49 71 19 42 6A 12 07 F7

MIDI-OX told me that it received 408 bytes.

This jives with the specification:

"There are 400 data bytes sent for a single program dump, which corresponds to 350 bytes of program data. With the header, the total number of bytes transmitted with a program dump is 408. The location of each parameter within a program dump is shown in the next section."

The "Program Dump" should be composed of these values:

F0 00 00 0E 0E 00 <program#> <data> F7

That means the data should begin with "00 60" and end with "07 F7".

Now I should be able to convert these 400 bytes to the "350 bytes of packed parameter data" for this program. Following the "Program Data Format", 10 digits of the program name should be located within the packed data somewhere. Patch 73 is called either "BlowDeTune" or "PanBristle", not totally sure if it starts at 0 or 1.

So how do you go about make the conversion? Page 1 of the specification gives the transmission format, but I don't understand how to unpack it.

Can anyone help?

The Alesis QS MIDI Sysex Specification is here:

http://www.midiworld.com/quadrasynth/qs_swlib/qs678r.pdf

MIDI-OX can be found here:

http://www.midiox.com/

Answer 1

You are lucky, because some years ago I played a bit with Midi (with my Atari ST 520) so I had enough interest in the topic to investigate a bit...

For the record, I found the MIDI System Exclusive Message format, in accordance with the reference you give for your synth.
I first thought the packing algorithm was described in this page, but after implementing its decoding and founding garbage, I saw I was wrong... I will give this code just in case it could be useful to you elsewhere...

This first try was useful because when I re-read the spec in the PDF file, I understood the A7 to G0 symbols were actually bits...

Data is "packed" because Midi non-control words must be 7bit clean (high bit always unset).
They take 7 bytes of raw data, see it as a big word of 56 bits, and split that word every 7 bits, leaving the high bit always at 0:

Original data (using a different notation):

0 - b07 b06 b05 b04 b03 b02 b01 b00
1 - b17 b16 b15 b14 b13 b12 b11 b10
2 - b27 b26 b25 b24 b23 b22 b21 b20
3 - b37 b36 b35 b34 b33 b32 b31 b30
4 - b47 b46 b45 b44 b43 b42 b41 b40
5 - b57 b56 b55 b54 b53 b52 b51 b50
6 - b67 b66 b65 b64 b63 b62 b61 b60

Transmitted/encoded data:

0 -  0  b06 b05 b04 b03 b02 b01 b00
1 -  0  b15 b14 b13 b12 b11 b10 b07
2 -  0  b24 b23 b22 b21 b20 b17 b16
3 -  0  b33 b32 b31 b30 b27 b26 b25
4 -  0  b42 b41 b40 b37 b36 b35 b34
5 -  0  b51 b50 b47 b46 b45 b44 b43
6 -  0  b60 b57 b56 b55 b54 b53 b52
7 -  0  b67 b66 b65 b64 b63 b62 b61

So we have:

0 - 00000000 0x00
1 - 01100000 0x60
2 - 00100100 0x24
3 - 00001011 0x0B
4 - 00100111 0x27
5 - 00100111 0x27
6 - 00000001 0x01
7 - 01100100 0x64

0 - 00011110 0x1E
1 - 00011001 0x19
2 - 00011001 0x19
3 - 00000101 0x05
4 - 00100011 0x23
5 - 00011001 0x19
6 - 00011110 0x1E
7 - 00101010 0x2A

and once decoded, we should have:

0 - 00000000 0x00
1 - 00110000 0x30
2 - 01101001 0x69
3 - 01110001 0x71
4 - 00111010 0x3A
5 - 00000101 0x05
6 - 11001000 0xC8

0 - 10011110 0x9E
1 - 01001100 0x4C
2 - 10100110 0xA6
3 - 00110000 0x30
4 - 11001010 0xCA
5 - 01111000 0x78
6 - 01010100 0x54

I believe I decoded correctly the data, but still have garbage (ie. non readable strings)...
Perhaps you will see a logic error in my code, which might be a starting point anyway.

I saw that MIDI-OX can be scripted with WSH, so I wrote a JS script which I ran with WSH, with output on console:

var midiData =
[
  0xF0, 0x00, 0x00, 0x0E, 0x0E, 0x00, 0x73,
  0x00, 0x60, 0x24, 0x0B, 0x27, 0x27, 0x01, 0x64, 0x1E, 0x19, 0x19, 0x05, 0x23, 0x19, 0x1E, 0x2A,
  0x41, 0x0D, 0x23, 0x46, 0x19, 0x1E, 0x06, 0x00, 0x47, 0x0D, 0x23, 0x30, 0x6C, 0x18, 0x63, 0x30,
  0x6C, 0x18, 0x40, 0x3F, 0x0A, 0x67, 0x1B, 0x16, 0x20, 0x40, 0x00, 0x60, 0x18, 0x00, 0x18, 0x06,
  0x05, 0x0C, 0x2B, 0x41, 0x13, 0x70, 0x05, 0x30, 0x40, 0x31, 0x63, 0x70, 0x05, 0x00, 0x40, 0x31,
  0x63, 0x70, 0x05, 0x00, 0x40, 0x31, 0x63, 0x00, 0x4C, 0x2A, 0x51, 0x00, 0x46, 0x7F, 0x78, 0x18,
  0x40, 0x0F, 0x40, 0x31, 0x40, 0x31, 0x04, 0x30, 0x0C, 0x00, 0x30, 0x6C, 0x03, 0x30, 0x3C, 0x0F,
  0x00, 0x00, 0x05, 0x0A, 0x0F, 0x14, 0x19, 0x1E, 0x23, 0x28, 0x2D, 0x72, 0x00, 0x76, 0x34, 0x3C,
  0x54, 0x42, 0x19, 0x46, 0x0C, 0x33, 0x3C, 0x0C, 0x00, 0x0E, 0x1B, 0x46, 0x60, 0x58, 0x31, 0x46,
  0x61, 0x58, 0x31, 0x00, 0x7F, 0x14, 0x4E, 0x37, 0x6C, 0x74, 0x13, 0x00, 0x40, 0x31, 0x00, 0x30,
  0x0C, 0x0A, 0x18, 0x56, 0x02, 0x27, 0x60, 0x0B, 0x60, 0x00, 0x63, 0x46, 0x61, 0x0B, 0x00, 0x00,
  0x63, 0x46, 0x61, 0x0B, 0x00, 0x00, 0x63, 0x46, 0x01, 0x18, 0x55, 0x22, 0x01, 0x0C, 0x7F, 0x71,
  0x31, 0x00, 0x1F, 0x00, 0x63, 0x00, 0x63, 0x08, 0x60, 0x18, 0x00, 0x60, 0x58, 0x07, 0x60, 0x18,
  0x1E, 0x00, 0x00, 0x0A, 0x14, 0x1E, 0x28, 0x32, 0x3C, 0x46, 0x50, 0x5A, 0x64, 0x01, 0x0C, 0x2D,
  0x15, 0x29, 0x05, 0x36, 0x0C, 0x19, 0x66, 0x78, 0x18, 0x00, 0x1C, 0x36, 0x0C, 0x41, 0x31, 0x63,
  0x0C, 0x43, 0x31, 0x63, 0x00, 0x7E, 0x29, 0x1C, 0x6F, 0x58, 0x00, 0x01, 0x02, 0x00, 0x63, 0x00,
  0x60, 0x18, 0x14, 0x30, 0x2C, 0x05, 0x4E, 0x40, 0x17, 0x40, 0x01, 0x46, 0x0D, 0x43, 0x17, 0x00,
  0x00, 0x46, 0x0D, 0x43, 0x17, 0x00, 0x00, 0x46, 0x0D, 0x03, 0x30, 0x2A, 0x45, 0x02, 0x18, 0x7E,
  0x63, 0x63, 0x00, 0x3E, 0x00, 0x46, 0x01, 0x46, 0x11, 0x40, 0x31, 0x00, 0x40, 0x31, 0x0F, 0x40,
  0x71, 0x3D, 0x00, 0x00, 0x14, 0x28, 0x3C, 0x50, 0x64, 0x78, 0x0C, 0x21, 0x35, 0x49, 0x03, 0x58,
  0x4C, 0x71, 0x31, 0x1C, 0x6C, 0x18, 0x32, 0x4C, 0x71, 0x31, 0x00, 0x38, 0x6C, 0x18, 0x02, 0x63,
  0x46, 0x19, 0x06, 0x63, 0x46, 0x01, 0x7C, 0x53, 0x00, 0x60, 0x18, 0x53, 0x37, 0x6C, 0x70, 0x0D,
  0x03, 0x40, 0x31, 0x28, 0x60, 0x58, 0x0A, 0x1C, 0x01, 0x2F, 0x00, 0x03, 0x0C, 0x1B, 0x06, 0x2F,
  0x00, 0x00, 0x0C, 0x1B, 0x06, 0x2F, 0x00, 0x00, 0x0C, 0x1B, 0x06, 0x60, 0x54, 0x0A, 0x05, 0x30,
  0x7C, 0x47, 0x47, 0x01, 0x7C, 0x00, 0x0C, 0x03, 0x0C, 0x23, 0x00, 0x63, 0x00, 0x00, 0x63, 0x1E,
  0x3C, 0x63, 0x18, 0x00, 0x00, 0x28, 0x50, 0x78, 0x20, 0x49, 0x71, 0x19, 0x42, 0x6A, 0x12, 0x07,
  0xF7
];

// Show original data
DumpData(midiData, 16);

var headerLength = 7; // Bytes to skip
var resultData = new Array();
var decodedByteCount = 0;  // Number of expanded bytes in result

var cumulator = 0;
var bitCount = 0;
for (var i = headerLength; // Skip header
    i < midiData.length - 1; // Omit EOF
    i++)
{
  var rank = (i - headerLength) % 8; // We split the data in runs of 8 bytes
  // We cumulate the bits of these runs (less the high bit) to make a big word of 56 bits
/*
  cumulator |= midiData[i] << (7 * rank);
  if (rank == 7)  // End of the run
  {
    // Split the cumulator in 7 bytes
    for (var j = 0; j < 7; j++)
    {
      var shift = 8 * j;
      var byte = (cumulator & (0xFF << shift)) >> shift;
      WScript.StdOut.Write(ByteToHex(byte) + ' ');
      resultData[decodedByteCount++] = byte;
    }
    cumulator = 0;  // Reset the buffer
  }
*/
  // Actually, we cannot do that, because JS' bit arithmetic seems to be limited to signed 32 bits!
  // So I get the bytes out as soon as they are complete.
  // Somehow, it is more elegant anyway (but reflects less the original algorithm).
  cumulator |= midiData[i] << bitCount;
  bitCount += 7;
//~   WScript.StdOut.Write((i - 7) + ':' + ByteToHex(midiData[i]) + ' (' + bitCount + ') ' + DecimalToHex(cumulator) + '\n');
  if (bitCount >= 8)
  {
    var byte = cumulator & 0xFF;
    bitCount -= 8;
    cumulator >>= 8;
    resultData[decodedByteCount++] = byte;
//~     WScript.StdOut.Write((i - 7) + ':' + ByteToHex(midiData[i]) + ' (' + bitCount + ') ' + DecimalToHex(cumulator) + ' > '  + ByteToHex(byte) + '\n');
  }
}
DumpData(resultData, 14);

The utility routines:

function DumpData(data, lineLength)
{
  WScript.StdOut.Write("Found " + data.length + " bytes\n");
  var txt = '';
  for (var i = 0; i < data.length; i++)
  {
    var rd = data[i];
    if (rd > 31)
    {
      txt += String.fromCharCode(rd);
    }
    else
    {
      txt += '.';
    }
    WScript.StdOut.Write(ByteToHex(rd) + ' ');
    if ((i+1) % lineLength == 0)
    {
      WScript.StdOut.Write(' ' + txt + '\n');
      txt = '';
    }
  }
  WScript.StdOut.Write(' ' + txt + '\n');
}

function NibbleToHex(halfByte)
{
  return String.fromCharCode(halfByte < 10 ?
      halfByte + 48 : // 0 to 9
      halfByte + 55); // A to F
}

function ByteToHex(dec)
{
  var h = (dec & 0xF0) >> 4;
  var l = dec & 0x0F;
  return NibbleToHex(h) + NibbleToHex(l);
}

function DecimalToHex(dec)
{
  var result = '';
  do
  {
    result = ByteToHex(dec & 0xFF) + result;
    dec >>= 8;
  } while (dec > 0);
  return result;
}

Output:

Found 350 bytes
00 30 69 71 3A 05 C8 9E 4C A6 30 CA 78 54  .0iq:.ȞL¦0ÊxT
C1 C6 C8 98 F1 18 00 C7 C6 08 C6 C6 8C 61  ÁÆȘñ..ÇÆ.Æƌa
6C 0C F0 A7 38 6F 2C 20 20 00 8C 01 60 0C  l.ð§8o,  .Œ.`.
05 C6 2A 38 81 17 60 C0 D8 18 5E 00 00 63  .Æ*8.`ÀØ.^..c
63 78 01 00 8C 8D 01 4C 55 14 60 FC E3 31  cx..Œ.LU.`üã1
C0 07 30 06 8C 11 60 0C 00 8C 3D 80 F1 1E  À.0.Œ.`..Œ=€ñ.
00 40 41 F1 A0 64 3C 23 54 4B 0E B0 D3 78  .@Añ d<#TK.°Óx
54 61 C6 C8 98 F1 18 00 C7 C6 08 C6 C6 8C  TaÆȘñ..ÇÆ.Æƌ
61 6C 0C F0 A7 38 6F 6C FA 04 00 8C 01 60  al.ð§8olú..Œ.`
0C 05 C6 2A 38 81 17 60 C0 D8 18 5E 00 00  ..Æ*8.`ÀØ.^..
63 63 78 01 00 8C 8D 01 4C 55 14 60 FC E3  ccx..Œ.LU.`üã
31 C0 07 30 06 8C 11 60 0C 00 8C 3D 80 31  1À.0.Œ.`..Œ=€1
1E 00 40 41 F1 A0 64 3C 23 54 4B 0E 30 5A  ..@Añ d<#TK.0Z
95 54 C1 C6 C8 98 F1 18 00 C7 C6 08 C6 C6  •TÁÆȘñ..ÇÆ.ÆÆ
8C 61 6C 0C F0 A7 38 6F 2C 20 20 00 8C 01  Œal.ð§8o,  .Œ.
60 0C 05 C6 2A 38 81 17 60 C0 D8 18 5E 00  `..Æ*8.`ÀØ.^.
00 63 63 78 01 00 8C 8D 01 4C 55 14 60 FC  .ccx..Œ.LU.`ü
E3 31 C0 07 30 06 8C 11 60 0C 00 8C 3D 80  ã1À.0.Œ.`..Œ=€
F1 1E 00 40 41 F1 A0 64 3C 23 54 4B 0E B0  ñ..@Añ d<#TK.°
CC 78 8C C3 C6 C8 98 F1 18 00 C7 C6 08 C6  ÌxŒÃÆȘñ..ÇÆ.Æ
C6 8C 61 6C 0C F0 A7 00 30 66 7A 63 C3 1B  ƌal.ð§.0fzcÃ.
03 60 0C 05 C6 2A 38 81 17 60 C0 D8 18 5E  .`..Æ*8.`ÀØ.^
00 00 63 63 78 01 00 8C 8D 01 4C 55 14 60  ..ccx..Œ.LU.`
FC E3 31 C0 07 30 06 8C 11 60 0C 00 8C 3D  üã1À.0.Œ.`..Œ=
BC 31 06 00 40 41 F1 A0 64 3C 23 54 4B 0E  ¼1..@Añ d<#TK.

And just in case, the other unpacking algorithm:

// Here the 8 bits of 7 bytes of raw data are coded as 7 bytes of data stripped off of the high bit,
// while the stripped bits are grouped in the first byte of the data run.
// In other words, when we have a run of 8 bytes, the first one groups the high bits of the 7 next bytes.
// Information found at http://crystal.apana.org.au/ghansper/midi_introduction/file_dump.html

var headerLength = 7;
var resultData = new Array();
var decodedByteCount = 0;  // Number of expanded bytes in result
var runCount = -1; // Number of runs in the encoded data
for (var i = headerLength; // Skip header
    i < midiData.length - 1; // Omit EOF
    i++)
{
  var rank = (i - headerLength) % 8; // We split the data in runs of 8 bytes
  if (rank == 0)  // Start of the run
  {
    // Get the high bits
    var highBits = midiData[i];
    runCount++;
//~     WScript.StdOut.Write(runCount + ' > ' + (i - 7) + ' >> ' + ByteToHex(highBits) + '\n');
  }
  else
  {
    resultData[decodedByteCount++] = midiData[i] |
        ((highBits & (1 << (7 - rank))) << rank);
//~     WScript.StdOut.Write((i - 7) + ' >> ' +  ByteToHex(midiData[i]) + ' > ' +
//~         ByteToHex(midiData[i] | ((highBits & (1 << (7 - rank))) << rank)) + '\n');
  }
}