Sysex question

I'm working on a very big patch at the moment of which a core part is receiving bulk sysex dumps from a synthesizer, decoding/unpacking them and using [pattr] to distribute the extracted values across dozens of subpatches.

I tried a few approaches, and in the end I found a solution that works really well.

I send a bulk dump request to intiate the data stream, and receive it using [midiin]. I pass this data stream straight to a js object and decode it using javascript.

SysEx messages tend to fall into one of 3 types:

1) Single byte for a single parameter
2) Multiple one-byte messages grouped to form a set of related parameters (ie: a voice name, one message per character)
3) MSB/LSB encoded messages (2 bytes for a single parameter

Of course each message is preceded by a SysEx Header (usually 10 bytes) and a footer (usually a Checksum/End-Of-Sysex byte pair).

I created an array-based decoding mechanism which deals with all of the above.

Here it is. The array data in here is highly specific to the XS, but it should give you some ideas and could easily be changed to another synth or device. The code executes 'instantly' on my Mac, at least, it's done by the time the bulk dump is finished, so although not realtime it's very fast.

I have a bunch of additional states to add to it, but this is just for my application. The logic is complete 'as is'

The key to understanding it is the concept of 'state'. Basically, the SysEx arrives in a known format, so I use a variable (streamPointer) which is incremented for every byte received (so I know when the end is reached). For any given byte in the incoming sysex there is a value I'm interested in, or a value I'm not.

There are two arrays. One is called stateArray and one is called actionsArray. For every single byte in the incoming data stream there is a number in stateArray. This number is 0 for a byte I'm not interested in, -1 for the first byte of an MSB/LSB pair and any other number is the 'state' defined for that byte.

As the code runs through this array with each incoming byte, it extracts the state value and if it's not 0 or -1 then I use the state as an index into the actionsArray which defines the [pattr] namespace for the value.

Have a look and see if it gives you any ideas. Feel free to post back if you have any questions.

Posting it here might have messed up the line breaks a little bit, but it's not rocket science to see where ;-)

Eddy

-----

// =====================================================
// Decode_XS.js
//
// Eddy Deegan, 2009
//
// Decode a single Performance Bulk dump in SysEx format
// from a Yamaha Motif XS workstation
//
// Version 0.1
// 27th Nov, 2009
// Code created
// Requires an uninterrupted stream of sysEx (no interleaved data)
// Does not decode while asleep (is asleep by default)
// Wakes up (stops ignoring) on Bang
// Resets to an awake and waiting for SysEx state if bang sent mid-SysEx
// Goes back to sleep at end of SysEx
// Implemented state 1
// 30th Nov
// Added states up to and including 15 (Common Resonance)
// Optimised switches. Noted that array or matrix lookup would be better than switch
// 5th Dec
// Added states up to and including 23 (Common Volume)
// 6th Dec
// Added array-based state lookup and output generation (Nice speed increase!)
// Added states up to and including 42 (Common A/D Insertion connect type)
// 16th Dec
// Added special case for pan levels (translate from XS-native 1-127 to XStremeTouch 0-126)
// Added states up to and including 44 (figured out the bitflag thing for insert switches)
// Divided state and action arrays into blocks
// 18th Dec
// Added mechanism for parsing and decoding dual-byte (MSB/LSB parameters)
// Added states up to and including 138 (end of block 6)
//
// To Do:
// - Verify XS SysEx header before decoding data when awake
// =====================================================

// -------------------------------
// Ins and outs
//
// Inlets:
// 0 - Incoming MIDI
// Outlets:
// 0 - decoded parameter data
// 1 - Bang when starting, resetting or finished
// -------------------------------
inlets = 1;
outlets = 2;

// ---------------------------------
// Global variables, required to store state
// ---------------------------------
var sysexBuffer = new Array(32); // Buffer to rebuild substreams
var streamPointer = 1; // Byte counter as the stream comes in
var state = 255; // Current state of the engine
var substate = 0; // Supplementary state for use by state functions
var textBuffer = ""; // Used to build strings

// Used for decoding 2-byte values
var MSB;
var MSBFlag=0;

// ======= Arrays defining sysex blocks =======
// Dual value parameters (MSB/LSB) are flagged using -1 as the state for the first byte

// Preamble
var Preamble = new Array(0,0,0,0,0,0,0,0,0,0,0,0);

// Performance Common block 1 (30 00 00)
var ComStates1 = new Array(0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,2,3,0,0,0,0,4,5,0,6,7,8,9,10,11,0,12,13,14,15,0,0,16,17,0,18,19,20,0,21,22,0,23,24,25,26,27,28,29,30,31,32,0,33,34,35,36,37,38,39,40,41,42,0,43,0,44,0,0);

// Performance Common - Reverb Params 2 (30 01 00)
var ComStates2 = new Array(0,0,0,0,0,0,0,0,0,0, -1,45,46,-1,47,-1,48,-1,49,-1,50,-1,51,-1,52,-1,53,-1,54,-1,55,-1,56,-1,57,-1,58,-1,59,-1,60,-1,61,-1,62,63,64,0,0);

// Performance Common = Chorus Params 3 (30 02 00)
var ComStates3 = new Array(0,0,0,0,0,0,0,0,0,0, -1,65,66,-1,67,-1,68,-1,69,-1,70,-1,71,-1,72,-1,73,-1,74,-1,75,-1,76,-1,77,-1,78,-1,79,-1,80,-1,81,-1,82,83,84,85,0,0);

// Performance Common = A/D Input Insertion A Params 3 (30 03 00)
var ComStates4 = new Array(0,0,0,0,0,0,0,0,0,0, -1,86,87,-1,88,-1,89,-1,90,-1,91,-1,92,-1,93,-1,94,-1,95,-1,96,-1,97,-1,98,-1,99,-1,100,-1,101,-1,102,-1,103,0,0);

// Performance Common = A/D Input Insertion B Params 3 (30 04 00)
var ComStates5 = new Array(0,0,0,0,0,0,0,0,0,0, -1,104,105,-1,106,-1,107,-1,108,-1,109,-1,110,-1,111,-1,112,-1,113,-1,114,-1,115,-1,116,-1,117,-1,118,-1,119,-1,120,-1,121,0,0);

// Common Master EQ
var ComStates6 = new Array(0,0,0,0,0,0,0,0,0,0, 122,123,124,125,126,127,128,0,129,130,131,0,132,133,134,0,135,136,137,138,0,0);

// ======== Glue all the above together ========
var stateArray = Array.concat(Preamble, ComStates1, ComStates2, ComStates3, ComStates4, ComStates5, ComStates6);
// ======== End Sysex Arrays ==============

// ======== Arrays defining pattr targets =======
var pattr1 = new Array("dummy", "dummy","CategoryMain","CategorySub","KnobFunction","CommonMaster::ComPan", "CommonEG::ComAEGAttack", "CommonEG::ComAEGDecay", "CommonEG::ComAEGSustain", "CommonEG::ComAEGRelease", "ComFEGAttack", "ComFEGDecay", "ComFEGRelease", "ComFEGDepth", "CommonEG::ComCutoff", "CommonEG::ComResonance", "CommonGeneral::ComEQLowFreq", "CommonGeneral::ComEQLowGain", "CommonGeneral::ComEQMidFreq", "CommonGeneral::ComEQMidGain", "CommonGeneral::ComEQMidQ","CommonGeneral::ComEQHiFreq", "CommonGeneral::ComEQHiGain", "CommonMaster::ComVolume","CommonGeneral::ComPortaTime", "ComPortaSwitch", "CommonMaster::ComRevSend", "CommonMaster::ComChoSend", "CommonGeneral::ComAssignKnob1", "CommonGeneral::ComAssignKnob2", "CommonGeneral::ComAssignButton1Mode", "CommonGeneral::ComAssignButton2Mode", "CommonGeneral::ComRibbonMode", "CommonCtrlAsgn::ComRibbonCtrlNo","CommonCtrlAsgn::ComFootSwAssign","CommonCtrlAsgn::ComAssign1CtrlNo", "CommonCtrlAsgn::ComAssign2CtrlNo", "CommonCtrlAsgn::BCCtrlNo", "CommonCtrlAsgn::ComFC1CtrlNo", "CommonCtrlAsgn::ComFC2CtrlNo", "CommonCtrlAsgn::ComAssignBtn1", "CommonCtrlAsgn::ComAssignBtn2","ComADInsertType", "ComInsertionPartSwAD", "ComInsertionPartSw1234");

var pattr2 = new Array("ComReverbType","ComRevTemplateNo","ComRevParam1","ComRevParam2","ComRevParam3","ComRevParam4","ComRevParam5","ComRevParam6","ComRevParam7","ComRevParam8","ComRevParam9","ComRevParam10","ComRevParam11","ComRevParam12","ComRevParam13","ComRevParam14","ComRevParam15","ComRevParam16","ComRevReturn","ComRevPan");

var pattr3 = new Array("ComChoType","ComChoTemplateNo","ComChoParam1","ComChoParam2","ComChoParam3","ComChoParam4","ComChoParam5","ComChoParam6","ComChoParam7","ComChoParam8","ComChoParam9","ComChoParam10","ComChoParam11","ComChoParam12","ComChoParam13","ComChoParam14","ComChoParam15","ComChoParam16","ComChoReturn","ComChoPan","ComChoToRev");

var pattr4 = new Array("ComADInsAType","ComADInsATemplate","ComADInsAParam1","ComADInsAParam2","ComADInsAParam3","ComADInsAParam4","ComADInsAParam5","ComADInsAParam6","ComADInsAParam7","ComADInsAParam8","ComADInsAParam9","ComADInsAParam10","ComADInsAParam11","ComADInsAParam12","ComADInsAParam13","ComADInsAParam14","ComADInsAParam15","ComADInsAParam16");

var pattr5 = new Array("ComADInsBType","ComADInsBTemplate","ComADInsBParam1","ComADInsBParam2","ComADInsBParam3","ComADInsBParam4","ComADInsBParam5","ComADInsBParam6","ComADInsBParam7","ComADInsBParam8","ComADInsBParam9","ComADInsBParam10","ComADInsBParam11","ComADInsBParam12","ComADInsBParam13","ComADInsBParam14","ComADInsBParam15","ComADInsBParam16");

var pattr6 = new Array("CommonMaster::MasterEQGain1", "CommonMaster::MasterEQFreq1", "CommonMaster::MasterEQQ1", "CommonMaster::MasterEQShape1","CommonMaster::MasterEQGain2", "CommonMaster::MasterEQFreq2", "CommonMaster::MasterEQQ2", "CommonMaster::MasterEQGain3","CommonMaster::MasterEQFreq3","CommonMaster::MasterEQQ3","CommonMaster::MasterEQGain4","CommonMaster::MasterEQFreq4","CommonMaster::MasterEQQ4","CommonMaster::MasterEQGain5","CommonMaster::MasterEQFreq5","CommonMaster::MasterEQQ5","CommonMaster::MasterEQShape5");

// ======== Glue all the above together ========
var actionsArray = Array.concat(pattr1, pattr2, pattr3, pattr4, pattr5, pattr6);
// ======== End Pattr arrays ===============

// -------------------------------------------------------------------------
// This function is called when we reach the end of a bulk dump.
// If action is sleep, then set state to 255 (will cause sysEx to be ignored until next bang
// If action is wait, then set state to 0 (will expect incoming performance dump SyxEx
// -------------------------------------------------------------------------
function reset_state(action)
{
streamPointer = 1;
state = action;
substate = 0;
index = 0;
textBuffer = "";
outlet(1, "bang");
}

// ---------------------------------------------------------------------------
// Reset the state on receipt of a bang. There are two kinds of reset, based on the state value.
// 255 is a non-SysEx state, and no decoding is attempted
// 0 is a 'wait' state and starts decoding from the start of a performance dump
// ---------------------------------------------------------------------------
function bang()
{
reset_state(0);
}

// ---------------------------------------------------------------------------
// Incoming bytes are received sequentially here (data)
// The state is checked, and appropriate action taken
// ---------------------------------------------------------------------------
function msg_int(data)
{
if (state==255) return;

// Based on state, call decoding function or output named parameter
// --------------------------------------------------------
// First check for special cases
switch (state)
{
case 1:
PerfName(data); break;

case 5: // -1 from Pan to translate to 0-126 (XS uses 1-127)
data--; break;
}

if (streamPointer < stateArray.length)
{
// Is this the first byte of a MSB/LSB pair?
if (state == -1)
{
// Store the byte, and set a flag to record that we did
MSB = data;
MSBFlag=1;
}

if (state > 1)
{
if (MSBFlag > 0)
{
// The MSB flag is set so this must be an LSB
// Decode the pair
MSB &= 127; // Mask off 8th bit
MSB *= 128; // Shift into upper bits of result
outlet(0, actionsArray[state], MSB+data);
// Reset the flag
MSBFlag=0;
} else {
outlet(0, actionsArray[state], data);
}
}

state = stateArray[streamPointer];

} else {
// If state > stateArray.length ...
state=0;
}

// Wrap up the main loop
streamPointer++;
// Detect end of stream
if (streamPointer == 1060)
{
// The dump is completed, reset and go to sleep
reset_state(255);
}
}

// -------------------------------------------------------------
// Treat this byte as a performance name character
// -------------------------------------------------------------
function PerfName(character)
{
textBuffer += String.fromCharCode(character);
substate++;
if (substate == 20)
{
substate=0;
outlet(0, "PerfName", textBuffer.substr(0,20));
}
return;
}