Parsing real-time sysex messages
Anyone know of an alternative method for receiving and recognizing incoming real-time sysex messages? Unless there’s a hidden option, the [sysexin] object sends out received bytes one by one rather than as a list. Since incoming sysex messages can be different lengths, I don’t see how [match] can be used as it seems to require the exact number of bytes that will be in the message to recognize it.
I looked around but couldn’t see any third party libraries that had such a feature. I’m sure someone must have addressed this issue before me.
I’m wondering if it would make sense to write an external that would accept sysex in messages and produce a single packed message out of it.
I think you can use the end marker (247) to trigger [zl group].
----------begin_max5_patcher---------- 318.3ocyS90SCBCEE+4xmhl6y3Bf3.7M8qgYwTnUVMkVBsDYaYe2kV.2exzs kEM9Bj64dNjS+ArwCA4pNlFvOheAiPa7PHmjU.MNifJRWgfnc1.I6CU96f+v JCqy3j0LANJNYRW1VwkBlwkIZT7Mkzn4qYVsvfYA67pZMGatlXJVxkku1vJL CELKtOCNLZt8V7vP5r.7hwHCOEypZ1feHmHKAeL.eYgScss+DbWFrWsjjJWH 3oFNQfeVInfc6VOO6E+aCNqE3xFUa8uJcBuOwQj4C3I6734aQS5eGZzqzrNt 7jjI7ZHS3OQFGRRevAljyCFtzbRtjbEbwkGDb4w+d4JnU+PXoUsMESEX76S7 tNRYZCWRLbkbOOoG3YImRYt0SHphSqU8mlwJfWbxWbWZiRtfFk8uqQ2Bi5G1 58Ip8.iP -----------end_max5_patcher-----------
Nice — that’s certainly a good way to get this working in the short term — I never think about the [zl] stuff. Thanks for that.
I do wish however there was a sysexin variation that produced a single list per received message rather than sending out each byte separately. There can be a lot of real-time sysex data and it seems a shame to have to process them all one byte at a time in Max. It’s both awkward and expensive. I understand it for ‘traditional’ sysex messages where real-time is less important, but if you move 10 sliders at the same time with your hand and each of them is generating continuous real-time sysex messages, that starts to add up.
I wonder if the Cycling74 team would be willing modify sysexin (perhaps with an attribute) that would allow it to buffer sysex messages and send them out as lists.
Also, would it make sense to detect 240 before storing any data and send a zlclear message out in case something gets screwed up before a 247 byte arrives.
Be aware that using sysex messages for real time control is inherently problematic as any status byte (eg. note-on) which is sent after 240 and before 247 may "abort" the current message.
That’s why I made the suggestion to detect 240 as a means of restarting.
I have not yet done the experiment to see what data comes out of each keyboard if I play notes WHILE moving sliders. Even though those real-time sysex messages are short (my Kronos produces 14 byte messages), I don’t know whether the Kronos will interrupt (and abort) a real-time sysex message that it’s sending out with note on events or whether it will just put them on the front of a queue so it goes out immediately after a real-time sysex is sent.
I’ve previously used the following patch to turn a sysex message into a single max list.
I went with coll rather than zl group to overcome the ‘max list length’ of the zl family, though zl would also work assuming you know in advance how long the sysex messages you are dealing with are going to be (e.g. use [zl 1000 group] if you knew you’d be dealing with 1000 byte messages)
----------begin_max5_patcher---------- 675.3ocwW1rbaBCDG+L9oPiN65AI.amdqSeD5wNY7H.Ua0ADLHQqcyj28JsB RnokZTA6d.Tz5Eoc+s+0G4oUA3zpybEF8dzmQAAOsJH.LYMDz0O.WxNmUvTf a3rpxRtTiW69MM+rFr+oKJ9YDSgXHkPdrfiJDJ8ZToP1pP5SbjRyZzHlLGwM OldZyOjdQal+tAqPH4YUsRXDocF+RkTKYkbXV9PifUz6tHGrUk902kfG3sR7 CvaR3lvNyxVSbTv0PNPd0XUqt2Zuq0Lc1ISFbngmocjgjPMiDJJFZHIasMzG 1Dhdz9MOuZk805IRPI+6lP92.nFQv9jxwQ8FcIg9RM2EtXgkfONahP9KDIFX .I9A.ENtreYAxQll6GQniRD+vA0abPCcDfXa1scSxxiCMJqfyZP4sk0nP+.C YTv.iIdMBaGVa6BochFGVaSbRm8.qhuERGEu.Qi2Yd7DTgiBpTl7nkO8s2GH Qb3I5lr.qtgWa2Jtj2bj6ojJZ2BsVi7OtViFBa8re2sfLYUEEHk8DsOZ9K6T 6CaniuyLnb5eV.NEecN4TR6GrNq6aTruwyOXBXymbfo0MhzVs6x.Aufq.LuL kC4VHX..7Bg4RQtPH8hsj+iG54HYB7NZd5N2D9mzN3k4FKtheTbDnAR5e+Bf 5uFmcdwCR.vA3hXu4VgvjXs+qYkppsIquDzUfPuNM4bkVHYZQkbfO1sYG3zI QdNWNLiLBi5JSAsKFFAwSMjhmRLYWz5SLYN3xbHu4.FPGXtLXeuYFr1MVW7f ct.LbJEUx8unRuVLQuuwDcBb5ME36BmHKrdhDBWHY4U+fRaRUUx8ifQSQ86o RiXNFw9OIrKBZbDD5sDD7Z06Xeq2QD2c1CgvrK1s8lYzRmBamyhXSmmW8Sv7 LCNA -----------end_max5_patcher-----------
That’s a nice implementation and I’ll try it out.
I’m a little leary of having to use a bunch of separate objects (with the implication of a lot of messages) to process real-time sys-ex messages but maybe it will work fine.
I have a feeling though that it would be nice to have a single external into which you could say things like "Match sysex messages that have particular values in known locations (e.g, the bytes that represent individual sliders)" and then emit one or two bytes somewhere else in the message, i.e, the bytes representing the new position of the slider.
Yes, it would make sense to have something like [sxparse] as complement of [sxformat].
Because the format of a SysEx message is arbitrary, it would be very tough to build a parser for them. There are a variety of "compression" modes used to fit data bytes into the 7-bit world of MIDI. Here’s a zl solution, which seems pretty straightforward to me:
----------begin_max5_patcher---------- 605.3ocyV10aaCBEF9Z6eEHtNsxf+JLseG6looIhMIkNavxP5RRU+uO9vVIo MtIyyNZQxAwwGCu7nWNvqgAvUxcLED7Ev2AAAuFFD3BYCDz0O.VS2UTQUtzf Ex5ZlPCW3emlsS6hul9KFnloTzMLvZYKPyTZtXSehqkBshefYSFgeLpKrXaM WTwztAG0ErgpKdx7s+rkUn8hKNeo4a.n3LWSVjsAidLB7iSl.As1MAvuwZKo BZ+jyKcQkqd9ATF73TK2p6m6Hav2BCs+s3FYgf8ayP9ATv0r19f9YPuug4WG P3YB9egHInSHRLZz.I4x.AMBfLj43qO.TLMvjKPKADyOvp8FCxDYNvK8q9Lh 2bjMZVDOclit8BefEHD.iAwwfjDPZJHKCjmOQtE7mfHbr2lfcM44NRQFAhPS mcYHDcnpnhQamepfV53PGbRHiEJjoiICTSQwp.3jHyyvlkUTa41is+cXJ9pX BszWCNkL18W4WlSwSn2wBoKsE6SI2jYn5KJG4IEYzayRmeG0dEaGWLHS3l53 S0YU8FnD29rzzw5evyNUNT4OdZSqbayv9kEyPQH+oWjjQe30kgC9D33FRXEW 79674jkM94DSI21Vzun6ur.3nxJc2wip4RwIIkeVNOwKKYtW2ClRthtphUN7 Iq2pbRuA07NIOmxA+eEbP2BbrIgtKxwtvwWSNWAN07xFoovTm6M1b22Tyn5K nfHG6cbdlT8ilV8iyty5+Z1AxcybRtEy4HUiuJGso4EVqpaHcBwbBvyxVa2r EttbguqaDgsrW384mFZGs2B+CzWiz8K -----------end_max5_patcher-----------
Not sure what kind of parsing you’re talking about — I would be very happy to have a mechanism where an incoming sysex arrives as a single list and then use a SLICE function to extract sublists based on start position and length. A couple of operations to allow a few consecutive bytes to be combined into a single value (e.g, byte0 + byte1< <7 + byte2<<14)
Converting such a sublist to a string would let regex work as well.