Sysex and Live 9 - is it now possible?
This is a question that despite my best efforts I can't get answered, put simply:
Can an M4L device send SYSEX out through Ableton's Midi ports? I was previously of the understanding that Ableton filtered SYSEX on the way out, is this still the case with V9?
Basically all I want to do is build devices to edit parameters on pre-CC synths. I have built devices for more modern machines like the Roland Gaia but would like to do the same with things like D50's etc etc.
it's my understanding that you can use objects such as thesE:
to get access to sysex data "around" Live, instead of through it.
Thanks Wetterberg, that's sounds just like what I need. Thankyou very much for the heads-up
any chance you could share your Roland Gaia patch? Would love to integrate my keyboard more closely. Thanks, Gen
Sadly I've had some issues getting Leigh's objects to working in 64 bit, the alternative I've come up with is to manage the sysex in/out in Max and build the control surface in M4L, syncing between these OSC. I actually find I prefer this system now for managing patch storage etc.
I've got controllers for the D-50, Alpha Juno 2, ESQ-1, DW8000 and a few others I use, I'll post them when I get the chance.
hei brooke -
it'd be amazing (!) if you could post an exemplary patch of yours.
and: do you run your max-part of the sysex-code from within the max application or did you create a standalone device?
any clue which of the two is more performant?
Hi Brooke,
How much work does the standalone app do? Are you prepping the sysex in MaxForLive and sending it as a message via OSC, or are you sending the app controller values and then converting that to sysex?
I only have MaxForLive. I wanted to put together a MaxForLive device for the Matrix 6/6R/1000. I believe I could make the device send the sysex pre-formatted over OSC. With that said, I cannot build a standalone device that receives sysex over OSC and sends it off as midi.
Any chance you can share this? I will be most happy to share my Matrix 6 device with the entire audio community if I can make this happen :)
PS - this is a serious limitation in Ableton Live. I understand not implementing a way for Live to parse sysex, but to completely truncate it even when originating from a MaxForLive device is excessive and frustrating to say the least...
I found this - https://cycling74.com/tools/midi-and-osc-mapper/ which lead to this http://lamplightforestboutique.com/control-core-midi-osc.html but I am on PC ... /sigh
Woops sorry hadn't noticed updates in the thread.
Benzzzz - Either, packaging it into a max application is just a convenience for people that dont have Max, they are both very fast since we're just dealing with messages.
Basically I'm working on a way to send OSC messages from Live to be converted into Sysex messages by an app (built in Max, packaged as an executable/app), that has a library of OSC<>MIDI sysex conversions availa. That way you can build a M4L device that has all your gui details and save the device state as a live preset that sends its state as a dump message when it's loaded.
Previously I'd been building a OSC<>MIDI conversion app for each device I wanted to control, but now I realise building an app that lets you simply select a sysex format to use from a list and tells you what OSC messages to send there would be much easier. Once you get used to writing sysex editors doing a bunch of them is a breeeze, older devices have occasional quirks though.
I'll work on things later this week then post the patch to github. It'd be great to have some collaborators so we can design nice editors for all those midi devices.
Live not having sysex is a serious limitation, almost as bad as being restriced to midi channel 1 in M4L. But I think designing a OSC to sysex conversion app is actually a superior way to deal with managing sysex.
+1 for the silliness of the midi limitations in live. filtering of duplicate CC messages, no sysex, midi channel limitations. Would love to see this all go away!
i've got a stack of external gear here (dsi, moog, korg, kawai, hohner...). i was planning to beginn a librarian-patch asap, but would very much more like to help out / collaborate. if desired also with the annoying part of interfacing / gui (i recently did m4l's 'polyrhythmus' as a visual reference if wished for).
thanks brooke! we need a decent tool for preset management!
Thank you very much for your reply Brooke.
I maybe misunderstood what you said, so apologies if the following is already what you were describing.
Would it not be easier to have the standalone simply receive a sysex message in the form of a list over OSC, and then send the individual values of the list off to MIDI? My understanding is that any device will automatically know something is sysex if it receives F0h. So if the standalone receives /sysex "F0 xx xx xx xx xx .. F7" then it can forward that off as raw midi data, symbol by symbol (or int by int, if we do "240 xx xx xx .. 247") to the corresponding MIDI channel to get the sysex message through. Of course the symbols would need to be converted to bytes.
This would allow the standalone to simply be responsible for MIDI routing and all the heavy work to be done inside MaxForLive. The standalone becomes just a wrapper to overcome the lack of sysex in Live. We can go further and choose different network ports for different MaxForLive "sysex controllers," with each port corresponding to a different midi port or channel, and then it becomes a sort of wrapper/router. Please let me know your thoughts. I'm about to patch this together in MaxForLive. I would just be unable to compile it, but I'll share what I come up with shortly.
PS - once the wrapper implementation is established, I call dibs for the matrix 6,6r,1k for the MaxForLive "sysex controller" design :) I tried to call dibs on the alpha juno, but as I reread the thread, I see you've already got one made for yourself!
EDIT -
going off my thoughts from earlier regarding the type of list to send, I think a list of ints is ideal. if you monitor sxformat with a print, you'll see it spits out the bytes (ints) sequentially as I suggested the standalone should. patcher incoming tomorrow (I hope) .. I became quite busy today, so I apologize for not having something ready by now.
OK! Patcher time. This is a very simple example of what I feel may be the most light-weight and effective way to get sysex out of Live and into our synths.
I originally planned to use OSC messages like so "/sysex 240 xx xx xx xx 247" and then send that off to a port on the computer, where the standalone application "OSC-routes" /sysex to get to the actual list of ints for the midi. Then, I remembered that CNMAT updates their externals versions at different rates for different objects and OSes. Furthermore, there is no need for the tree/branch/twig address arrangement with this implementation. So I wondered why I needed OSC at all. I thought maybe it was necessary to properly use the network functions, but Max sends data just fine without OSC.
Please note the two different ways I thought about transmitting the midi data. I personally feel that it's more robust to send each sysex message as a list, even if it requires the use of an extra object in the standalone, but you may disagree with these semantics (I just think that we may stall the receiving midi device in the rare chance that the sysex end byte doesn't get through between udpsend and udpreceive).
If anyone (willing to compile this thing up) likes this idea, I will develop it further with a UI to allow the user to enter midi port and channel. If at that point, things are still running smoothly and the application proves useful, we can work on adding multiple network ports for different sysex receiving midi devices.
(btw this is the first time I post a patcher up like this .. so if something is screwy, I'll give it another try)
Nice thinking Matias, since (all?) sysex lists have the manufacturer and model in the list that makes routing messages easier as well.
It should be fine to manage all this with udpsend/receive, since its just a simpler osc subset.
I'll post up an example for the D-50 at the end of the week, sorry things are a bit crazy right now. Feel free to come up with a alpha juno editor, mine is just a bunch of number boxes atm.
OK .. here are the beta products.
Max_Midi_oIP_Thru should be run from Max or compiled as a standalone device. It's a very simple and lightweight device. The user chooses the network port the device should "listen on," the user selects the midi port the data will be forwarded to, and the device repeats whatever comes through the network port as midi bytes to the midi port.
I removed the "iter" object as upon closer investigation of the midiout object, I learned that it can take a list and parse that as individual bytes on its own. I was unable to figure out how to specify what channel it should midi-thru stuff on, and since I don't have an actual copy of Max, I can't know exactly how the midi port drop down populates. In theory though, I think the logic is sound; the midi channel may just be defaulting to 1 or something.
****
Alpha_Juno_SysEx_oIP is the MaxForLive device that transmits the SysEx to the standalone device. It takes user input for every control on the Alpha Juno and transmits it as a packaged sysex message, a list of ints, to whatever port the user chooses within the local machine. I took inspiration from the original Alpha Juno controller, the PG 300... not a potentiometer in sight :)
****
So for those that have Max, MaxForLive, and an Alpha Juno, I hope this thing works correctly! I shall wait patiently for a kind soul to compile Max_Midi_oIP_Thru into a standalone executable to be able test it out as well :) From here the challenge is figuring out how to select midi channel when using the midiout object, updating the standalone to allow multiple devices to be used concurrently, and creating more templates for other midi-sysex devices. I am open to any and all critique/feedback of both devices, so I look forward to hearing from interested parties!
EDIT - I made a small correction (I forgot to change the key for the VCA envelope type to "Gate"). I updated this and re-uploaded the file thinking it would overwrite what was already uploaded. It didn't, so now there are two files, the newest one is Alpha_Juno_SysEx_oIP1. Functionally, both files are the same.
throwing a little bump out there :)
Ok here's a quick (naive) implementation of how this would work. With the alpha Juno. Just checked and all the editing controls seem to work.
Unfortunately after looking at the midi spec the alpha juno doesn't have a device id, at lease one that follows the manufacturer id, which somewhat spoils the idea of having the sysex message identify what kind of device is sending sysex easily I was hoping for.
I've made a few changes to your m4L interface on this basis, and am instead sending the message /sysex/[manufacturer]/[device].
I'm leaning more to handling all the logic in max when it comes to handling preset dumps and updating on patch changes etc. I'm thinking about something like having m4L send and receive /sysex/[manufacturer]/[device/param [control] [value] and /sysex/[manufacturer]/[device/dump [list...] for presets.
But if people want to handle all the midi messaging in m4L, well that's very easy. I've included 'simple' versions of the patches for that.
Thank you for checking out the work and offering your insight. I'll be checking out the edited patch tonight or maybe tomorrow night depending on the time I have available.
I like your idea of using OSC to break up the sysex code. I considered leaving OSC out of the picture due to the extra memory, cpu cycle requirements, and compatibility issues across different OSes. While there is a benefit to using OSC for preset dumps, it is important to note that for real time patch editing, a parameter will start out as a control change, then it will get packaged to an OSC message, and then it will get parsed to sysex. I felt going straight from CC to sysex would be less bloat, but maybe I'm being overly paranoid about overall performance.
I am still concerned though that after the first two sysex bytes ("sysex start" and manufacturer id), it's essentially an anything goes situation between different hardware, as you've found. Never-the-less, we can work around that. If you look at the differences in sysex between the 106 and alphas (http://www.vintagesynth.com/roland/sysex.php), you'll see that the third byte is still different between the two synth engines. We can utilize it to id our devices, even if the manufacturer didn't intend that third byte to denote the device id. While I was reviewing the sysex implementation in the alpha juno for the device id being missing, I also figured out where the midi channel is specified, so this can be properly implemented now :)
I'll check out what you've put together and recap again soon. Cheers and, again, thank you very much for your time and efforts.
Cool, yeah I see the merits of both approaches. I was just looking ahead to maybe automatically identifying the sysex message format and translating it. But simplicity has some definite strengths.
Check out the simple versions, it's just an osc to to midi converter than has no expectations for messages (other than maybe 240 ... 255 to be added). It's closer to your vision I think. I'll look into update management and presets for that when I get a chance
I finally got some time to go over this, and I gave the non-simple version some thought. Attached are the results.
I implemented midi channel selection to the alpha juno template. Then, since you added the osc address to the maxforlive device, and you mentioned you wanted to keep the logic in the midi manager, I took out the "global" sysex out of the messages being sent out of the maxforlive device (ie the alphajuno template).
I edited the midimanager to make it "intelligent" so that it would re-add the correct sysex based on the osc address. I left everything un-abstracted for the sake of visibility / simplicity, but eventually there would be an abstraction for every sysex midi device that the manager supports.
I still only have MaxForLive, so I still cannot test out the work that has been done :( By the size of the patcher windows, I take it you're on a mac... is there any way for you to compile the midimanager into an executable for windows?
bump?
Please vote for Ableton to support sysex on Centercode - this is where Ableton support directed me to. You need to create an account to access it but the developers do monitor requests here and the number of votes they get so it is definitely worth it if you want this feature :)
I joined centercode just to upvote this - the SysEx issue has been bothering me for over five years, and here it is bothering me yet again on a new project.
Unfortunately, even now I've created an account and signed in, I can't find any such pages on the site. Could it be that they got deleted? :-(
Hi everyone,
I am not a frequent M4L user but used it occasionally for some stuff that Ableton alone cannot do.
I have noticed that you Brooke have managed to create something I need and would gladly pay for. Is it possible to have your ESQ-1 patch as I recently started using my ESQ again and would love to control it from Ableton.
Thanks !
Hi Everyone,
als i'm a newbie and would love to have a editor controlling my juno alpha 2. I see above al sort of patches but seem not able to get them working. Can somebody please help. Love to control my Juno!!