tangent

I am evaluating Max 6 for a real-time midi and audio processing project.

I am currently doing the 30-day trial on Windows 7 64-bit, i7-2600K, 16GB Ram, and I have been concerned by some of the performance issues I'm seeing in the early stages of the exploration, particular concerning midi timing. Is anyone doing real-time midi processing w/ accurate timing on Win7?

I would like to determine if the problems I'm seeing are:

c) Max 6 vs Max 5 (Is there are way to eval 5?)

I'm hoping that the staff or community here can help point me in the right direction. I'll explain an example case of the timing problems:

I wrote a small abstraction for controlling the Novation Launchpad. It uses a few tables to track the LED states, button Up/Down states, and periodically renders the overlays to the Launchpad by writing a series of midi messages to the Launchpad midi port (a USB device).

Using this abstraction I built a number of test patchers, and I noticed that when its running, the UI gets "laggy" and bang timing has significant (50-200ms) glitches. I wrote a quick step sequencer and if I am displaying to the launchpad the seq timing goes off audibly and is jerky (timer on the tempo obj jumps around by 50-100 ms). If I replace the "midiport b" obj in the abstraction with a "print" the problem goes away, so it seems to be the midi side causing the issues.

The data to the midi port is ~120 bytes every 100-200 ms. I just can't imagine that this should be enough overhead to already see timing glitches. At no time does my CPU usage crack maybe 10% btw, so it doesn't seem to be on that side...

I'm concerned because our hope was to have 5+ midi ports running at decent rates, along with several streams of real-time audio processing.

I've seen a range of stability issues as well - I'm in day 4 of the eval and I've probably had 5-10 crashes. Lost some work, reminded to save more often. ;)

Max 6 seems to be an amazing program and I am very excited by the possibility if I can work all the kinks out.

a) Win7 64 vs 32-bit?
b) Win7 vs. Mac OSX
c) Max 6 vs Max 5 (Is there are way to eval 5?)
d) Driver specific issues (midi side?)
e) I'm doing something wrong...

evaluating-max-6-questions-about-midi-timing-and-overall-stabilityperformance

I did try Overdrive, however this seemed to have a strange effect on the order of message processing. For example, in my little sequencer there is a counter that outputs addresses for table lookup - In non-overdrive this works as expected, the counter is stored and used for a lookup each counter value. When I engage overdrive, the stored value is always the last value of the counter, as if a portion of the msgs were processed out of order. Should overdrive change the order of operations within max?

I don't know if this is a clue that I'm doing something wrong, but in any case, overdrive seems to make the patch break completely. It also completely disables the UI (which I think is expected), and therefore seems of limited practical value (I guess mainly its for when using external control and no on-screen interactions?)

I did try Overdrive, however this seemed to have a strange effect on the order of message processing. For example, in my little sequencer there is a counter that outputs addresses for table lookup - In non-overdrive this works as expected, the counter is stored and used for a lookup each counter value.  When I engage overdrive, the stored value is always the last value of the counter, as if a portion of the msgs were processed out of order. Should overdrive change the order of operations within max?

I don't know if this is a clue that I'm doing something wrong, but in any case, overdrive seems to make the patch break completely.  It also completely disables the UI (which I think is expected), and therefore seems of limited practical value (I guess mainly its for when using external control and no on-screen interactions?)

The following is rather generalised, as I am on OSX and haven't experienced such issues personally, and it may even be that it may turn out to be a Max Win 7 specific issue, but with that said I would suggest you try and localise the issue to reduce the possibilities in your list: (Win 7? Midi Audio/IO hardware/drivers, etc.).

Is the MIDI timing and general performance with your current hardware stable with another MIDI app? Is there any chance to try your current setup out with different MIDI hardware? Having done some work with other cross platform software there does seem to be some reported variability in Win7 timing that may have something to do with USB MIDI hardware.

The following is rather generalised, as I am on OSX and haven't experienced such issues personally, and it may even be that it may turn out to be a Max Win 7 specific issue, but with that said  I would suggest you try and localise the issue to reduce the possibilities in your list: (Win 7? Midi Audio/IO hardware/drivers, etc.). 

Is the MIDI timing and general performance with your current hardware stable with another MIDI app?  Is there any chance to try your current setup out with different MIDI hardware? Having done some work with other cross platform software there does seem to be some reported variability in Win7 timing that may have something to do with  USB MIDI hardware.


It would be helpful if u could post your patch for inspection....

RE: from our favorite desert sauce vanille béchamel : "....if your patch is not order-proof (tips and tricks : ALWAYS use [trigger])."

The point not always made clear to max learners is that if you have a

which have any order dependencies between it's individual statements, and

you wish to implement it correctly and repeatably in Maxmspjitter, use [trigger] to *enforce execution order*.

It's like the "PROG" function in Lisp, or the body of a do block in several languages...

And beyond execution order, if you need to have synchronized input of sets of data from several different processes/execution chains/"patchcords", (like to make sure all the args are there and ready to send on to the multiple required inputs of an object like makenote....) then *do* check out [buddy].

(and [uzi] can b used to imitate for loop w/ index, etc. etc.)

Perhaps a few more toots could be written about adapting programming practice in going from declarative or functional languages to dataflow style programming...?

RE:  from our favorite desert sauce vanille béchamel : "....if your patch is not order-proof (tips and tricks : ALWAYS use [trigger])."

The point not always made clear to max learners is that if you have a
process/'algorithm'/set of instructions
which have any order dependencies between it's individual statements, and
you wish to implement it correctly and repeatably in Maxmspjitter, use [trigger] to *enforce execution order*.
It's like the "PROG" function in Lisp, or the body of a do block in several languages...
And beyond execution order, if you need to have synchronized input of sets of data from several different processes/execution chains/"patchcords", (like to make sure all the args are there and ready to send on to the multiple required inputs of an object like makenote....) then *do* check out [buddy].
(and [uzi] can b used to imitate for loop w/ index, etc. etc.)
Perhaps a few more toots could be written about adapting programming practice in going from declarative or functional languages to dataflow style programming...?

Hey, with the amount of CC data u could easily generate, r u close to the max driver speed for your Win7 usb midi drivers ("burst" rate: your cpu/max will crank all the data out for lp real fast...)?: Max maybe providing the data fast enough, but bad usb midi driver could be choking on the stream?, although at this late date/os, I hope not. I've seen it happen before, tho...but never having plugged my launchpad into a windows box, of any generation os, i canna truly say.

some usb midi can turn out being an issue. i´m still stuck with pci/pcmcia and xp. usb has not such high speed access (not transfer rate)-priority for the os and possibly it´s even worse on windows 7. there is so many people out there complaining about usb audio/midi-interfaces, before they change to pci/pmcia or, because there are just a very few pcmcia-interfaces available, to mac. i even tried an old atari for midi, but it is too old for me.

however, on bigger projects maybe do audio and midi on seperate machines with one as clock.

Apologies for hijacking this thread, but does anyone know of specific MIDI interfaces/drivers to avoid on a windows 7 os (in conjunction with max, version 5 btw)??? And any to recommend?

Apologies for hijacking this thread, but does anyone know of specific MIDI interfaces/drivers to avoid on a windows 7 os (in conjunction with max, version 5 btw)??? And any to recommend?
thanks
T


all this is speculation till we get to see tangent's patch... (see max forum instructions: 

https://cycling74.com/forums/welcome-to-the-cycling-74-forums-take-a-moment-to-read-these-instructions)

all this is speculation till we get to see tangent's patch... (see max forum instructions: https://cycling74.com/forums/welcome-to-the-cycling-74-forums-take-a-moment-to-read-these-instructions)


We evaluated PureData first (which we're no longer considering on Win7 due to abysmal audio latency and midi performance), but I spent some time there and learned to mind my "t b b i"s.

I feel like I have a decent (starter) understanding of msg order processing, etc. I have read a good chunk of the help and tutorials, as well a chunk of an oft recommended book on Max (can't remember the name). I believe in RTFM (I even read the forum instructions before I posted ;) btw dtr, your links busted). I didn't post a patch initially because I hadn't yet isolated it to one area, but was looking for more general advice on stable configurations.

I have isolated the Overdrive behavior I'm seeing - I'm curious about Overdrive because we don't need UI interaction during actual live performance, so if it made midi timing better, that would be a workable option.

This patch is a cut-out from a portion of the test step sequencer. The crux of the overdrive issue appears around the use of "send xyz $1" to do a table lookup. This may well be a case where "I'm doing something wrong" ;)

If you load this patch, open "table grid" and make all the values non-zero. Start the tempo obj and you will see it print the active indexes for each column as selected by the tempo value 0-7 (if tbl data is non-zero). In non-overdrive this works fine for me. When I engage overdrive, all the indexes become 63s, as if all the outputs from the "counter" obj were processed before the send/receive. I hope that's enough explanation/makes sense.

Is send/receive used as a table look-up meant to be synchronous or asynchronous? It seems to behave one way w/o Overdrive and another with it... I had assumed that synchronous operation is what makes it most useful. In this contrived example I could just route the table out and use a normal look-up, but in real-world use it seems one needs to use "send xyz" because otherwise you have all the look-ups coming out the same outlet. Am I missing something here about how to use tables efficiently?

I've made some improvements to the Launchpad midi issue. I moved from constantly rendering the display (which was sending about 120 bytes every 120ms) to only sending a frame when the data changes. This created a little improvement in overall timing.

My next experiment is to look if I can throttle the data going to the midiout object in some way to improve performance. I chose the rendering interval of 120ms based on Novation documentation (states it can update the double buffer frame in 100ms), but I'm wondering if I'm somehow still filling the driver buffer such that there is some type of blocking. It weird because this just isn't that much data.

I will also try outputting the midi data to a different device, to see if the problem seems driver specific.

If it ends up being a driver issue, maybe we can send raw midi data out a TCP socket or virtual COM port to another process queuing data to the midi drivers. Max seems to have no problem w/ the timing when I'm not tied to the midiout.

I do think I have much more to learn about internals in Max. Is there a good resource for in-depth aspects like threading effects, synchronous/async behaviors, etc? The in-app docs are pretty good, but I can't seem to find much on the internals.

One example where I feel like there is priority issues I don't understand: my launchpad abstraction takes midi note in data to detect key presses. Outside the abstraction the tempo object is outputting timing that drives the steps, but if I repeatedly send note on/off the tempo processing seems to halt until I stop. Is there any threading at work in Max? Are there technique I need to consider to make things more responsive? I must be doing something wrong. Once I get things cleaned up a bit further I'll post the launchpad patch if problems persist.

I know I've got lots to learn - thanks again for advice. I haven't been this excited about learning a new program/technology in quite some time!

I feel like I have a decent (starter) understanding of msg order processing, etc.  I have read a good chunk of the help and tutorials, as well a chunk of an oft recommended book on Max (can't remember the name). I believe in RTFM (I even read the forum instructions before I posted ;) btw dtr, your links busted). I didn't post a patch initially because I hadn't yet isolated it to one area, but was looking for more general advice on stable configurations.

This patch is a cut-out from a portion of the test step sequencer.  The crux of the overdrive issue appears around the use of "send xyz $1" to do a table lookup. This may well be a case where "I'm doing something wrong" ;)

If you load this patch, open "table grid" and make all the values non-zero.  Start the tempo obj and you will see it print the active indexes for each column as selected by the tempo value 0-7 (if tbl data is non-zero).  In non-overdrive this works fine for me.  When I engage overdrive, all the indexes become 63s, as if all the outputs from the "counter" obj were processed before the send/receive.  I hope that's enough explanation/makes sense.

I've made some improvements to the Launchpad midi issue.  I moved from constantly rendering the display (which was sending about 120 bytes every 120ms) to only sending a frame when the data changes. This created a little improvement in overall timing. 

My next experiment is to look if I can throttle the data going to the midiout object in some way to improve performance. I chose the rendering interval of 120ms based on Novation documentation (states it can update the double buffer frame in 100ms), but I'm wondering if I'm somehow still filling the driver buffer such that there is some type of blocking.  It weird because this just isn't that much data.

One example where I feel like there is priority issues I don't understand: my launchpad abstraction takes midi note in data to detect key presses. Outside the abstraction the tempo object is outputting timing that drives the steps, but if I repeatedly send note on/off the tempo processing seems to halt until I stop. Is there any threading at work in Max? Are there technique I need to consider to make things more responsive? I must be doing something wrong.  Once I get things cleaned up a bit further I'll post the launchpad patch if problems persist.

There are some issues in the posted patch...I think the [uzi] and [counter] are causing the problems. It's not totally clear what you want to do with the numbers, but there are a lot of them flying around :) :) hey, by your post you are already an "expert beginner Maxer", and we appreciate that you've done your homework and want to dive in!

If you're looking to just grab all the Y values across an X domain of 0-63, and do it once every 8 pulses (with the [% 8] etc.), you can eliminate several objects in the patch. This isn't just trivial cleanup---the [uzi] and [counter] are doing a ton of extra legwork which isn't needed, and I bet these are causing the bottlenecking...especially [uzi] which kind of does its own thing regardless of what other objects are playing at the time. See if the patch below does the lookup that you want. [print] will also eventually cause lag and memory consumption if you send it thousands of things too.

Also, for the [send] issue, I think that can be a problem in some cases, but I don't think it would be in this one. You can always use a [prepend] and [pack] to create two-element lists to send around, which can then use [route] at the other end to grab by index.

Anyway, great questions and trust that these timing etc. issues can (most likely) be cleared up. Most likely the only problems which might not be solvable would be due to hardware and bad drivers.

Hopefully the patch below does what you're looking for, if not, just clarify and we can get you on the right track so that it does what you need as optimized as possible!

There are some issues in the posted patch...I think the [uzi] and [counter] are causing the problems. It's not totally clear what you want to do with the numbers, but there are a lot of them flying around  :)  :)  hey, by your post you are already an "expert beginner Maxer", and we appreciate that you've done your homework and want to dive in!

Hopefully the patch below does what you're looking for, if not, just clarify and we can get you on the right track so that it does what you need as optimized as possible!


Thanks for the input. Sorry if I've created the confusion of having what appears to be two distinct issues - one is "overdrive changes msg order", the other is unrelated midi timing problems tied (I think) to the Launchpad midi driver. There weren't really any timing problems in the patch I posted - that's the overdrive issue.

The snippet is taken out of context, but the counter values are also used elsewhere in the patch - they are feeding a look-up for each table entry value. I did see elsewhere recently that one can use uzi alone to generate counts, as you show in your snippet, rather than counter and uzi.

I was trying to remove as much as possible to demonstrate that Overdrive changes the order of msg processing. Does it do that on your setup? Are you running win7, mac, 32/64, etc? On my setup the print output changes in overdrive or non-overdrive mode. There is no timing problems in this snippet.

In your snippet, everything behaves the same in and out of overdrive, but it does not use "send" to do the lookup. Your lookup feeds the index into the inlet and takes the result from the outlet. What if I need (as I did) to lookup from the same table in 4 different parts of a patch? Now the resultant values all come from the same outlet - do I add extra logic to switch in advance route them to each destination? I assumed this was the purpose of the "send xyz" table message (to route lookups to diff parts of a patch), except that it appears to behave different based on overdrive.

I need to think more about using "prepend" and "pack" to route from the outlet as you mention, but its not clear to me how this would work - If I do a lookup by sending the index to the inlet, I get the stored value out of the outlet. At this point, even if I prepend/pack/etc, I do not have any stored notion of the needed receiver. I guess I could set the desired receiver before sending the lookup - I remember seeing some object that can be set to a receiver in one inlet, and send to it in the other (maybe just send can). Again, I just thought the "send" table command was supposed to work around needing that scaffolding.

I expect there are many areas where I can improve efficiency - I just figured I had enough "overhead" on this setup that updating LEDs on one device wouldn't even scratch the surface. We are certainly hoping to do much more...

One area where I've struggled to find efficient solutions is copying/iterating tables - is there anything more efficient than load/dump/normal between two tables? The table copies make up the bulk of the rendering logic - I need to copy a few tables directly, and iterate a few to selectively copy non-zero values. The only approaches I've found are load/dump/normal or using a counter/uzi setup to iterate the table and use some logic to take actions for each value.

I did also make some progress on the midi timing side of the problem. I tried a few things that didn't help (defer, deferlow, speedlim) then discovered at least what appears to the be the culprit area. I am taking midi input via notein and ctlin: I was triggering a rendering cycle on each key up or down to set a color for key down. This looks nice, but appears to be the root of the problem where repeated key presses delays the tempo object. If I disable this aspect timing is more stable.

I am cleaning up the full launchpad abstraction now, then I'll post it for comments or if anyone wants to try it out. I'm hoping if I can get any kinks worked out that it might also be useful for others (w/ a launchpad at least).

Thanks for the input. Sorry if I've created the confusion of having what appears to be two distinct issues - one is "overdrive changes msg order", the other is unrelated midi timing problems tied (I think) to the Launchpad midi driver.  There weren't really any timing problems in the patch I posted - that's the overdrive issue.

I was trying to remove as much as possible to demonstrate that Overdrive changes the order of msg processing.  Does it do that on your setup? Are you running win7, mac, 32/64, etc?  On my setup the print output changes in overdrive or non-overdrive mode. There is no timing problems in this snippet.

In your snippet, everything behaves the same in and out of overdrive, but it does not use "send" to do the lookup.  Your lookup feeds the index into the inlet and takes the result from the outlet. What if I need (as I did) to lookup from the same table in 4 different parts of a patch? Now the resultant values all come from the same outlet - do I add extra logic to switch in advance route them to each destination? I assumed this was the purpose of the "send xyz" table message (to route lookups to diff parts of a patch), except that it appears to behave different based on overdrive.

I expect there are many areas where I can improve efficiency - I just figured I had enough "overhead" on this setup that updating LEDs on one device wouldn't even scratch the surface. We are certainly hoping to do much more... 

One area where I've struggled to find efficient solutions is copying/iterating tables - is there anything more efficient than load/dump/normal between two tables?  The table copies make up the bulk of the rendering logic - I need to copy a few tables directly, and iterate a few to selectively copy non-zero values.  The only approaches I've found are load/dump/normal or using a counter/uzi setup to iterate the table and use some logic to take actions for each value.  

I did also make some progress on the midi timing side of the problem. I tried a few things that didn't help (defer, deferlow, speedlim) then discovered at least what appears to the be the culprit area.  I am taking midi input via notein and ctlin: I was triggering a rendering cycle on each key up or down to set a color for key down.  This looks nice, but appears to be the root of the problem where repeated key presses delays the tempo object.  If I disable this aspect timing is more stable.

I am cleaning up the full launchpad abstraction now, then I'll post it for comments or if anyone wants to try it out. I'm hoping if I can get any kinks worked out that it might also be useful for others (w/ a launchpad at least).


https://cycling74.com/articles/event-priority-in-max-scheduler-vs-queue/

Thanks Chris - This is exactly the kind of internal info I was seeking. I have a feeling I will return for a re-read a few times!

I was surprised to see that everything is handled in two threads. Is that still true (the article is 2004) - I wonder how that impact Max's ability to leverage multi-core processors?

Thinking further about the comments here, and also the information in this article, I think my mistake is that I am allowing too much processing to be triggered from a single "event". The logic in my launchpad abstraction is all triggered from a single "render" bang, and I think that means that the problem is that if this sequence is triggered from the high priority thread, the entire sequence must be completed (possibly multiple times if triggered repeatedly) before processing other competing events (like pending tempo/metro ticks). This explains the "freeze" behavior I am seeing from tempo, and suggests that repeated note up/down events are consistently being treated as higher priority than the pending tempo event.

I tried using defer/deferlow to defer rendering of the note up/down but this seemed to only make a _tiny_ difference (maybe a little less laggy on repeated key presses). I am wondering if I need to play with these a bit more to force all the rendering action onto the low prio queue, as well as try to break up the rendering into steps using delay/pipe.

Thanks for the article - off to try delay/pipe to break up the rendering events...

Thinking further about the comments here, and also the information in this article, I think my mistake is that I am allowing too much processing to be triggered from a single "event". The logic in my launchpad abstraction is all triggered from a single "render" bang, and I think that means that the problem is that if this sequence is triggered from the high priority thread, the entire sequence must be completed (possibly multiple times if triggered repeatedly) before processing other competing events (like pending tempo/metro ticks).  This explains the "freeze" behavior I am seeing from tempo, and suggests that repeated note up/down events are consistently being treated as higher priority than the pending tempo event.

I tried using defer/deferlow to defer rendering of the note up/down but this seemed to only make a _tiny_ difference (maybe a little less laggy on repeated key presses).  I am wondering if I need to play with these a bit more to force all the rendering action onto the low prio queue, as well as try to break up the rendering into steps using delay/pipe.

Thanks for the article - off to try delay/pipe to break up the rendering events...


yes james is right, the event scheduling is something on its own, especiall when using [uzi] or a recursive [counter]-loop inside [uzi]. [uzi] is great and i use it really often. but you have to know that gun. call it a mess or extra potential, sometimes what you logically see breaks from what it iternally does.

using [pipe] or [delay] rarely is suited for good timing, but it´s great to correct the dark side of the scheduler.

generally try to find ways to work around and work with tidyness, like james showed up. a counter to count uzi bangs makes no sense since uzi counts by itself. [table]/[coll]-dumps or doing much with the help of lists (the [zl] is wonderful) are essential. dumping => list-filtering/manipulation ([zl group], [zl slice/ecils], [zl rev], [zl rot], [zl nth] etc.]) => routing => unpacking. that manages huge masses of data really quick.

I did pause when I saw the 'uzi' outlet comment "Watch Out!" and wondered if I was overusing them... ;)

I'll have to spend some time looking at zl. I read over the ref page once but didn't realize it had so many options. I felt like I was doing table stuff "the hard way" and I bet that's got some useful bits.

Regarding Midi Timing - I've done some new experiments and it does seem to be specific to the Novation Launchpad USB driver. I can switch the midiout device between the Launchpad and another midi out device and immediately see a drastic impact on timing.

For example, I took the little demo sound player that is in the help pages, selected osc, noise, pulse, 35ms. If I run my renderer going to another midi device, I get a fast tick every 34ms - looks fine. If I switch to the launchpad the rate drops to a quarter the old rate immediately. Something with the launchpad driver (maybe again some 32/64 thing, or something) is causing a huge delay when reading/writing within Max. I just picked up the device so I can't comment if its see elsewhere...

So, overall good news for our Max eval. We're not planning to use the launch pad as a key part of the real-time project, so that alleviates some of my concerns. We're off to do some more experiments w/ real-time audio + midi now, so we'll see how it goes...

I will continue to mess w/ the launchpad and if anyone is interested in the mgr abstraction for it, let me know. I might try putting the midi I/O and overlay logic in a little C app and pipe the data over from max somehow...

Thanks for everyone's help - so far I think I'll be a full-fledged Max owner in about "25 Days". Cheers!

Regarding Midi Timing - I've done some new experiments and it does seem to be specific to the Novation Launchpad USB driver.  I can switch the midiout device between the Launchpad and another midi out device and immediately see a drastic impact on timing.

For example, I took the little demo sound player that is in the help pages, selected osc, noise, pulse, 35ms.  If I run my renderer going to another midi device, I get a fast tick every 34ms - looks fine.  If I switch to the launchpad the rate drops to a quarter the old rate immediately.  Something with the launchpad driver (maybe again some 32/64 thing, or something) is causing a huge delay when reading/writing within Max.  I just picked up the device so I can't comment if its see elsewhere...

So, overall good news for our Max eval. We're not planning to use the launch pad as a key part of the real-time project, so that alleviates some of my concerns.  We're off to do some more experiments w/ real-time audio + midi now, so we'll see how it goes... 

Thanks for everyone's help - so far I think I'll be a full-fledged Max owner in about "25 Days". Cheers!


regarding zl, i made a variation using lookup/group. it doesn´t make so much sense in this example, but it´s good to know zl.

Hi All - Thought I'd follow-up since so many people helped out...

I rewrote the launchpad mgr abstraction using many of the ideas people put forward, and I think improved it quite a bit. I continue to see slow midi I/O w/ the Launchpad USB driver, however by fixing my test seq patch's issue w/ Overdrive, I am now able to use overdrive to work around that problem.

In the abstraction, I now use "defer" to force the rendering portion of the operation to the lower priority thread in Overdrive mode. This works beautifully and I can bang on the Lanuchpad all day w/o any timing glitches (and the launchpad UI still looks/feels very responsive). I also added a "speedlim 100" on the main render bang, which ensures that it doesn't try to render too rapidly and avoids overloading the USB driver. I think this was also key.

With overdrive + defer the rendering operation and subsequent midi output to the problem driver occur on the low prio thread. The high prio thread is still active to process tempo events or other high prio items.

It does seem to be true that Overdrive is changing expected msg order as in the patch I posted above. We tested this patch on OSX w/ Max 6 as well, and it has the same behavior.

Reflecting on Chris' link on internals, I think it makes sense that a "send" causes a msg to be deferred (equivalent to defer but not deferlow, which sends to back of the low prio queue). In non-overdrive mode, Max is single-thread (if the '04 article is still accurate), so it makes sense that one would not see this reordering during send in that case, since defer only changes high to low prio (no effect in non-overdrive).

This means the "send xyz $1" approach with tables is not very safe in terms of ordering. I reworked all of my table look-ups to use "expr" instead of the send method to avoid this problem. The "expr" $s2[$i1] syntax to deref a table is exactly what I needed for table look-ups and expr has made my code much more compact.

If anyone has a launchpad and would like to test out the abstraction, let me know. It remaps input into 8x8 and 8x2 (for top/right ctrl buttons), and displays up to 3 OR'ed overlay layers of LED values to display bkgnd + active + cursor layer or whatever, and auto xor's a key up/down layer on top. Control of the LED value for each layer is handled using table refers, so you just drop the abstraction and however many tables you need, set the refs, and its running.

Anyway, thanks for the assistance. I am very excited about Max and hope to be around here a bit more often.

I rewrote the launchpad mgr abstraction using many of the ideas people put forward, and I think improved it quite a bit.  I continue to see slow midi I/O w/ the Launchpad USB driver, however by fixing my test seq patch's issue w/ Overdrive, I am now able to use overdrive to work around that problem.

In the abstraction, I now use "defer" to force the rendering portion of the operation to the lower priority thread in Overdrive mode.  This works beautifully and I can bang on the Lanuchpad all day w/o any timing glitches (and the launchpad UI still looks/feels very responsive). I also added a "speedlim 100" on the main render bang, which ensures that it doesn't try to render too rapidly and avoids overloading the USB driver. I think this was also key.

It does seem to be true that Overdrive is changing expected msg order as in the patch I posted above. We tested this patch on OSX w/ Max 6 as well, and it has the same behavior.  

This means the "send xyz $1" approach with tables is not very safe in terms of ordering. I reworked all of my table look-ups to use "expr" instead of the send method to avoid this problem.  The "expr" $s2[$i1] syntax to deref a table is exactly what I needed for table look-ups and expr has made my code much more compact.

If anyone has a launchpad and would like to test out the abstraction, let me know.  It remaps input into 8x8 and 8x2 (for top/right ctrl buttons), and displays up to 3 OR'ed overlay layers of LED values to display bkgnd + active + cursor layer or whatever, and auto xor's a key up/down layer on top.  Control of the LED value for each layer is handled using table refers, so you just drop the abstraction and however many tables you need, set the refs, and its running.

Glad things are working better for you. On the table lookup thing, why aren't you just hitting it with an address, and using the result?

Because I am using the lookups and their results in multiple places in the patch. If I passed in the address I'd need to route the output to multiple places and have a way to distinguish which destination was appropriate for each case. Does that make sense?

By using expr I can also lookup from multiple tables and combine the result, for example the final expr for calculating the value of a LED state is:

expr (($i1 | $s3[$i2] | $s4[$i2] ^ $s5[$i2]) & 51)

which gives me three table lookups and masking in one expr and saved me lots of unnecessary objs/wiring. :)

Because I am using the lookups and their results in multiple places in the patch.  If I passed in the address I'd need to route the output to multiple places and have a way to distinguish which destination was appropriate for each case. Does that make sense?

which gives me three table lookups and masking in one expr and saved me lots of unnecessary objs/wiring. :)


This thread is great. It's obvious you have a lot of experience in other programming areas and it's very interesting to hear how you're approaching things in the Max environment. Great questions, experimentation, and reporting on the results...I'm learning a ton by following all this :)

And I trust you're seeing that Max can do what you need...even though with some projects, you might need to take time for setup and fiddling, with that kind of flexibility comes a lot of power. So when it's all worked out, it totally rocks!

This thread is great. It's obvious you have a lot of experience in other programming areas and it's very interesting to hear how you're approaching things in the Max environment. Great questions, experimentation, and reporting on the results...I'm learning a ton by following all this  :)

And I trust you're seeing that Max can do what you need...even though with some projects, you might need to take time for setup and fiddling, with that kind of flexibility comes a lot of power. So when it's all worked out, it totally rocks!


Uzi can be a pain in overdrive - its "too fast" and can break message order. In your first patch, it works as expected in overdrive when you trigger the uzi with a button (=low priotity ui event) but breaks when you trigger it from tempo. A single defer in front of uzi fixes it. This is indeed the dark side of the scheduler ...

Regarding the launchpad, I don't have one, but I've seen reports on problems with speed, especially when updating the leds, see for example here 

http://createdigitalmusic.com/2009/11/novation-releases-all-midi-details-for-launchpad/

Uzi can be a pain in overdrive - its "too fast" and can break message order. In your first patch, it works as expected in overdrive when you trigger the uzi with a button (=low priotity ui event) but breaks when you trigger it from tempo. A single defer in front of uzi fixes it. This is indeed the dark side of the scheduler ...
Regarding the launchpad, I don't have one, but I've seen reports on problems with speed, especially when updating the leds, see for example here http://createdigitalmusic.com/2009/11/novation-releases-all-midi-details-for-launchpad/


"which gives me three table lookups and masking in one expr and saved me lots of unnecessary objs/wiring."

yeah expr really invites to such excesses, you´re funny. sometimes expr is more like a subpatcher than an object. but if one has to read such code a few months later it takes some time to decypher it again.

about fixing [uzi]-stuff with [deferlow]. i´m no expert and willing to learn, but when this strange behaviour of the scheduler became obvious to me i tried some stuff and did measurements. finally using deferlow (despite its coolness factor) always and with increasing patchsize was more critical then a simple delay. it did work, yes, but it was slower and variable (depending on the processing queue of course) in tempo. a [delay] of short 5ms always fixed the same issue much quicker and on a stable timebase.

as far as my experiences go. it may be different in that specific launchpad issue, where it might make sense to really put down the midi-io-bottlenecking.

about fixing [uzi]-stuff with [deferlow]. i´m no expert and willing to learn, but when this strange behaviour of the scheduler became obvious to me i tried some stuff and did measurements. finally using deferlow (despite its coolness factor) always and with increasing patchsize was more critical then a simple delay. it did work, yes, but it was slower and variable (depending on the processing queue of course) in tempo. a [delay] of short 5ms always fixed the same issue much quicker and on a stable timebase. 

as far as my experiences go. it may be different in that specific launchpad issue, where it might make sense to really put down the midi-io-bottlenecking.


seejayjames - I don't mind some fiddling if the results are stable in the end. The possibilities with Max seem well worth the effort.

Max's reputation preceded it - Unfortunately I had always thought of Max as a "Mac" program and stuck in my cave, hadn't realized it was available for Win. The full-featured trial is a great practice - So far I'm very impressed and I've just scratched the surface.

grg - Its good to know that one can work around the "send xyz" table limitation by careful attention to threading and defer.

I have been happy overall with the launchpad, despite the driver issues. For the price (they can be had for $115ish now) it seems like a pretty good deal. No velocity sense is the only significant missing feature.

As the article and the LP docs point out, it can be slow if you try to update button by button, but this approach would be a programming nightmare anyway. The double buffered mode allows up to 100ms refresh rate, and that's writing every LED possible + swapping.

In practice I am seeing about 112ms/refresh in Max. Just under 10 fps is fine for most use cases I think - If I crank my little demo seq up to useless speeds (350bpm 16ths, lol) the audio stays stable and the launchpad shows some minor "tearing" and latency in the seq cursor/line LED updates, but any normal rate its been fluid and responsive. How it will hold up w/ more stuff going on in Max, I can't say (that's where driver issues might be a factor again).

Also to clarify for others considering the LP, that all this is about output to the device LEDs only. Note/ctrl input is not laggy at all, and my abstraction keeps the input at high prio going out, then defers only the rendering portion, so input is fine and fast...

The beauty of Max (or encapsulation in general) is that the abstraction now hides all this LP specific stuff nicely and I can just output color values to a few overlay grids and create any kind of custom interface I need.

xidance - I guess it depends on perspective. As someone w/ a C-centric background (mostly embsys and drvr stuff), the bitwise ops are straight-forward and seeing the C-style array indexing [] makes me even more at home... :) If you think that's excessive you should see the network of objects and message passing it replaced! ;) I also make liberal use of comments in everything I create, esp around "dense" clauses like that.

My only gripe is that "expr" doesn't support the (dreaded by some) C ternary op "?:" syntax. Being able to embed comparisons into exprs would be a huge boost. Then my expr would have looked EVEN UGLIER! ;p

The ORs are a hackish compromise that requires some thought in how to use the overlay grids. What I really wanted was something like:

expr (((($s4[$i2]!=0)?($s4[$i2]):((($s3[$i2]!=0)?($s3[$i2]):($i1))))) & 51)

How's THAT for excessive... ;) Ok maybe it crosses a line, but I'd include a nice comment! (for the record, that makes the overlays behave such that 0 is transparency, rather than ORing color values).

As the article and the LP docs point out, it can be slow if you try to update button by button, but this approach would be a programming nightmare anyway.  The double buffered mode allows up to 100ms refresh rate, and that's writing every LED possible + swapping. 

In practice I am seeing about 112ms/refresh in Max.  Just under 10 fps is fine for most use cases I think - If I crank my little demo seq up to useless speeds (350bpm 16ths, lol) the audio stays stable and the launchpad shows some minor "tearing" and latency in the seq cursor/line LED updates, but any normal rate its been fluid and responsive.  How it will hold up w/ more stuff going on in Max, I can't say (that's where driver issues might be a factor again).

Also to clarify for others considering the LP, that all this is about output to the device LEDs only.  Note/ctrl input is not laggy at all, and my abstraction keeps the input at high prio going out, then defers only the rendering portion, so input is fine and fast...

xidance - Re the deferlow stuff: I think your right that it would be problematic in certain contexts. The Max ref explains that deferlow moves the event to the _back_ of the low prio queue, which causes a (potentially drastic) change in msg processing order. defer just changes from high to low prio (although this can still impact ordering due to threading effects).

I am just using defer right now, although as you say, my case might actually be one where deferlow would be appropriate. I am only deferring the main "bang" that triggers the complete rendering cycle, so the process itself is reordered in time, but the individual substeps are not.

I think it is an ideal case because I want max to always treat UI rendering as the lowest possibly prio. If it needs to skip 5 updates to keep audio going or process a note ON, that's what I want, and defer+speedlim seems to do just that...

I think its probably important to be careful where one places a defer/deferlow to make sure only the correct subset of operations are deferred. Sprinkling deferlow around a patch for example, would be a mess due to msg reordering.

I am just using defer right now, although as you say, my case might actually be one where deferlow would be appropriate.  I am only deferring the main "bang" that triggers the complete rendering cycle, so the process itself is reordered in time, but the individual substeps are not.

I think its probably important to be careful where one places a defer/deferlow to make sure only the correct subset of operations are deferred. Sprinkling deferlow around a patch for example, would be a mess due to msg reordering.


FYI - If anyone is looking at this thread as it relates to the Novation Launchpad:

Novation has a set of max patch tutorials on their website. I didn't realize when I started this process, so I just worked off the programmers manual, but if you are looking for info on Max and LP, or hints on the midi side, its worth a glance:

http://www.novationmusic.com/download/899/

They do not implement double buffering, which is a pretty big omission, but it goes through the basics of the midi side. They show the use jit.matrix instead of tables to represent grid state. I haven't even started to look at jitter, so I can't say which is better, but presumably using a jit.matrix would make graphically relevant permutations easier and probably open up some cool possibilities. At some point I might rewrite my abstraction to use jit.matrix instead of tables, but for now on to other work...

Oh, and the Novation tutorials had a nice snippet for searching the "launchpad" device name using midiinfo to select the correct midi port automatically. I borrowed the idea for my abstraction and it works nicely...

They do not implement double buffering, which is a pretty big omission, but it goes through the basics of the midi side.  They show the use jit.matrix instead of tables to represent grid state. I haven't even started to look at jitter, so I can't say which is better, but presumably using a jit.matrix would make graphically relevant permutations easier and probably open up some cool possibilities. At some point I might rewrite my abstraction to use jit.matrix instead of tables, but for now on to other work...

Oh, and the Novation tutorials had a nice snippet for searching the "launchpad" device name using midiinfo to select the correct midi port automatically. I borrowed the idea for my abstraction and it works nicely...


"expr (((($s4[$i2]!=0)?($s4[$i2]):((($s3[$i2]!=0)?($s3[$i2]):($i1))))) & 51)"

sure thats what i meant where you will end up going this road ;)

I came to this late, so I didn't have time to read everything, but it seems to me that the issue with message order has not been fully addressed.

These are the posts you are looking for....

https://cycling74.com/forums/metro-doesnt-start-according-to-right-left-ordering

Hope this helps - I was going to try to explain it all again here, but my brain got all muddled....

Also, from what I briefly gather it seems like you are deferring stuff that you maybe shouldn't. Better to move everything up to the high priority thread earlier in the correct order. It may also interest you to know that my package of externals contains a general purpose object for scheduling in the high priority thread, along with tools for thread debugging and management... (www.alexanderjharker.co.uk/Software). I have local windows builds that I could send by email if you get in touch.

BTW - the thread is little long - and you probably need to read all of it to get handle on this stuff.

Hi Alex - Thanks for the helpful link, it was a good read and clarified a few areas. I'll drop you a line about the Win builds...

My thought is that the "defer" approach makes sense in this case is because I truly would like the LED rendering to be last/lowest priority vs. timing or audio.

The basic flow of the abstraction is notein/ctlin taking data from the LP. I believe in OD mode this comes in on a high priority thread, it gets mapped/written to a "Key Down" table, then mapped values are passed out an outlet (so client patch can handle key press on high prio thread), THEN a bang is sent first to "speedlim 100" (I never want to generate midi updates faster than this rate), then "defer", the render logic, then midi out to LP.

The hope (and what I seem to see) is that input from notein/ctlin is still processed by client patch at high prio thread, while rendering and midi output is deferred to the low prio thread. If I replace the "defer" with a "delay 1" this doesn't work and I can lag the client patch timing with repeated note on/off. The "defer" seems to fix this issue.

On a side note, a "dtr" post jogged my memory on solving the if/then in expr issue. Since boolean expressions are supported (==,!=), one can multiply each boolean exp output (0 or 1) by its "truth" value and sum the expressions, e.g. for the hypothetical "?:" form expression I mentioned above:

expr (((($s4[$i2]!=0)*($s4[$i2]))+(($s3[$i2]!=0)*($s3[$i2]))+((($s3[$i2]==0) && ($s4[$i2]==0))*($i1))) & 51)

Hi Alex - Thanks for the helpful link, it was a good read and clarified a few areas.  I'll drop you a line about the Win builds...

The hope (and what I seem to see) is that input from notein/ctlin is still processed by client patch at high prio thread, while rendering and midi output is deferred to the low prio thread.  If I replace the "defer" with a "delay 1" this doesn't work and I can lag the client patch timing with repeated note on/off. The "defer" seems to fix this issue.

Concerning Launchpad, did you see this thread :

https://cycling74.com/forums/launchpad-objects#post-171564

Concerning Launchpad, did you see this thread :
https://cycling74.com/forums/launchpad-objects#post-171564

Evaluating Max 6 - Questions about Midi Timing and Overall Stability/Performance