List processing troubles, too many calculations causing mistakes (I think?)
I have a patch that modifies a list of numbers and then feeds into a sequencer. There are a couple of parameters that have a tendency to cause the patch to output incorrect results. This occurs at points where the list gets broken down into smaller pieces and then reassembled. It happens most often with larger lists, like thousands of items, but it also happens with smaller lists too, just not as often so it's not as noticeable. I think there's probably too many calculations happening where the list gets reassembled and that's causing the mistakes? [speedlim] gets rid of the problem for the most part, but with large lists, it requires high values and it makes using the patch sluggish. Of course, I could limit the list to a more reasonable size, but I want to keep my options open. This is the part of patch that causes the most problems. It shuffles the list by n places, so for example if you shuffle by 2, instead of 1 2 3 4 5 you get 1 3 5 2 4.
I could barely get any mistake on my end, but I bet [thresh] is the culprit here.
Yep... using [thresh] for anything that is supposed to happen instantaneously is pretty much always a mistake. [list.group] and that middle [uzi] outlet were made for each other.
As an aside, I wonder if for large lists it would be better to do things using array objects.
Ok, yeah that works much better. I didn't know that about [thresh], I've been using it whenever I want to build a list.
I'm still getting mistakes when I swap [thresh] for [zl.group] in my main patch, but it's not as often. The problem is that it sometimes outputs the wrong size list. It wouldn't be a big deal, it usually lands on the correct result eventually, but it causes problems downstream when the list length changes. I think a combination of [zl.group], [speedlim] and [defer]/[deferlow] will minimize the problems and I think I can figure something out to avoid the downstream problems.
Although I don't quite know when to use [defer] or [deferlow], or which one to use. You used it your patch, TFL, but it seems like it performs the same without it to me. But in my main patch, [deferlow] seems to work better.
Tyler, I've wondered about arrays but I've only just started working with lists since I started building this sequencer so they're, uh, a bit beyond me at the moment.
defer will once more, just like tresh and speedlim, scramble the order of your messages.
all those tools are meant to "fix" timing/scheduling/threading issues, which means that in most "regular" patches they have no business beeing there.
while we are on "bad" objects... you should also get rid of that pipe.
and if you delete that s/r, the patch will be even be readable. (though you are using it correctly)
The [defer] in my previous post is totally overlooked, and I actually don't think it has any effect there since the whole process can be executed either on the main or scheduler thread, and will be done as such depending on how you are controlling all this in your main patch.
What worried me here is your [t 1 5000] and [pipe 10]. The later is a time-based object and its output (and subsequent operations) will be executed on the scheduler thread, while the 5000 for list generation will remain in the main thread, which is not a very good idea for two parts of the same "algorithm". You better want everything processed in the same thread.
[thresh] is also time-based and will make its output "prioritized" on the scheduler thread, which can be yet another source of confusion/bug.
By the way I forgot that [thresh] in the initial list-creation part, maybe that's the part causing issue, although it is not clear if in your main patch you rebuild that entire list from scratch every time or if you just scramble whatever list is coming in the [zl.reg] right inlet.
This has no [thresh], no [pipe], no [defer], and should just work:
And for the science here is a array-based version which, for some reason, performs worse than the zl version, by almost an order of magnitude (with the exception of [array.fill] which is 10 times faster than [Uzi] + [zl.group]):
The first [uzi]/[thresh] combo just builds the initial list, which gets stored in [zl.reg] and then modified from there. They don't do anything after the initial list gets built. And the [pipe] isn't in my main patch, I was just having problems initializing my share patch on load and put it there to force things to work. As it turns out, yeah the problems were caused by [thresh]. I don't have those problems in my main patch, but maybe I have enough mistakes in there to cancel each other out, I dunno. Can time-based objects cause issues even if they're not being used?
TFL - where do you get that [threadcheck] object? No such object exists for me, though it's familiar, I think from a video I watched. I've been reading/watching vids on the scheduler, threads, etc. since I figured that's where my problems might be coming from. I also don't have [array.fill], but I'm still on Max 8.
Roman - you don't want to see main patch then, so many sends/receives!
My bad for not precising this: [threadcheck] and [array.fill] are from Max 9.0.0.
Still trying to figure out some scheduler issues. I wonder under what circumstances a process might move from the main thread to the scheduler thread? Using an object like [defer], whose function directly relates to the scheduler, or timing objects like [pipe], [thresh] obviously will move processes to the scheduler. Are there objects that are not obvious that might cause scheduler problems? Or certain combination of objects?
I went through my larger, main list-processing patch and cleaned it up and it functions much more smoothly. But there's one spot that causes problems unless I put a [deferlow] in there, then all is golden. There's no problem objects left, as far as I know anyways. Hence my continued head-scratching.
I can only think of midi objects ([midiin], [notein]...), [snapshot~] and time-based objects (exluding [qmetro]) that output events on the scheduler thread. But there are probably others.
Also if I remember correctly, some actions are always defered to the low priority queue even if they come from the scheduler thread (I can think of anything that goes into js/v8/jsui/v8ui, and read/write messages to objects that can import/export files).
Also check your settings for Overdrive and Scheduler in audio interrupt since both greatly change scheduling behaviors.
Maybe you can get more info on this page if you don't know it already.
"I wonder under what circumstances a process might move from the main thread to the scheduler thread?"
1. when a scheduler thread exists (= when overdrive is on)
2. when a process requires it or might require it.
when a process in a compiled object does something with "timing", it is always build so that it outputs to the scheduler thread.
the most simple example for that is [del] and [pipe]: implementing a function which should delay a message for 50,77 milliseconds would often be useless when its output would not have priority over other things happening elsewhere.
(the big exception here is probably realtime video, where processes can be very expensive and slow and you rather deliver a frame too late than to skip it - that´s why they once made [qmetro], which originally was a jitter object and is more or less the same as metro -> defer.)
if there are two threads and a [delay 50] receives a bang button´s message it will output to the high priority thread to ensure that the difference between the input and the output is exactly 50ms.
and for the rare case where you need to output something the high priority which isnt yet, you can use del 0 or pipe 0, which are basically the opposite of [defer]. for example a bang button (mouse input = main thread) could pass [del 0] to ensure that under any circumstances both connections going from it to two different targets remain "successive".
for that "error" of using
loadbang
del 1000 -> B
-> A
it is interesting to consider the order in which the stuff happens in max when you open the first main patch:
1. obejcts are loaded
2. stuff which objects request is loaded
3. scheduler is turned on
4. loadbangs are executed
5. audio is turned on
but it is even worse, because not even this:
loadbang
del 1000 -> defer -> B
-> A
will fully solve the issue.
it will rule out that B can overtake A, but the time difference can still be incorrect if the processor is busy (and eventually too short for what you do.)
Thanks TFL and Roman. I've also been looking through this article/video/comment thread: https://cycling74.com/articles/advanced-max-learning-about-threading Unfortunately, the threadcheck external from that link doesn't work on my machine, 'incorrect architecture' says the Max console. I did find a java object and help patch called 'WhichThread' that seems to do the same thing though.
Also I think I got mixed up in my comment above. Correct me if I'm wrong, but [defer] and [deferlow] don't move processes from the main thread to the scheduler, it bumps processes that would be on the scheduler over to the main thread. So processes in the main thread with either object in the mix will stay in the main thread.