Cycling 74 » Topic: Big coll : performance hit? http://cycling74.com/forums/topic/big-coll-performance-hit/feed Tue, 18 Jun 2013 00:06:20 +0000 http://bbpress.org/?v=2.2.4 http://cycling74.com/forums/topic/big-coll-performance-hit/#post-53854 <![CDATA[Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-53854 Fri, 10 Dec 2010 16:33:07 +0000 Christopher Bailey I am wondering if anyone has experience with Very Very Large colls. Like with 50000 lines, or 150000 lines. I assume there is no maximum line limit. If I ask for line# 49834, is there a performance hit?

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193841 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193841 Sat, 11 Dec 2010 01:23:21 +0000 Emmanuel Jourdan coll uses a linklist. So in order to fin the #49834 it has to iterate through everything. First apply the usual rule: don’t optimize, but code! and if you have a performance issue, you can for instance split your data among multiple coll by doing something more sophisticated on the key.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193842 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193842 Sat, 11 Dec 2010 05:06:32 +0000 Christopher Bailey Thank you for the information.

Hmmmm . . .ahh, but if I am stepping through the coll using BANG or NEXT message, then perhaps it does not have to iterate through each time you step; it “remembers” or is still “there”? In other words, I step through 1, 2, 3, . . . . 29854, and the time it takes to go from 2 to 3, is the same as the time to go from 29853 to 29854. Is this correct? If so, I am OK with what I am doing, I think . . . .

I am not sure I understand the “usual rule” that you mention. Do you mean, “program what you want, worry about performance issues when they actually happen” ? I am afraid I have done that in past and then been very sad.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193843 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193843 Sat, 11 Dec 2010 05:14:29 +0000 seejayjames [text] doesn’t use a linked list, does it? am wondering for an upcoming project where I’d need several thousand word entries, each with some category flags. would this be best to do with [text] on a line-by-line basis, or with [coll]? They wouldn’t need to be changed once they’re in there, just accessed with their flags.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193844 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193844 Sat, 11 Dec 2010 08:15:07 +0000 Roman Thilenius you will often find that it is not possible to replace a [coll] with [text] so easily.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193845 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193845 Sat, 11 Dec 2010 13:35:43 +0000 Christopher Bailey I think my original follow-up question got buried:

So if I am STEPPING through a coll sequentially, does that avoid the possible performance issue of having to iterate through the linked-list for each step?

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193846 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193846 Sat, 11 Dec 2010 17:25:55 +0000 Emmanuel Jourdan @christopher

Yes it does. next/prev starts from the last known location in the linklist.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193847 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193847 Sat, 11 Dec 2010 17:27:31 +0000 pizza olives Hello Christopher Bailey,

Test ?

– Pasted Max Patch, click to expand. –
]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193848 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193848 Sun, 12 Dec 2010 00:23:30 +0000 Floating Point Emmanuel
Is the struct traversal algorithm smart enough to know that if two consecutive indexes are being addressed (ie 1, then 2) then this is the same as using ( next) ie (1, then next)?

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193849 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193849 Sun, 12 Dec 2010 16:47:07 +0000 pizza olives Hello Terry McDermott,

Is the struct traversal algorithm smart enough to know that if two consecutive indexes are being addressed (ie 1, then 2) then this is the same as using ( next) ie (1, then next)?

according this test, i should say NO ;-)

– Pasted Max Patch, click to expand. –
]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193850 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193850 Sun, 12 Dec 2010 18:32:22 +0000 Chris Muir Mmm… empirical data…

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193851 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193851 Sun, 12 Dec 2010 19:52:52 +0000 Alexandre We see so much topics about [coll] performance issue, sorry to repeat that again, but i find that, because of this “iterate-through-everything” neccesity, coll is the most inefficient piece of object in max. Hundreds of thousand of computer cycles to just recover one single value. ( http://cycling74.com/forums/topic.php?id=28901 )

I might be wrong but my feeling is that maxers rarely use this “Linked list” feature of coll, so i think that a kind of [arraylist] object is missing in max. This ideal [arraylist] object wouldn’t need to iterate through everything to retrieve the list n°49834 stored in it. It would just look directly for the list n°49834.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193852 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193852 Sun, 12 Dec 2010 20:41:28 +0000 johnpitcairn Bear in mind that coll can use symbols as indexes.

To improve coll’s performance with longer lists, a hashtab would probably be needed. Its performance would be worse than a linked list for (quite) short lists.

There’s a project for somebody. The coll source is available in the SDK – try rewriting it to use a t_hashtab. Doing a “next” in a hashtab is impossible(?), so it’s likely you’d need a t_linklist in parallel anyway (storing just the coll indexes), ie you’re gonna wind up with an additional performance hit when storing/removing indexes etc.

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193853 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193853 Mon, 13 Dec 2010 05:23:06 +0000 Christopher Bailey Thanks for the discussion, everybody. I am using coll as a kind of storage for sequencer data (where each line contains a variable number of parameters for different kinds of note/events). So, except for an initial “Go To” event, I’ll be stepping through, so I suppose I am OK with that.

If anyone has other suggestions on how to do sequencing, I’m all ears. . .

I know I’m not the first to use coll this way..

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193854 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193854 Mon, 13 Dec 2010 07:06:31 +0000 pizza olives Hello Christopher Bailey,

I use [jit.matrix] to store data for a sequencer ; but not sure that’s best …

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193855 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193855 Mon, 13 Dec 2010 10:00:48 +0000 Floating Point Chris, you say:
I am using coll as a kind of storage for sequencer data (where each line contains a variable number of parameters for different kinds of note/events).

Isn’t [detonate] optimized for this kind of application? Have you looked at that?
T

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193856 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193856 Mon, 13 Dec 2010 11:32:43 +0000 andrea agostini [detonate] is really about MIDI notes… I’d probably use [qlist] as a more general tool. But I also happened doing the same with [coll], and it worked quite well.

@johnpitcairn: unfortunately the source for coll is not in the sdk… it might not be too difficult replicating its basic behaviour using hashtabs, but it sounds like a very boring job! on the other hand, in the sdk there are a couple of examples dealing with databases, one of which is called “dbcuelist” – it looks quite basic, but not bad at all…

aa

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193857 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193857 Mon, 13 Dec 2010 11:36:55 +0000 andrea agostini … or you might give a look at the slots of bach.roll and bach.score – http://www.bachproject.net

aa

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193858 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193858 Mon, 13 Dec 2010 20:37:45 +0000 Alexandre > “I use [jit.matrix] to store data for a sequencer ; but not sure that’s best …”

From what i remember, in term of efficiency, It is the best in max to do that. Around X100 faster than coll.
A while ago, i made some efficiency test to store sequence data in an array, comparing : jit.matrix, coll, ftm, peek~ and buffer, etc.. to store and read values :

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193859 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193859 Tue, 14 Dec 2010 07:20:51 +0000 pizza olives Hello Alexandre,

[jit.matrix] have few problems for data collection use :

1. cannot be zero sized ; at least one.
2. always use [jit.qfaker] for high prioritize it.
3. cannot easily be resized without loosing values.

but it seems the best native object ;-)

PS : i didn’t test it with 150 000 values …

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193860 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193860 Tue, 14 Dec 2010 19:40:13 +0000 johnpitcairn @andrea agostini: The coll source is in the Max 4 SDK. I suspect it’s little different, if at all, in Max 5. I wonder what the comparative lookup times are for a database?

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193861 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193861 Tue, 14 Dec 2010 22:25:25 +0000 andrea agostini mmm… I should have a Max 4 SDK somewhere. But if it’s a pre-obex version, quite a lot of things could have changed. I’ll go and check it out.

I’d be curious too about the performance of a database. In principle, I’d expect it to be very efficient on huge collections of data, not as much for smaller ones.

aa

]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193862 <![CDATA[Re: Big coll : performance hit?]]> http://cycling74.com/forums/topic/big-coll-performance-hit/#post-193862 Wed, 15 Dec 2010 00:54:49 +0000 Christopher Bailey I was using a coll because I can stash text, numbers, all kinds of things etc. in there. I’m not sure that jit.matrix can do this? (Although of course there are ways around that).

]]>