Mattijs

If I'm stepping through a coll with 'next', how could I output an entry I reached -without- jumping to the next entry? I would expect a bang to do this, but bang seems similar to 'next'.

Just making sure I didn't miss something terribly obvious..

coll-simple-output-current-option

That works, only requesting an index from coll requires searching through the Linked List and can be slow for bigger Colls.

In this case, we'd like to be able to address the current item directly. A 'current' message would seem like a logical addition to prev and next. We currently have to store the output so we can access it directly.

The patch below would be a workaround, but doesn't win a beauty prize:

In this case, we'd like to be able to address the current item directly. A 'current' message would seem like a logical addition to prev and next. We currently have to store the output so we can access it directly. 

The patch below would be a workaround, but doesn't win a beauty prize: 

Here are two possible workarounds, slightly different. The first one ignores the current location of the "goto" pointer, but keeps track of the "most recently output" address, so that you can output that item as many times as you want. The second one uses the current location of the "goto" pointer, then immediately resets the pointer to where it was.

From what I see in the code of coll, some messages should be much faster than others. 'next' does nothing but to look at the current entry and acquire the pointer to the next element in the linked list, which should be very fast. Inputting an index on the other hand runs through all the entries starting from the top until it reaches the required one.

BUT now that I'm trying this, the results don't make sense (see patch below).

Is there a guru on this forum that can explain why dump is so much faster than uzi->next and why uzi->next is not much faster than uzi->index?

BUT now that I'm trying this, the results don't make sense (see patch below). 

However, i'm afraid both of them still use an index to access or re-access the data. We're using colls that are quite large in real-time situations and we simply don't want to spend CPU time on searching through a linked list (

http://en.wikipedia.org/wiki/Linked_list)

 while i presume there could well be a pointer to the current node.

I don't remember whether Mattijs used incremental integer indexes for this particular thing, but if so, the only thing we'd need would be an option to turn coll into an array.... j/k :-)

However, i'm afraid both of them still use an index to access or re-access the data. We're using colls that are quite large in real-time situations and we simply don't want to spend CPU time on searching through a linked list (http://en.wikipedia.org/wiki/Linked_list) while i presume there could well be a pointer to the current node.

I don't remember whether Mattijs used incremental integer indexes for  this particular thing, but if so, the only thing we'd need would be an option to turn coll into an array.... j/k :-)


>BUT now that I'm trying this, the results don't make sense (see patch

>Is there a guru on this forum that can explain why dump is so much

>faster than uzi->next and why uzi->next is not much faster than uzi->index?

Connections are wrong, you need to use 3rd outlet of uzi to get index. 

It seems index are faster than next, and bang are slightly faster than next. 

Still - I can't tell why dump is so much faster. 

But my suggestion would be to use dump and do the filtering after the coll.

>BUT now that I'm trying this, the results don't make sense (see patch
below). 
>
>Is there a guru on this forum that can explain why dump is so much
>faster than uzi->next and why uzi->next is not much faster than uzi->index?
>

Connections are wrong, you need to use 3rd outlet of uzi to get index. 
It seems index are faster than next, and bang are slightly faster than next. 
Still - I can't tell why dump is so much faster. 
But my suggestion would be to use dump and do the filtering after the coll.

Jakob Riis
jr.abstractions for MaxMSP
http://www.sonicescape.net/maxmsp/
__________________________________


Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38

----------------------------------------------------

> Connections are wrong, you need to use 3rd outlet of uzi to get index.

> It seems index are faster than next, and bang are slightly faster than next. 

> Still - I can't tell why dump is so much faster. 

> But my suggestion would be to use dump and do the filtering after the coll.

Yeah, I'll see if I can modify my patch to use dump. I only need to output 2 or 3 entries at a time though, from a possibly long coll. I need to do this every 2 milliseconds, so every performance upgrade would be welcome. If only there was a 'stopdump' message...

Anyhow, the fact that next is even slower than index can be considered a bug, right?

Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38
----------------------------------------------------

> Connections are wrong, you need to use 3rd outlet of uzi to get index. 

> It seems index are faster than next, and bang are slightly faster than next. 
> Still - I can't tell why dump is so much faster. 
> But my suggestion would be to use dump and do the filtering after the coll.

Yeah, I'll see if I can modify my patch to use dump. I only need to output 2 or 3 entries at a time though, from a possibly long coll. I need to do this every 2 milliseconds, so every performance upgrade would be welcome. If only there was a 'stopdump' message... 

Random idea? use a char matrix with jit.str, and use getcell for the 

data? I have not been actively following the discussion (apologies for 

butting in!), but perhaps it would take some time to build a wrapper, 

Im not sure what type of data the coll will contain, but it might be 

On Nov 13, 2007, at 7:47 AM, Mattijs Kneppers wrote:

> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38

> ----------------------------------------------------

>> Connections are wrong, you need to use 3rd outlet of uzi to get 

>> It seems index are faster than next, and bang are slightly faster 

>> Still - I can't tell why dump is so much faster.

>> But my suggestion would be to use dump and do the filtering after 

> Yeah, I'll see if I can modify my patch to use dump. I only need to 

> output 2 or 3 entries at a time though, from a possibly long coll. I 

> need to do this every 2 milliseconds, so every performance upgrade 

> would be welcome. If only there was a 'stopdump' message...

> Anyhow, the fact that next is even slower than index can be 

> Hard- and software for interactive audiovisual sampling

Random idea? use a char matrix with jit.str, and use getcell for the  
data? I have not been actively following the discussion (apologies for  
butting in!), but perhaps it would take some time to build a wrapper,  
but might perform a bit better?

Im not sure what type of data the coll will contain, but it might be  
worth a shot to try?

>
> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38
> ----------------------------------------------------
>
>> Connections are wrong, you need to use 3rd outlet of uzi to get  
>> index.
>
> Oops. Thanks.
>
>> It seems index are faster than next, and bang are slightly faster  
>> than next.
>> Still - I can't tell why dump is so much faster.
>> But my suggestion would be to use dump and do the filtering after  
>> the coll.
>
> Yeah, I'll see if I can modify my patch to use dump. I only need to  
> output 2 or 3 entries at a time though, from a possibly long coll. I  
> need to do this every 2 milliseconds, so every performance upgrade  
> would be welcome. If only there was a 'stopdump' message...
>
> Anyhow, the fact that next is even slower than index can be  
> considered a bug, right?
>
> Mattijs
>
> --
> SmadSteck - http://www.smadsteck.nl
> Hard- and software for interactive audiovisual sampling


I don't seem to have a jit.str. There are several others. The closest is

jit.str.op but I don't see a getcell message.

> Random idea? use a char matrix with jit.str, and use getcell for the

> data? I have not been actively following the discussion (apologies for

> butting in!), but perhaps it would take some time to build a wrapper,

> Im not sure what type of data the coll will contain, but it might be

> On Nov 13, 2007, at 7:47 AM, Mattijs Kneppers wrote:

>> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38

>> ----------------------------------------------------

>>> Connections are wrong, you need to use 3rd outlet of uzi to get

>>> It seems index are faster than next, and bang are slightly faster

>>> Still - I can't tell why dump is so much faster.

>>> But my suggestion would be to use dump and do the filtering after

>> Yeah, I'll see if I can modify my patch to use dump. I only need to

>> output 2 or 3 entries at a time though, from a possibly long coll. I

>> need to do this every 2 milliseconds, so every performance upgrade

>> would be welcome. If only there was a 'stopdump' message...

>> Anyhow, the fact that next is even slower than index can be

>> Hard- and software for interactive audiovisual sampling

I don't seem to have a jit.str.  There are several others.  The closest is
jit.str.op but I don't see a getcell message.

> Random idea? use a char matrix with jit.str, and use getcell for the
> data? I have not been actively following the discussion (apologies for
> butting in!), but perhaps it would take some time to build a wrapper,
> but might perform a bit better?
>
> Im not sure what type of data the coll will contain, but it might be
> worth a shot to try?
>
>
> On Nov 13, 2007, at 7:47 AM, Mattijs Kneppers wrote:
>
>>
>> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38
>> ----------------------------------------------------
>>
>>> Connections are wrong, you need to use 3rd outlet of uzi to get
>>> index.
>>
>> Oops. Thanks.
>>
>>> It seems index are faster than next, and bang are slightly faster
>>> than next.
>>> Still - I can't tell why dump is so much faster.
>>> But my suggestion would be to use dump and do the filtering after
>>> the coll.
>>
>> Yeah, I'll see if I can modify my patch to use dump. I only need to
>> output 2 or 3 entries at a time though, from a possibly long coll. I
>> need to do this every 2 milliseconds, so every performance upgrade
>> would be welcome. If only there was a 'stopdump' message...
>>
>> Anyhow, the fact that next is even slower than index can be
>> considered a bug, right?
>>
>> Mattijs
>>
>> --
>> SmadSteck - http://www.smadsteck.nl
>> Hard- and software for interactive audiovisual sampling
>

Cheers
Gary Lee Nelson
Oberlin College
www.timara.oberlin.edu/GaryLeeNelson

On Nov 13, 2007, at 2:09 PM, Gary Lee Nelson wrote:

> I don't seem to have a jit.str. There are several others. The 

> jit.str.op but I don't see a getcell message.

>> Random idea? use a char matrix with jit.str, and use getcell for the

>> data? I have not been actively following the discussion (apologies 

>> butting in!), but perhaps it would take some time to build a wrapper,

>> Im not sure what type of data the coll will contain, but it might be

>> On Nov 13, 2007, at 7:47 AM, Mattijs Kneppers wrote:

>>> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38

>>> ----------------------------------------------------

>>>> Connections are wrong, you need to use 3rd outlet of uzi to get

>>>> It seems index are faster than next, and bang are slightly faster

>>>> Still - I can't tell why dump is so much faster.

>>>> But my suggestion would be to use dump and do the filtering after

>>> Yeah, I'll see if I can modify my patch to use dump. I only need to

>>> output 2 or 3 entries at a time though, from a possibly long coll. I

>>> need to do this every 2 milliseconds, so every performance upgrade

>>> would be welcome. If only there was a 'stopdump' message...

>>> Anyhow, the fact that next is even slower than index can be

>>> Hard- and software for interactive audiovisual sampling

> I don't seem to have a jit.str.  There are several others.  The  
> closest is
> jit.str.op but I don't see a getcell message.
>
>
> On 11/13/07 10:42 AM, "vade"  wrote:
>
>> Random idea? use a char matrix with jit.str, and use getcell for the
>> data? I have not been actively following the discussion (apologies  
>> for
>> butting in!), but perhaps it would take some time to build a wrapper,
>> but might perform a bit better?
>>
>> Im not sure what type of data the coll will contain, but it might be
>> worth a shot to try?
>>
>>
>> On Nov 13, 2007, at 7:47 AM, Mattijs Kneppers wrote:
>>
>>>
>>> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38
>>> ----------------------------------------------------
>>>
>>>> Connections are wrong, you need to use 3rd outlet of uzi to get
>>>> index.
>>>
>>> Oops. Thanks.
>>>
>>>> It seems index are faster than next, and bang are slightly faster
>>>> than next.
>>>> Still - I can't tell why dump is so much faster.
>>>> But my suggestion would be to use dump and do the filtering after
>>>> the coll.
>>>
>>> Yeah, I'll see if I can modify my patch to use dump. I only need to
>>> output 2 or 3 entries at a time though, from a possibly long coll. I
>>> need to do this every 2 milliseconds, so every performance upgrade
>>> would be welcome. If only there was a 'stopdump' message...
>>>
>>> Anyhow, the fact that next is even slower than index can be
>>> considered a bug, right?
>>>
>>> Mattijs
>>>
>>> --
>>> SmadSteck - http://www.smadsteck.nl
>>> Hard- and software for interactive audiovisual sampling
>>
>
>
> Cheers
> Gary Lee Nelson
> Oberlin College
> www.timara.oberlin.edu/GaryLeeNelson
>
>


Some version of the coll source is given out with the SDK under 

On Nov 13, 2007, at 4:47 AM, Mattijs Kneppers wrote:

> output 2 or 3 entries at a time though, from a possibly long coll. 

> I need to do this every 2 milliseconds, so every performance 

> upgrade would be welcome. If only there was a 'stopdump' message...

San Francisco, CA    Work: 857-544-3967

Some version of the coll source is given out with the SDK under  
example-externs.

>
> Quote: Jakob Riis wrote on Tue, 13 November 2007 12:38
> ----------------------------------------------------
>
>> Connections are wrong, you need to use 3rd outlet of uzi to get  
>> index.
>
> Oops. Thanks.
>
>> It seems index are faster than next, and bang are slightly faster  
>> than next.
>> Still - I can't tell why dump is so much faster.
>> But my suggestion would be to use dump and do the filtering after  
>> the coll.
>
> Yeah, I'll see if I can modify my patch to use dump. I only need to  
> output 2 or 3 entries at a time though, from a possibly long coll.  
> I need to do this every 2 milliseconds, so every performance  
> upgrade would be welcome. If only there was a 'stopdump' message...
>
> Anyhow, the fact that next is even slower than index can be  
> considered a bug, right?
>
> Mattijs
>
> --
> SmadSteck - http://www.smadsteck.nl
> Hard- and software for interactive audiovisual sampling

San Francisco, CA	Work: 857-544-3967
Email: bthrew@gmail.com
IM: captogreadmore (AIM)
http:/www.barrythrew.com


Quote: barry threw wrote on Tue, 13 November 2007 22:31

> Some version of the coll source is given out with the SDK under 

I considered that. But the version in the sdk is an old one. coll has been updated since then.

Of course I could roll my own buffer/sync external altogether (that's what I'm after, ultimately), but I was hoping to save time and do it with max native objects, which one would expect to be possible..

Quote: barry threw wrote on Tue, 13 November 2007 22:31
----------------------------------------------------
> Some version of the coll source is given out with the SDK under  
> example-externs.
>
> Add your method.
>

Quote: vade wrote on Tue, 13 November 2007 20:52

That's an interesting approach, vade. In my case though, performance is the primary concern. I'll post an example of what I'm working on tomorrow, so that all you generous participants of this thread can see why I'm so keen on this.

Quote: vade wrote on Tue, 13 November 2007 20:52
----------------------------------------------------
> I meant something like:

Hm. I did not give it much time/thought today, but 'dumping' the whole 

coll into a large enough matrix, then getting the cell(index) you 

want, might improve your performance. Thats what i was hinting at.

On Nov 13, 2007, at 5:33 PM, Mattijs Kneppers wrote:

> Quote: vade wrote on Tue, 13 November 2007 20:52

> That's an interesting approach, vade. In my case though, performance 

> is the primary concern. I'll post an example of what I'm working on 

> tomorrow, so that all you generous participants of this thread can 

Hm. I did not give it much time/thought today, but 'dumping' the whole  
coll into a large enough matrix, then getting the cell(index) you  
want, might improve your performance. Thats what i was hinting at.

>
> Quote: vade wrote on Tue, 13 November 2007 20:52
> ----------------------------------------------------
>> I meant something like:
>
> That's an interesting approach, vade. In my case though, performance  
> is the primary concern. I'll post an example of what I'm working on  
> tomorrow, so that all you generous participants of this thread can  
> see why I'm so keen on this.
>
> Mattijs
> --
> SmadSteck - http://www.smadsteck.nl
> Hard- and software for interactive audiovisual sampling


These would be too much objects to even start looking at if I were you, but well, just in case you're interested..

This is the basic version. I dump the entire coll every 2 ms. It should be possible to do this more efficiently, knowing that coll is just a linked list that I could step through a few steps at a time with 'next', only when needed.

> Is there a guru on this forum that can explain why dump is so much

> faster than uzi->next and why uzi->next is not much faster than

I am not a guru, but why would you expect that message handling has no 

Its easy to explain why this result is the one to expect. Even though I 

always prefer abstractions over externals, I know that externals are way 

faster to do what they do. Especially a dump message is done internally 

with the highest possible optimisation. It does have to deal and decode 

a single message. using next, bang, or numbers will have to decode and 

Your computer seems to be 10 times faster than mine with the patch you 

thought is faster (more than 25 seconds). The dump needed only 3 time 

It seems thinking about efficiency is more dangerous if you have a 

> Anyhow, the fact that next is even slower than index can be

To decode a string will always be slower than decoding a number...

As you deal only with numbers, you could dump the numbers into several 

buffer~s and access them with peek~/poke~.

I don't know why you would want to dump the whole coll every 2 ms. That 

doesn't make sense for me. You should always know when something changed 

I also don't know why the existing next message will be a problem, 

including the workarounds for a current output. It looks you are dealing 

with Midi. And Midi is comparatively slow...

It could be even feasable to record your data into a soundfile, and 

access this from your clients... just a thought...

Mattijs Kneppers schrieb:
> Is there a guru on this forum that can explain why dump is so much
> faster than uzi->next and why uzi->next is not much faster than
> uzi->index?

I am not a guru, but why would you expect that message handling has no 
effect on the overhead of a patch?

Its easy to explain why this result is the one to expect. Even though I 
always prefer abstractions over externals, I know that externals are way 
faster to do what they do. Especially a dump message is done internally 
with the highest possible optimisation. It does have to deal and decode 
a single message. using next, bang, or numbers will have to decode and 
route 20000 messages to its places...

Your computer seems to be 10 times faster than mine with the patch you 
thought is faster (more than 25 seconds). The dump needed only 3 time 
more... (150 ms).

It seems thinking about efficiency is more dangerous if you have a 
faster machine... ;-)

Mattijs Kneppers schrieb:
 > Anyhow, the fact that next is even slower than index can be
 > considered a bug, right?

To decode a string will always be slower than decoding a number...
I'd call it expected behaviour...

Mattijs Kneppers schrieb:
 > Here is what I'm working on:

As you deal only with numbers, you could dump the numbers into several 
buffer~s and access them with peek~/poke~.

I don't know why you would want to dump the whole coll every 2 ms. That 
doesn't make sense for me. You should always know when something changed 
and could just take that info...

I also don't know why the existing next message will be a problem, 
including the workarounds for a current output. It looks you are dealing 
with Midi. And Midi is comparatively slow...

It could be even feasable to record your data into a soundfile, and 
access this from your clients... just a thought...

-- 
Stefan Tiedje------------x-------
--_____-----------|--------------
--(_|_ ----|-----|-----()-------
-- _|_)----|-----()--------------
----------()--------www.ccmix.com

'To decode a string will always be slower than decoding a number...

Could you explain this a little bit more? I could well be true that decoding 'next' takes longer than decoding an int (even though both are 4 byte, haha), but surely decoding 'next' doesn't take longer than following 10k of pointers on average? Or am i missing some of the functionality of the linkedlist?

'To decode a string will always be slower than decoding a number...
I'd call it expected behaviour...'

I haven't done a great deal of speed testing, but i know that in another tool, we use a single line in (a number of) colls for every frame we render, and we try to avoid rendering compositions that are much longer than 5 minutes (which is only 300x50 = 6k lines, maybe 20 colls total i think). It appears the thing really slows down towards the end, which makes us believe the thing follows the pointers from the beginning of the coll.

But i would indeed be interested in more info about this, sometime...

Coll is incrementally slower at looking up higher indices. The size doesn't matter, it's the index position:

> It appears the thing really slows down towards the end, which makes

> us believe the thing follows the pointers from the beginning of the

I hacked together a little test, and I can confirm that its slower 

towards the end. Not really slow, but significantly on my machine.

Till about 1000 entries its not measurable, at 20000 it will need 6 or 7 

milliseconds... No matter whats in the coll as data (symbols or numbers)

I tested a buffer~ as well. That seems to be the random access to go 

for... (You might need several of them but who cares if its fast... ;-)

Bas van der Graaff schrieb:
> It appears the thing really slows down towards the end, which makes
> us believe the thing follows the pointers from the beginning of the
> coll.

I hacked together a little test, and I can confirm that its slower 
towards the end. Not really slow, but significantly on my machine.

Till about 1000 entries its not measurable, at 20000 it will need 6 or 7 
milliseconds... No matter whats in the coll as data (symbols or numbers)

I tested a buffer~ as well. That seems to be the random access to go 
for... (You might need several of them but who cares if its fast... ;-)

Quote: johnpitcairn wrote on Wed, 21 November 2007 23:06

> Coll is incrementally slower at looking up higher indices. The size doesn't matter, it's the index position:

That's right, and it makes sense because coll is simply a linkedlist that you run through until you reach the index you need.

BUT the 'next' command should only go to the next entry of the linkedlist, so running through the coll with the 'next' command should be much faster than entering incremental indeces. This is not the case, as you can see in my second message in this thread (#120901). Which is very weird.

Quote: johnpitcairn wrote on Wed, 21 November 2007 23:06
----------------------------------------------------
> Coll is incrementally slower at looking up higher indices. The size doesn't matter, it's the index position:
>

That's right, and it makes sense because coll is simply a linkedlist that you run through until you reach the index you need. 

> BUT the 'next' command should only go to the next entry of the

> linkedlist, so running through the coll with the 'next' command

> should be much faster than entering incremental indeces. This is not

> the case, as you can see in my second message in this thread

It seems to be a bad test, if I test it with the patch I posted 

recently, a next or bang in the range of 19000 (after a slow goto) is as 

fast as a recall in the beginning of the coll...

coll simple 'output current' option