so when i am running max/msp to sort data [long txt files... 100,000s of lines] as it gets closer to finishing the job it starts to crawl... what's going on it was working pretty fine before?

i suppose i could try saving the data at strategic points [say every 50,000 lines] and adding the files together in a text editing suite?

oh and what's the longest txt file that max/msp can really use?

so when i am running max/msp to sort data [long txt files... 100,000s of lines] as it gets closer to finishing the job it starts to crawl... what's going on it was working pretty fine before?
i suppose i could try saving the data at strategic points [say every 50,000 lines] and adding the files together in a text editing suite?

oh and what's the longest txt file that max/msp can really use?
cheers.


sorry-to-ask-so-many-question-bbut-speed-issues

the reason [i think] this was happening is that the patch just wasn't doing what i intended and was just creating [and counting!] lots of error messages instead. i only noticed by accident having the message window thing open!

anyway - some of the data i have is in the form e.g. 1.5259e-005. i need that to read as a zero but don't know how to do that! any help :) ?

edit if i just knew how to take the first digit it would be a synch. maybe zl..?

hi,
the reason [i think] this was happening is that the patch just wasn't doing what i intended and was just creating [and counting!] lots of error messages instead. i only noticed by accident having the message window thing open!

edit if i just knew how to take the first digit it would be a synch. maybe zl..?


Being able to help you hinges on knowing exactly what it is you're doing. That said, [regexp] and some basic knowledge of Regular Expressions (there are many very good tutorials floating around) will probably help you a lot. (to find your example string, you'd use something like: (d.dddd(e-)ddd) off the top of my head)

My main question is, why use Max for text file sorting? There are much better tools for that out there. I've personally been using TextWrangler (the little brother of BBEdit) the past few days to re-arrange some sound file lists (in the range of 1000-5000 lines) with great success.

Being able to help you hinges on knowing exactly what it is you're doing.  That said, [regexp] and some basic knowledge of Regular Expressions (there are many very good tutorials floating around) will probably help you a lot. (to find your example string, you'd use something like: (d.dddd(e-)ddd) off the top of my head)

My main question is, why use Max for text file sorting?  There are much better tools for that out there.  I've personally been using TextWrangler (the little brother of BBEdit) the past few days to re-arrange some sound file lists (in the range of 1000-5000 lines) with great success.


hmm yeah. you're right but i'm started now and feel pig headed.

Just a quick note: All those d I had, should be \d in Max - I'd never actually looked at the max Regex implementation before.

One other idea: have you also considered using a database? Flat files are fine when you don't have a lot of information, don't need to query it, etc., but it seems like you might have better luck with a database, and Max comes with one (SQLite). You'll have to set up your tables and read your data in, but after that things get much easier and significantly faster, since databases are optimized for operations like sorting.

For instance, this would take care of what you were asking about:

UPDATE mytable SET yourcolumnname = 0. WHERE yourcolumnname < 0.00001

This would get everything (!) sorted by the field "yourcolumnname":

SELECT * from mytable ORDER BY yourcolumnname

It requires more initial effort, but that is repaid many times over. Maybe we need a "large data" tutorial, as it seems to be something that's coming up more.

One other idea: have you also considered using a database?  Flat files are fine when you don't have a lot of information, don't need to query it, etc., but it seems like you might have better luck with a database, and Max comes with one (SQLite).  You'll have to set up your tables and read your data in, but after that things get much easier and significantly faster, since databases are optimized for operations like sorting.

For instance, this would take care of what you were asking about:
UPDATE mytable SET yourcolumnname = 0. WHERE yourcolumnname < 0.00001

This would get everything (!) sorted by the field "yourcolumnname":
SELECT * from mytable ORDER BY yourcolumnname

It requires more initial effort, but that is repaid many times over.  Maybe we need a "large data" tutorial, as it seems to be something that's coming up more.


sorry to ask so many question bbut... [speed issues]