Augustine Bannatyne - Leudar

I am looking to log data every 20ms for a twenty four hour period - any idea of how i can do that with max msp ?

i've done a search and looked at some of the previous solutions but Im not sure they will work with such a high volume and long term data logging - would coll last that long ?

I am looking to log data every 20ms for a twenty four hour period - any idea of how i can do that with max msp ?
i've done a search and looked at some of the previous solutions but Im not sure they will work with such a high volume and long term data logging - would coll last that long ?


long-term-data-logging

That's quite a volume and speed you need. The only option i can think of is the SQlite database. Maybe you should think of ways to increase the read/write intervals i.e. by concatenating 25 data points into a comma seperated string and only write every 500 ms.

Hi Jan. The only thin is the events I a particularly interested in logging only last 20 ms ! Is sqlite compatible with max ? I have a lab jack u3 data acquisition device maybe that has some other software and then I can just feed the data into max later......

 and record into it with peek~/poke~. A 100s buffer at 44.1kHz will already give you enough space.

Couldn't you use a buffer~ and record into it with peek~/poke~. A 100s buffer at 44.1kHz will already give you enough space.


Here's a good tutorial of using SQLite with Max:

https://cycling74.com/tutorials/data-collection-building-databases-using-sqlite/

Max comes with a built in SQlite implementation that you can use directly from Max.

Maybe everything works fine with that speed and amount of data. You should just test.

My idea was not to reduce the amount of data but the amount of read/write actions and the amount of table entries.

I.e. if your first 5 data points are 1, 5, 6, 10, 0 you can save them in a single action/table line as one string "1,5,6,10,0" in one column and a timestamp in a second. Like this you can find and parse out every point in time. But that will also depend on what you need to do with that data.... A 24h long log of every 20 ms will result in 4.320.000 individual records. Not that it is unmanagable but it is an awful lot! :)

Hi Augustine,
Max comes with a built in SQlite implementation that you can use directly from Max.
Maybe everything works fine with that speed and amount of data. You should just test.
My idea was not to reduce the amount of data but the amount of read/write actions and the amount of table entries.
I.e. if your first 5 data points are 1, 5, 6, 10, 0 you can save them in a single action/table line as one string "1,5,6,10,0" in one column and a timestamp in a second. Like this you can find and parse out every point in time. But that will also depend on what you need to do with that data.... A 24h long log of every 20 ms will result in 4.320.000 individual records. Not that it is unmanagable but it is an awful lot! :)


If it is only the last (few) item(s) you're interested in, you could set up a [zl.queue] and only keep the last items.

no i need all the data the whole 24 hours.

it sounds like a mission but I will give it a go - thanks for the advice.

I'd second the database route. If something bad happens, (Max crashes, etc.) you'll still have some data. And it will be searchable. If you use Postgres you even get handy window functions for analyzing it.

That said, if you need sample accurate intervals, sound files will be more accurate.

I'd second the database route.  If something bad happens, (Max crashes, etc.) you'll still have some data.  And it will be searchable.  If you use Postgres you even get handy window functions for analyzing it.

That said, if you need sample accurate intervals, sound files will be more accurate.


I've never looked at java script before ...I followed the tutorial in the link above

but I keep getting a message in the max window "no database is currently loaded" I opened the h's script in max im sure its a schoolboy error - any ideas ?

I've never looked at java script before ...I followed  the tutorial in the link above 

but I keep getting a message in the max window "no database is currently loaded" I opened the h's script in max im sure its a schoolboy error - any ideas ?


Just discovered that sfrecord~ (recently added?) has a resampling mode. From the refs:

The word resample , followed by a float, will upsample or downsample the file. Sample rates are expressed as floating-point values - 1.0 is the current sampling rate, 0.5 is half the current, 2.0 is twice the current sample rate, etc.

Every 20 ms should be 0.02, I think (1/50th).

Just discovered that sfrecord~ (recently added?) has a resampling mode.  From the refs:

Every 20 ms should be 0.02, I think (1/50th).


This might be a really good idea actually. Im giving it a go at the moment.

hi there...im trying to do something very similar..did you guys find a good solution for this?

I got there in the end - you can see it here called "Time lapse Data logging" middle bottom , though theres a few vids on plant electrophysiology and max there- Ill try and dig the patch out been a while since I used it :

http://www.augustineleudar.com/Offline%20Website/products.html

Long term data logging ?