karl krach

sorry, this is a complete noob question but i just did not dive into list processing yet and if someone could show me the way here, it would save me a LOT of time:

i need to generate a very simple database i can store in ram and process rapidly and with minmum cpu-load. it should contain the following data:

index (0-250.000), boolean (0/1), boolean (0/1), boolean (0/1), boolean (0/1) - basically i am looking for a table with 250.000 rows and five columns, the first the row number, the rest just booleans.

ideally, there would be an additional option to save the database to disk and load it back in but that's a nice-to-have, performance is more important.

what would be the most efficient way to do this? table? cellblock? coll? bag? jit.matrix? any other?

my first idea was to use a 500x500 matrix 4 char for this but i guess i would generate a lot of data here i do not need since jit.matrix only supports char but no bool...

hi,
sorry, this is a complete noob question but i just did not dive into list processing yet and if someone could show me the way here, it would save me a LOT of time:
i need to generate a very simple database i can store in ram and process rapidly and with minmum cpu-load. it should contain the following data:
index (0-250.000), boolean (0/1), boolean (0/1), boolean (0/1), boolean (0/1) - basically i am looking for a table with 250.000 rows and five columns, the first the row number, the rest just booleans.
ideally, there would be an additional option to save the database to disk and load it back in but that's a nice-to-have, performance is more important.
what would be the most efficient way to do this? table? cellblock? coll? bag? jit.matrix? any other?

what-should-i-use-as-list

Do you really mean two hundred fifty thousand entries? That's a lot of data.

Depends on how much inernal processing you want to do, and how much you're just looking stuff up and getting/setting...

I've had success in the (distant) past using Nick Rothwell's net.loadbang-SQL package. Or you might want to have a look at FTM -- it handles big matrices pretty well (although it doesn't have a boolean type either). Don't bother with coll, way too slow. You also might want to have a look at dict, but i'm not sure if it will server your purposes....

I've had success in the (distant) past using Nick Rothwell's net.loadbang-SQL package.  Or you might want to have a look at FTM -- it handles big matrices pretty well (although it doesn't have a boolean type either).  Don't bother with coll, way too slow.  You also might want to have a look at dict, but i'm not sure if it will server your purposes....

interesting...if you don't need to store the actual index, you could just use [jit.matrix 1 char 4 250000] and reference the lines directly. You only need 1 plane of char if you set the X dimension to 4. If you need the index numbers, not sure...

I wouldn't worry about wasting memory with char versus bool, that's no big deal when you consider that jit.matrix regularly works with 1024 x 768 matrices (or multiple ones) at high FPS, with 4 planes of data, without issue.

thank you all for your brains! i think i will try it with the matrix approach seejayjames suggested - sounds clever and as well since i have most experience with matrixes. plus it allows me to save and read the database...

hey,
thank you all for your brains! i think i will try it with the matrix approach seejayjames suggested - sounds clever and as well since i have most experience with matrixes. plus it allows me to save and read the database...
i will post the outcome!
k


since i promised to post the outcome: i did what seejayjames suggested - using a jit.matrix, reading it out with jit.spill. and it does perform quite well, fast processing, little cpu load and it even allows me to save my database to disk.

the only difference is, i ended up not using one, but four separate matrixes for each variable since i did not find a good way to read out only one column of a matrix (except splitting it up first). it would probably work though if i would make my matrix the other way around, that is a row for each variable and the index on the x-axis (which is not an option in my case but...)

since i promised to post the outcome: i did what seejayjames suggested - using a jit.matrix, reading it out with jit.spill. and it does perform quite well, fast processing, little cpu load and it even allows me to save my database to disk.
the only difference is, i ended up not using one, but four separate matrixes for each variable since i did not find a good way to read out only one column of a matrix (except splitting it up first). it would probably work though if i would make my matrix the other way around, that is a row for each variable and the index on the x-axis (which is not an option in my case but...)
again: thanks to all of you!
k


Alternately, you can save the four booleans as four bits inside a single char. Bit-logic operators will tell you which bits are set and can clear or set individual bits. A little fiddly, but not at all hard once you've got it under your belt.

Extra points if you can extend this suggestion to reduce the memory footprint to 125kB.

If there is interest I (or several other people here) can elaborate.

Extra points if you can extend this suggestion to reduce the memory footprint to 125kB. 

If there is interest I (or several other people here) can elaborate.


Peter's idea is a good one for saving memory, if that's a concern.

Here's a possible idea for easing your matrix usage. I think you only need to maintain a single matrix for your data.

It could also be encoded into around five seconds of 5-bit audio, I suppose, using poke~ and index~.

i'm currently fiddling around with some other stuff but i will get back to that...!

cool, thanks!
i'm currently fiddling around with some other stuff but i will get back to that...!
k


what should i use as list?