what should i use as list?
sorry, this is a complete noob question but i just did not dive into list processing yet and if someone could show me the way here, it would save me a LOT of time:
i need to generate a very simple database i can store in ram and process rapidly and with minmum cpu-load. it should contain the following data:
index (0-250.000), boolean (0/1), boolean (0/1), boolean (0/1), boolean (0/1) – basically i am looking for a table with 250.000 rows and five columns, the first the row number, the rest just booleans.
ideally, there would be an additional option to save the database to disk and load it back in but that’s a nice-to-have, performance is more important.
what would be the most efficient way to do this? table? cellblock? coll? bag? jit.matrix? any other?
my first idea was to use a 500×500 matrix 4 char for this but i guess i would generate a lot of data here i do not need since jit.matrix only supports char but no bool…
many thanks for help!
Do you really mean two hundred fifty thousand entries? That’s a lot of data.
Depends on how much inernal processing you want to do, and how much you’re just looking stuff up and getting/setting…
I’ve had success in the (distant) past using Nick Rothwell’s net.loadbang-SQL package. Or you might want to have a look at FTM — it handles big matrices pretty well (although it doesn’t have a boolean type either). Don’t bother with coll, way too slow. You also might want to have a look at dict, but i’m not sure if it will server your purposes….
interesting…if you don’t need to store the actual index, you could just use [jit.matrix 1 char 4 250000] and reference the lines directly. You only need 1 plane of char if you set the X dimension to 4. If you need the index numbers, not sure…
I wouldn’t worry about wasting memory with char versus bool, that’s no big deal when you consider that jit.matrix regularly works with 1024 x 768 matrices (or multiple ones) at high FPS, with 4 planes of data, without issue.
thank you all for your brains! i think i will try it with the matrix approach seejayjames suggested – sounds clever and as well since i have most experience with matrixes. plus it allows me to save and read the database…
i will post the outcome!
since i promised to post the outcome: i did what seejayjames suggested – using a jit.matrix, reading it out with jit.spill. and it does perform quite well, fast processing, little cpu load and it even allows me to save my database to disk.
the only difference is, i ended up not using one, but four separate matrixes for each variable since i did not find a good way to read out only one column of a matrix (except splitting it up first). it would probably work though if i would make my matrix the other way around, that is a row for each variable and the index on the x-axis (which is not an option in my case but…)
again: thanks to all of you!
Alternately, you can save the four booleans as four bits inside a single char. Bit-logic operators will tell you which bits are set and can clear or set individual bits. A little fiddly, but not at all hard once you’ve got it under your belt.
Extra points if you can extend this suggestion to reduce the memory footprint to 125kB.
If there is interest I (or several other people here) can elaborate.
Peter’s idea is a good one for saving memory, if that’s a concern.
Here’s a possible idea for easing your matrix usage. I think you only need to maintain a single matrix for your data.
----------begin_max5_patcher---------- 1137.3oc2XEsbahCE8Y6uBML6iTOHABLc1Y1reGc5jAajMJEj7HD0NsS+2qz U3D2XSLl3P7tODbjPHcz4duGcu5mSm3sPtiU6g9L5KnIS94zISftrcLos8Du prcKKypgg4sTVUwDZOe26zrcZne9JzixFz1LgFoknkRglKZX1+eqT8MzVtt. oKXl2T1TI7QJlYLq3qaTrbTVMJCoja8gIYYl.kKMilW6al.49EqjKLeTi.VQ RamqLqTM+GLaeXxrf1t2joWVvEqu2rNZ2FjjDadMhRRs+PRir+DFNK.801OR zTIazkLMrW2OU7bXGJW7vmHDuCVVQVErrd+qhmU9zaTqW.e9yXQwpMbVllKE GfmPZBfGLdF0GFt8wgfgK1iErsueMcp8g+azTsRpPOv0yp2vKKcLtTT9HRvL VBKoi1jozWIROJw9ZTTPncShoACfzwCgzeyzWEqtNaM6H5CmDfth+ck34PCO aH3nHftIQo1Vg3N4YbauttzOtg4lGOum9fCLA34m2DzgWNI1E0kfacveEubx .LSB1VCBOxJ8jCt2.nRPfHBCfMJDHzfNYRRmLoeWrY5fYyPryL6Dut9ZF+2x omPmaYiXBErQTvTc8b5SNuY5Fz8skTnowuWtuzKjWtlGd82eBssfurnMOBad B6y63etbth5jKcGDSvjyvUm3vIbzGxgSc31TJyxWjIVO.uF2AzofJcbvYHhN CjfE+jNMAifSiMpYIqrbQob42NEGj9DEXR17O1Il1EL95BfNvwuBQANIwwfN bRbDbNaRmLUTWLUIuVCgX9uLTy3YukmqKrCLL4XprGdblovEbGLB5T47pprM n6rOvnfgqXECx3XRzPkrLn49pLshuqSwqvQvM7U3p8n6RoHLABOoXWICz2QF B+wwPZzBD+xImnTHlDG3N+GhLGB43TNM0xZyG4TLCYLnllpEL0fIg1ZJC5oC hoROy1PyT2yDYKJYGdJ2Q7ikW7eMMd7soFOI3RE4wNkHGMZjkrZ7lRpFQQ9z 9KxiGAMdmlgQdeYQlBEYnzAmhAlBIlhwouihXI+ewSLgBjFwk1ANIz4KFNl9 hw82WL5Z4K1UgnxUqpYZzeMnDMnAzCtEJ2ESfmekJWj9AWsXyh1fz6LIjYhT IAn6ZYqf2PVYgIycwo32wv0w3f0t7nX61nVYpzFIjppuD7UezapBp1zWsIrc M8tl+F7tf4CtqkWbU+vtv1+eRa0xF0x8no07fddajyp0bAbyTGLF6kZcvfJ3 44LwgoSjyqsYX.aofSZ+5KbdwR0AdLLlIJXTvCsO3IYzvSXefCczLWI8.Nwi FZ5C4DMZnoWQV2TnIb7ByC5AblOdvoGn4bZfU77MRSYTs5v3P3PBZjKSj.xL B8nl3HWwbvMMAMdFDWsM27dr4RtoTLHiFZn2TnAStoLU.bvmANi2YM39DkhG p0xkHU1lMemopamS.JlLMePprMgR4Lw4BWSHIOOE6678iGVCuLkI8QsI2wFk KStcyi8lZWmeM82.5pxKr -----------end_max5_patcher-----------
It could also be encoded into around five seconds of 5-bit audio, I suppose, using poke~ and index~.
i’m currently fiddling around with some other stuff but i will get back to that…!