Cycling 74 » Topic: what should i use as list?

what should i use as list?

karl krach — Mon, 04 Mar 2013 22:17:41 +0000

hi,
sorry, this is a complete noob question but i just did not dive into list processing yet and if someone could show me the way here, it would save me a LOT of time:
i need to generate a very simple database i can store in ram and process rapidly and with minmum cpu-load. it should contain the following data:
index (0-250.000), boolean (0/1), boolean (0/1), boolean (0/1), boolean (0/1) – basically i am looking for a table with 250.000 rows and five columns, the first the row number, the rest just booleans.
ideally, there would be an additional option to save the database to disk and load it back in but that’s a nice-to-have, performance is more important.
what would be the most efficient way to do this? table? cellblock? coll? bag? jit.matrix? any other?

my first idea was to use a 500×500 matrix 4 char for this but i guess i would generate a lot of data here i do not need since jit.matrix only supports char but no bool…

many thanks for help!
k

Re: what should i use as list?

Chris Muir — Mon, 04 Mar 2013 23:26:04 +0000

Do you really mean two hundred fifty thousand entries? That’s a lot of data.

Re: what should i use as list?

mattyo — Tue, 05 Mar 2013 03:10:06 +0000

Depends on how much inernal processing you want to do, and how much you’re just looking stuff up and getting/setting…

I’ve had success in the (distant) past using Nick Rothwell’s net.loadbang-SQL package. Or you might want to have a look at FTM — it handles big matrices pretty well (although it doesn’t have a boolean type either). Don’t bother with coll, way too slow. You also might want to have a look at dict, but i’m not sure if it will server your purposes….

hth,

Re: what should i use as list?

seejayjames — Tue, 05 Mar 2013 06:22:02 +0000

interesting…if you don’t need to store the actual index, you could just use [jit.matrix 1 char 4 250000] and reference the lines directly. You only need 1 plane of char if you set the X dimension to 4. If you need the index numbers, not sure…

I wouldn’t worry about wasting memory with char versus bool, that’s no big deal when you consider that jit.matrix regularly works with 1024 x 768 matrices (or multiple ones) at high FPS, with 4 planes of data, without issue.

Re: what should i use as list?

karl krach — Tue, 05 Mar 2013 12:14:45 +0000

hey,
thank you all for your brains! i think i will try it with the matrix approach seejayjames suggested – sounds clever and as well since i have most experience with matrixes. plus it allows me to save and read the database…
i will post the outcome!
k

Re: what should i use as list?

karl krach — Sat, 09 Mar 2013 12:12:23 +0000

since i promised to post the outcome: i did what seejayjames suggested – using a jit.matrix, reading it out with jit.spill. and it does perform quite well, fast processing, little cpu load and it even allows me to save my database to disk.
the only difference is, i ended up not using one, but four separate matrixes for each variable since i did not find a good way to read out only one column of a matrix (except splitting it up first). it would probably work though if i would make my matrix the other way around, that is a row for each variable and the index on the x-axis (which is not an option in my case but…)
again: thanks to all of you!
k

Re: what should i use as list?

Peter Castine — Sat, 09 Mar 2013 14:19:02 +0000

Alternately, you can save the four booleans as four bits inside a single char. Bit-logic operators will tell you which bits are set and can clear or set individual bits. A little fiddly, but not at all hard once you’ve got it under your belt.

Extra points if you can extend this suggestion to reduce the memory footprint to 125kB.

If there is interest I (or several other people here) can elaborate.

Re: what should i use as list?

Jeremy Bernstein — Sat, 09 Mar 2013 15:11:24 +0000

Peter’s idea is a good one for saving memory, if that’s a concern.

Here’s a possible idea for easing your matrix usage. I think you only need to maintain a single matrix for your data.

– Pasted Max Patch, click to expand. –

Re: what should i use as list?

Wetterberg — Sat, 09 Mar 2013 18:03:30 +0000

It could also be encoded into around five seconds of 5-bit audio, I suppose, using poke~ and index~.

Re: what should i use as list?

karl krach — Sat, 09 Mar 2013 19:36:54 +0000

cool, thanks!
i’m currently fiddling around with some other stuff but i will get back to that…!
k