avantronica

I want to store subgroups of data in excess of the 33k limitations of some objects in the patch (not in an external file)

Okay - 'big data' is an exaggeration (but 

 limits) - but I wonder what I'm missing, I'm sure the answer is there, but I cannot find it despite searching a lot, nor have I found a good workaround 

I'm using jit.matrix to handle the data I may have previously shuffled about in a clunky manner , so I'm less familiar with the wealth of jitter possibilities (used to zl for this stuff and maybe pattr)- but I want to make a data management system for data and jitter's the way forward for sure

however the data I collect into the ideal size subgroups is ~50k bytes (it's 8bit) and I may want to hold 500>1000k maximum within the patch - the data will be changing/updated from time to the time

However, the jitter matrixset doesn't seem to offer much to work with and the info I'm processing with jit.matrix is ideally in excess of the ~32.8k limit for most objects, like pattr/zl etc - so I'm wondering about the most elegant way to keep the data in the patch (i'd rather avoid loading/saving to external files if possible) - workarounds even discuss using buffer~ but converting/reconverting doesn't feel right somehow.

Have I missed or misunderstood something or missed a good thread on this ? 

Is there a good way to store >33k Int8 data in a patcher that isn't clunky, I don't want to engineer a workaround to discover I overlooked something in there already (not too keen on relying on externals as a rule either) .. any pointers/corrections welcome

 I want to store subgroups of data in excess of the  33k limitations of some objects in the patch (not in an external file)

Okay - 'big data' is an exaggeration (but big relative to apparent limits) - but I wonder what I'm missing, I'm sure the answer is there, but I cannot find it despite searching a lot, nor have I found a good workaround 

I'm using jit.matrix to handle the data I may have previously shuffled about in a clunky manner , so I'm less familiar with the wealth of  jitter possibilities (used to zl for this stuff and maybe pattr)- but I want to make a data management system for data and jitter's the way forward for sure 

however the data I collect into the ideal size subgroups is ~50k bytes (it's 8bit) and I may want to hold 500>1000k maximum within the patch -  the data will be changing/updated from time to the time

However, the jitter matrixset doesn't seem to offer much to work with and the info I'm processing with jit.matrix is ideally in excess of the ~32.8k limit for most objects, like pattr/zl etc - so I'm wondering about the most elegant way to keep the data in the patch (i'd rather avoid loading/saving to external files if possible) - workarounds even discuss using buffer~ but converting/reconverting doesn't feel right somehow. 

Have I missed or misunderstood something or missed a good thread on this ? 
Is there a good way to store >33k Int8 data in a patcher that isn't clunky, I don't want to engineer a workaround to discover I overlooked something in there already (not too keen on relying on externals as a rule either)   .. any pointers/corrections welcome 

 I want to store subgroups of data in excess of the  33k limitations of some objects in the patch (not in an external file)
Okay - 'big data

embedding-'big-data'

It's never a good idea to store any significant amount of data in program code. Just deal with external files and maybe your problems go away. :-)

Well, I'd say that this depends on the kind of data you have to keep... and, in the Max context, the line between data and program code is blurred...

One option to keep it all in the patch is using coll or dict (the choice mainly depends on how you plan to access it). buffer~ is a very good option for floating point numbers, and you don't need any conversion—just look at the poke~ object.

Or you can check out the bach package, which among the other stuff provides a set of objects dealing with Max-like (albeit potentially more complex) messages without size limitation.

okay, thanks y'all - I'll have to think this through again - not keen on files, but presumably the default objects have working 'storage' limitations for good reason

are you saying that (each) pattrstorage has a limited storage capacity of 32.8kb?

which means it can store up to 32800 ints?

and how would that translate to other types of data? 

for example i'm storing mostly [numbox]'s and [live.number]'s (all ints?), but also various buttons (would that be boolean-1bit, or int again?), and some [textedit]'s as well (string? ie: 1byte/char?)

are you saying that (each) pattrstorage has a limited storage capacity of 32.8kb?
which means it can store up to 32800 ints?
and how would that translate to other types of data? 
for example i'm storing mostly [numbox]'s and [live.number]'s (all ints?), but also various buttons (would that be boolean-1bit, or int again?), and some [textedit]'s as well (string? ie: 1byte/char?)

yeah - all the objects which handle data that way are similarly limited - I guess pattrstorage still uses a byte even if the object doesn't need a byte, but who knows - I'd only been envisioning linking pattrstorage to one object - presumably further bytes are used up to identify the object and so on if you are storing for many objects in a patcher

Embedding 'big data' ?