Holiday notice: Cycling '74 will be closed 3 July, 2020. Manual authorization, support cases, and manual order processing will be delayed.
Faster bitwise manipulation
Jul 25 2019 | 3:24 pm
I am working on some crazy sysex file manipulation that I'll spare you the details on. But basically, it involves manipulating portions of 5 bytes worth of data to fit into 4 bytes. The exact details of which bits of the source 5 bytes, and where they end up in the destination 4 bytes, is included in the attached patch below. It uses the Bit and Lbyte objects from Peter Elsea's Lobjects package, FYI.
This works just fine, however I have to run this operation over 20 thousand times, and the result is quite slow. Can anyone suggest a different approach (either using regular Max objects, or perhaps Javascript?) to help speed things up?
Thanks in advance!
- Jul 25 2019 | 5:15 pmTry this one, it uses expr and bitwise arithmetic to do the job, also simplifies message passing, so it may be a bit (!) faster.
- Jul 25 2019 | 8:45 pmThanks! I *love* the elegance of this approach (and the fewer objects!), but sadly, after testing it takes roughly the same amount of time as my current approach. I think that expr is an "expensive" object, resource-wise.
- Jul 26 2019 | 12:19 amUgh, trying to replicate @KLSDIZ 's awesome expr code from the above example in Javascript was successful, but in fact performed more poorly, took almost twice the time:
inlets = 1; outlets = 1;var in1; var in2; var in3; var in4; var in5; var out1; var out2; var out3; var out4;function list () { in1 = arguments[0]; in2 = arguments[1]; in3 = arguments[2]; in4 = arguments[3]; in5 = arguments[4];out1 = (in1 & 127) | ((in2 & 1) << 7); out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6); out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5); out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4);outlet (0,out1, out2, out3, out4); } - Jul 26 2019 | 12:42 amSorry, I take that back - I made an error in my patching, it didn't take twice as long, it took roughly the same amount of time as the other two versions...For kicks, I tried to see if not using expr made a difference, and it did not, again roughly the same time (about 30 seconds for 20,000 iterations):Any other ideas out there?? Feels like there's gotta be a faster way somehow!!
- Jul 26 2019 | 10:34 pmI think you have a bottleneck somewhere else. Using either expr or max bitwise op objects takes about 30ms on my machine (i5 from 2015), js 80ms (as expected – I believe the calculations are done efficiently because of JIT, but repackaging data between js and max takes some time). If you want to go faster, you may have to write your external (OTOH a trivial one).
- Jul 27 2019 | 12:52 amWe're on the same wavelength; as I was sitting in traffic this morning, thinking about the problem, I realized that I should probably look elsewhere. Most likely how I'm fetching the values (from a sysex file on disk) to send into this bit manipulation stuff. (And BTW, thanks I saw your comment about the local vars in the Javascript! :-) ).Will mess around this weekend, hopefully I'll find a faster way than what I'm doing now. I really appreciate your help and discussion on this!
- Jul 27 2019 | 1:31 amHere's my master patch where I was comparing all of the different approaches. It includes how I'm reading the sysex file from disk. The counter range represents the bytes I have to read. If you (or anyone out there) can think of a better/faster way to read the bytes out of the file, I'm all ears. Was thinking that maybe if I just read all the bytes into a coll or text object, and then worked on that? But doubt that would be faster?
- Jul 27 2019 | 7:27 amoups maybe sent twice ? bzzz
- Jul 27 2019 | 7:38 amMBP 2012 i7 10.11.6 Max 8.06 seems dumping a coll faster zzz
- Jul 27 2019 | 11:41 amThanks - but the issue is that I have to get the binary file data out of the sysex file first, and only then can I worry about reading it in 5-byte increments. I'm blanking at the moment, but don't think there's any other way to read in binary file data, is there? Or perhaps that's where I look to Javascript??
- Jul 27 2019 | 8:00 pmany chance to read, process, and write files in groups of 25 instead of one by one?
- Jul 27 2019 | 9:19 pmWell, I have to process them 5 bytes at a time, but I can certainly read more in a time if that's possible...I'm going to try and see if reading the file in and doing all the processing within Javascript will be faster: https://docs.cycling74.com/max8/vignettes/jsfileobject
- Jul 27 2019 | 11:42 pmwhat i mean is that if you f.e. need to invert a single bit, and you do it for 200 input files at once, you could process them as a list. no idea if it makes sense in your context. :) do you need to do a lot of search and replace? then mabye the [filein] layer could be the one you want to work with.
- Jul 28 2019 | 7:07 amreplacing deferlow with del makes your patch faster. For me, the limit of coll is the list length of 256 for "big"sysex dumps. Filein takes a file in RAM, i think it must can be fast ?
- Jul 28 2019 | 7:56 amNothing to do with speed of execution, but coll has definitely no limit of 256 per line.
- Jul 28 2019 | 11:33 amyess ! and message too. My oldtimer meaning... I had a "zl join", not "zl 1024 join" in my path :-) bzzz
- Jul 28 2019 | 12:03 pmwhy del/defer at all?
- Jul 28 2019 | 12:03 pmBecause if you take it out, you get a stack overflow when I run my big sysex file through it...
- Jul 28 2019 | 12:08 pmI *will* say that I too see a big speed gain with the del rather than the deferlow, so am now doing testing to see if the output is still OK. Thanks to all!
- Jul 28 2019 | 12:24 pmNope - using the del rather than deferlow mangles the output up. I don't completely understand *why* that's the case, but regardless, no go. Will keep trying on the Javascript side...
- Jul 28 2019 | 1:25 pma last idea breaking an uzi... and loading this .syx to separate the sysex messages inside ( i'm unable to think js....)
- Jul 28 2019 | 1:32 pmoups an error in first line: initiate the uzi at 0 sorry
- Jul 28 2019 | 3:03 pmThanks! But too late, I've solved it now in Javascript - I haven't timed it, but it's basically just about instantaneous (versus the ~30 seconds that the previous way took). I've just pulled the file reading and segmenting into 5 byte chunks all within Javascript. Here's the Javascript, and the patch:
inlets = 1; outlets = 1; function fivetofour (filename) { f = new File(filename); var in1; var in2; var in3; var in4; var in5; var out1; var out2; var out3; var out4; data = new Array(); data = f.readbytes(12); //don't need these bytes, throw away do { data = f.readbytes(5); in1 = data[0]; in2 = data[1]; in3 = data[2]; in4 = data[3]; out1 = (in1 & 127) | ((in2 & 1) << 7); out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6); out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5); out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4); outlet (0,out1, out2, out3, out4); } while (f.position < 102416); f.close; } - Jul 28 2019 | 3:04 pmThanks to everyone for contributing!!
- Jul 28 2019 | 4:34 pmGreat you solved it! Just looking at the code – you don't need data = new Array() as you assign other objects in next steps, but you can benefit from declaring var data; inside function body to make it local. Have a good Sunday!
- Jul 28 2019 | 8:56 pmThanks for the tip - I'm very much of a Javascript noob, so it's very much appreciated!
