Dan Nigrin

I am working on some crazy sysex file manipulation that I'll spare you the details on. But basically, it involves manipulating portions of 5 bytes worth of data to fit into 4 bytes. The exact details of which bits of the source 5 bytes, and where they end up in the destination 4 bytes, is included in the attached patch below. It uses the Bit and Lbyte objects from Peter Elsea's Lobjects package, FYI.

This works just fine, however I have to run this operation over 20 thousand times, and the result is quite slow. Can anyone suggest a different approach (either using regular Max objects, or perhaps Javascript?) to help speed things up?

unpack

inlet

buddy

outlet

comment

lbyte

message

I am working on some crazy sysex file manipulation that I'll spare you the details on.  But basically, it involves manipulating portions of 5 bytes worth of data to fit into 4 bytes.  The exact details of which bits of the source 5 bytes, and where they end up in the destination 4 bytes, is included in the attached patch below.  It uses the Bit and Lbyte objects from Peter Elsea's Lobjects package, FYI.

This works just fine, however I have to run this operation over 20 thousand times, and the result is  quite slow.  Can anyone suggest a different approach (either using regular Max objects, or perhaps Javascript?) to help speed things up?  

I am working on some crazy sysex file manipulation that I'll spare you the details on.  But basically, it involves manipulating portions of 

faster-bitwise-manipulation

Try this one, it uses expr and bitwise arithmetic to do the job, also simplifies message passing, so it may be a bit (!) faster.

Thanks! I *love* the elegance of this approach (and the fewer objects!), but sadly, after testing it takes roughly the same amount of time as my current approach. I think that expr is an "expensive" object, resource-wise.

Thanks!  I *love* the elegance of this approach (and the fewer objects!), but sadly, after testing it takes roughly the same amount of time as my current approach.  I think that expr is an "expensive" object, resource-wise.

awesome expr code from the above example in Javascript was successful, but in fact performed more poorly, took almost twice the time:

var in1;
var in2;
var in3;
var in4;
var in5;
var out1;
var out2;
var out3;
var out4;

function list () {
    in1 = arguments[0];
    in2 = arguments[1];
    in3 = arguments[2];
    in4 = arguments[3];
    in5 = arguments[4];
 

    out1 = (in1 & 127) | ((in2 & 1) << 7);
    out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6);
    out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5);
    out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4);
 

    outlet (0,out1, out2, out3, out4);
}

Ugh, trying to replicate @KLSDIZ 's awesome expr code from the above example in Javascript was successful, but in fact performed more poorly, took almost twice the time:

function list () {
	in1 = arguments[0];
	in2 = arguments[1];
	in3 = arguments[2];
	in4 = arguments[3];
	in5 = arguments[4];
 

	out1 = (in1 & 127) | ((in2 & 1) << 7);
	out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6);
	out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5);
	out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4);
 

Sorry, I take that back - I made an error in my patching, it didn't take twice as long, it took roughly the same amount of time as the other two versions...

For kicks, I tried to see if not using expr made a difference, and it did not, again roughly the same time (about 30 seconds for 20,000 iterations):

Any other ideas out there?? Feels like there's 

Sorry, I take that back - I made an error in my patching, it didn't take twice as long, it took roughly the same amount of time as the other two versions...  

Any other ideas out there??  Feels like there's gotta be a faster way somehow!!

I think you have a bottleneck somewhere else. Using either expr or max bitwise op objects takes about 30ms on my machine (i5 from 2015), js 80ms (as expected – I believe the calculations are done efficiently because of JIT, but repackaging data between js and max takes some time). If you want to go faster, you may have to write your external (OTOH a trivial one).

bitman_20190727_002931.maxzip

We're on the same wavelength; as I was sitting in traffic this morning, thinking about the problem, I realized that I should probably look elsewhere. Most likely how I'm fetching the values (from a sysex file on disk) to send into this bit manipulation stuff. (And BTW, thanks I saw your comment about the local vars in the Javascript! :-) ).

Will mess around this weekend, hopefully I'll find a faster way than what I'm doing now. I really appreciate your help and discussion on this!

We're on the same wavelength; as I was sitting in traffic this morning, thinking about the problem, I realized that I should probably look elsewhere.  Most likely how I'm fetching the values (from a sysex file on disk) to send into this bit manipulation stuff.  (And BTW, thanks I saw your comment about the local vars in the Javascript!  :-)  ).  

Will mess around this weekend, hopefully I'll find a faster way than what I'm doing now.  I really appreciate your help and discussion on this!

Here's my master patch where I was comparing all of the different approaches. It includes how I'm reading the sysex file from disk. The counter range represents the bytes I have to read. If you (or anyone out there) can think of a better/faster way to read the bytes out of the file, I'm all ears. Was thinking that maybe if I just read all the bytes into a coll or text object, and then worked on that? But doubt that would be faster?

Here's my master patch where I was comparing all of the different approaches.  It includes how I'm reading the sysex file from disk.  The counter range represents the bytes I have to read.  If you (or anyone out there) can think of a better/faster way to read the bytes out of the file, I'm all ears.  Was thinking that maybe if I just read all the bytes into a coll or text object, and then worked on that?  But doubt that would be faster?


MBP 2012 i7 10.11.6 Max 8.06
seems dumping a coll faster
zzz


Thanks - but the issue is that I have to get the binary file data out of the sysex file 

, and only then can I worry about reading it in 5-byte increments. I'm blanking at the moment, but don't think there's any other way to read in binary file data, is there? Or perhaps that's where I look to Javascript??

Thanks - but the issue is that I have to get the binary file data out of the sysex file first, and only then can I worry about reading it in 5-byte increments.   I'm blanking at the moment, but don't think there's any other way to read in binary file data, is there?  Or perhaps that's where I look to Javascript??

any chance to read, process, and write files in groups of 25 instead of one by one?

Well, I have to process them 5 bytes at a time, but I can certainly read more in a time if that's possible...

I'm going to try and see if reading the file in and doing all the processing within Javascript will be faster: 

https://docs.cycling74.com/max8/vignettes/jsfileobject

I'm going to try and see if reading the file in and doing all the processing within Javascript will be faster:  https://docs.cycling74.com/max8/vignettes/jsfileobject 

what i mean is that if you f.e. need to invert a single bit, and you do it for 200 input files at once, you could process them as a list. no idea if it makes sense in your context. :)

do you need to do a lot of search and replace? then mabye the [filein] layer could be the one you want to work with.

what i mean is that if you f.e. need to invert a single bit, and you do it for 200 input files at once, you could process them as a list. no idea if it makes sense in your context. :)

do you need to do a lot of search and replace? then mabye the [filein] layer could be the one you want to work with.


replacing deferlow with del makes your patch faster. For me, the limit of coll is the list length of 256 for "big"sysex dumps. Filein takes a file in RAM, i think it must can be fast ?

but coll has definitely no limit of 256 per line.

Nothing to do with speed of execution,
but coll has definitely no limit of 256 per line.

yess ! and message too. My oldtimer meaning... I had a "zl join", not "zl 1024 join" in my path :-)

yess ! and message too. My oldtimer meaning... I had a "zl join", not "zl 1024 join" in my path :-)
bzzz


Because if you take it out, you get a stack overflow when I run my big sysex file through it...

I *will* say that I too see a big speed gain with the del rather than the deferlow, so am now doing testing to see if the output is still OK. Thanks to all!

I *will* say that I too see a big speed gain with the del rather than the deferlow, so am now doing testing to see if the output is still OK.  Thanks to all!

Nope - using the del rather than deferlow mangles the output up. I don't completely understand *why* that's the case, but regardless, no go. Will keep trying on the Javascript side...

Nope - using the del rather than deferlow mangles the output up.  I don't completely understand *why* that's the case, but regardless, no go.  Will keep trying on the Javascript side...

a last idea breaking an uzi... and loading this .syx to separate the sysex messages inside ( i'm unable to think js....)

bcf2000_1-10.syx.zip

a last idea breaking an uzi... and loading this .syx to separate the sysex messages inside ( i'm unable to think js....)


oups an error in first line: initiate the uzi at 0 sorry

Thanks! But too late, I've solved it now in Javascript - I haven't timed it, but it's basically just about instantaneous (versus the ~30 seconds that the previous way took). I've just pulled the file reading and segmenting into 5 byte chunks all within Javascript. Here's the Javascript, and the patch:

inlets = 1;
outlets = 1;

function fivetofour (filename) {
    f = new File(filename);
    var in1;
    var in2;
    var in3;
    var in4;
    var in5;
    var out1;
    var out2;
    var out3;
    var out4;
    
    data = new Array();
 
    data = f.readbytes(12); //don't need these bytes, throw away
     
    do {
        data = f.readbytes(5);
        
         in1 = data[0];
        in2 = data[1];
        in3 = data[2];
        in4 = data[3];     
            
        out1 = (in1 & 127) | ((in2 & 1) << 7);
        out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6);
        out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5);
        out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4);
         
         outlet (0,out1, out2, out3, out4);
       }
    while (f.position < 102416);
   
       f.close;
}



Thanks!  But too late, I've solved it now in Javascript - I haven't timed it, but it's basically just about instantaneous (versus the ~30 seconds that the previous way took).  I've just pulled the file reading and segmenting into 5 byte chunks all within Javascript.  Here's the Javascript, and the patch:

inlets = 1;
outlets = 1;

function fivetofour (filename) {
	f = new File(filename);
	var in1;
	var in2;
	var in3;
	var in4;
	var in5;
	var out1;
	var out2;
	var out3;
	var out4;
	
	data = new Array();
 
	data = f.readbytes(12); //don't need these bytes, throw away
 	
	do {
		data = f.readbytes(5);
		
	 	in1 = data[0];
		in2 = data[1];
		in3 = data[2];
		in4 = data[3]; 	
			
		out1 = (in1 & 127) | ((in2 & 1) << 7);
		out2 = ((in2 & 126) >> 1) | ((in3 & 3) << 6);
		out3 = ((in3 & 124) >> 2) | ((in4 & 7) << 5);
		out4 = ((in4 & 120) >> 3) | ((in5 & 15) << 4);
 		
 		outlet (0,out1, out2, out3, out4);
   	}
	while (f.position < 102416);
   
   	f.close;
}



Great you solved it! Just looking at the code – you don't need 

as you assign other objects in next steps, but you can benefit from declaring 

 inside function body to make it local. Have a good Sunday!

Great you solved it! Just looking at the code – you don't need data = new Array() as you assign other objects in next steps, but you can benefit from declaring var data; inside function body to make it local. Have a good Sunday!

Thanks for the tip - I'm very much of a Javascript noob, so it's very much appreciated!

Faster bitwise manipulation