SOLVED: How to use File Object to write a MIDI syx file?

Fora's icon

Hi peeps,

I am in the middle of creating a Sysex librarian to edit and write patches that one could load into any sysex editor or dump directly to a synthesiser. I have has some success in using jit.matrix to write a JIT binary file and manually changing the file extension to syx, but that's more of a hack and not a viable solution as I am hoping to distribute my final patch. (Max for live)

After having searched the forum for a few hours, my search to find a solution to write syx (system exclusive) files has lead me to the File Object but I'm hopelessly stuck as Javascript isn't my strong suit and was hoping to pick one of your fine brains!

Would anybody know the best way to get data stored in [capture] and save it as a syx file using JS that isn't written in text format but binary, or can I still use jit.matrix with File Object in JS to write binary files with a different extension?

p.s. I have looked at the mxj.fileout solution but it requires installing Java which Im trying to avoid.

Thanks in advance

Fora's icon

Ok, I figured out a solution using [Savedialog] using MIDI filetype to Jit.Matrix for anybody else who is struggling to export syx files.

1) [capture] - [jit.fill] - [jit.matrix]
2) [savedialog] - [jit.matrix]
3) ???
4) profit!

Dan Nigrin's icon

Funny, I just had a very similar need, and didn't come across your post after my initial searches. I've just solved it using Javascript, in case it's still of interest to you - hopefully it's intuitive enough - prepend "file_to_write" in front of full path to your file and send to Javascript, and then prepend "writebyte" in front of each actual byte you want to store. I'm sure it's easily extensible if you have a byte list that you'd like to send to it instead of individual bytes one by one.

inlets = 1;
var f;
function file_to_write (filename) {
    f = new File(filename, "write", "Midi");
}
function writebyte (thebyte) {
    f.writebytes(thebyte);
}
Fora's icon

That's great Dan! I was searching online for a javascript solution but couldn't figure out how to implement one due to my poor understanding of JS.

So if I wish to output modified data of varying lengths, do I have to declare it beforehand?

Do you have working patch I could look at?

Fora's icon

Another question, is it correct to connect the javascript object outlet to jit.matrix? Or are you sending the writebyte (JS) to another object, just having an issue of seeing any bytes appear on a test file which I managed to create.

Dan Nigrin's icon

Here's an example patch - obviously without the full sysex message, but just a few bytes as an example. I don't have time at the moment to change the Javascript to accept the list input directly, but I can't imagine that's too difficult (I'm a relative Javascript noobie too)....

Max Patch
Copy patch and select New From Clipboard in Max.

Fora's icon

It works, very elegant solution! many thanks!

Fora's icon

Would you know a way to add filename extensions directly into the JS code above, ideally syx?

Fora's icon

Ok I have also noticed an issue when writing bytes, in which they sometimes do and other times don't.

I am sending about 8kb of data to be written but regardless of the size, bytes aren't always written.

I guess it's because the JS script isn't holding bulk data to be written, like you stated, it isn't meant for lists.

Dan Nigrin's icon

Yeah - there's two things I usually do - I provide a default filename ending in .syx to the savedialog - this automatically selects the "MIDI Sysex" option in the file dialog. But there's nothing keeping a user from stripping that extension off, so I also add a bit of code that forces the addition of a .syx extension, if it doesn't exist already. See attached patch.

There's also sexy ways of doing this (ensuring the .syx extension) with regexp, but I always have a hard time wrapping my head around regexp, so I stick with my clunkier approach you see here. Hope this helps.

Max Patch
Copy patch and select New From Clipboard in Max.

Fora's icon

Cool, I like the default.syx which forces the syx extension to be on top of the list, also the sprintf is a nice touch.

Ok, i'm now trying to figure out how to write a long list of bytes. if I find a solution, I'll post it here.

Cheers Dan!

Dan Nigrin's icon

Yes I think you're correct re: your problem with it not writing all the data - definitely look into modifying the Javascript to accept a list input, then deliver it all the data at once to write. Will probably be more reliable.

Fora's icon

Ok I think I have found a solution.

inlets = 1;
var f;
function file_to_write (filename) {
    f = new File(filename, "write", "Midi");
}
function list()
{
    for (i=0; i < arguments.length; i++)
    {
    f.writebytes(arguments[i]);
    }
}

no need to prepend any messages before a list.

Fora's icon

Update*

I am having the same issues as before.

It seems this line "f = new File(filename, "write", "Midi");" is creating a file but for some reason, the rest of the code isn't writing data all the time.

Maybe because the two separate functions are running one after another perhaps?

Dan Nigrin's icon
  1. When you deliver the full list to the Javascript method, I don't think you have to write the bytes 1 by 1; you need to convert the incoming list into a Javascript array, and then I believe the writebytes method will write them all at once.

  2. You may need to deferlow the byte writing portion of your patch, to allow for the file creation operation to complete before writing to it. Maybe something like this, just off the top of my head, not sure if will work. If it doesn't you may have to send a bang out of the Javascript object when you are sure that the file has been successfully created, perhaps by using the "isopen" method? https://docs.cycling74.com/max5/vignettes/js/jsfileobject.html

Max Patch
Copy patch and select New From Clipboard in Max.

Fora's icon

Deferlow doesn't work here, I believe it's an issue with the file not closing when the write_to_file function is called, so with some further investigation, I came across (File.close) which closes any old open file in memory.

It seems to be working fine so far :)

I have also figured out how to include .syx within the JS itself, so all you need is a [savedialog] without any filetype message.

Here's the patch and JS code below.

Max Patch
Copy patch and select New From Clipboard in Max.


inlets = 1;
var f;
function file_to_write (filename)
{
    if(f) f.close();
    f = new File(filename+".syx", "Midi");
}
function list()
{
    for (i=0; i < arguments.length; i++)
    {
    f.writebytes(arguments[i]);
    }
}

Dan Nigrin's icon

Not sure why that's working, as you're calling the close() function on a variable whose type hasn't been defined yet? You only declare it as a File object in the line that follows... ?

But anyway, if it's working, I guess that's all that really matters! ;-)

Fora's icon

I believe it's to close that variable that is still open when calling a new function, if that makes sense.

So without the close() function, the previous JS script worked first time around, but the second time it wouldn't close the file after it was created as it was still open and no bytes were written.

I think that's how it's working, lol..

I'm in the same boat, I'm a total n00b!

But thanks for chiming in, you've really helped me with this one!

Dan Nigrin's icon

Ah OK, I see what you're saying. In that case, it might make more sense to put the close after you're done writing to the File, something like this. This is like the blind leading the naked, maybe a Javascript expert will come in and teach us *both* something! ;-)

inlets = 1;
var f;

function file_to_write (filename)
{
    f = new File(filename+".syx", "Midi");
}

function list()
{
    for (i=0; i < arguments.length; i++)
    {
        f.writebytes(arguments[i]);
    }
f.close();
}

Fora's icon

This is like the blind leading the naked

haha, indeed!

Ok, this is my take on it (and if there are any JS experts willing to chime in and make sense of it, please do)

I believe the (close) resets the f variable(?) which then clears the memory for a fresh function call.
This was seen in the previous JS script I posted whereas it seemed to write the bytes the first time the code was initiated but the second attempt of writing the bytes, the script would fail.

Fora's icon

your amendment works great btw!

Dan Nigrin's icon

your amendment works great btw!

Glad to hear it, good luck with your project!

Fora's icon

cheers, yours too!

kLSDiz's icon

By no means expert here, but I would keep a filename in a global var and do all the interaction with file system in one method/function.

Dan Nigrin's icon

Yes, definitely cleaner, thank you!

Fora's icon

apologies for the basic question, what would that involve code wise out of interest?

Dan Nigrin's icon

Like this:

inlets = 1;
var f;
var fn;

function file_to_write (filename)
{
    fn = filename;
}

function list()
{
f = new File(fn+".syx", "Midi");
    for (i=0; i < arguments.length; i++)
    {
        f.writebytes(arguments[i]);
    }
f.close();
}

Fora's icon

Ah gotcha! that is much neater.

I have discovered something else now, I'm not using the savedialog but just a message with a name for the file. Now if I write some bytes (say 1 kb worth) and decided to write over the same file but with a smaller amount of data, the file size stays the same.

Do either of you know a way to include a replace message somehow?

Dan Nigrin's icon

Don't know how to do that. I do know that you can't delete a file with straight Javascript, maybe with Node.js?

If you go back to using savedialog, and it prompts you that you already have a file with that name, does it do the overwriting as you would expect?

Fora's icon

I believe it does overwrite it with the savedialog prompt, but because I have included the file extension as part of the script, it writes an extra .syx at the end of the filename!

funny, solving one issue only presents another, It's like a leaking ship lol

kLSDiz's icon

You could also keep data inside the js object and provide filename only when necessary. I think it's more max-ish approach – other container objects (eg. table, buffer~) behave in similar way.

var data;

// reset data
function clear() {
    data = [];
}

// append to data
function append() {
    for(i in arguments)
            data.push(arguments[i]);
}

// replace data
function list() {
    clear();
    for(i in arguments)
            data.push(arguments[i]);
}

// write data to file
function write(fn) {
    var f = new File(fn, "write", "Midi");
    if (f.isopen) {    
        f.writebytes(data);
        f.close();
    } else {
        error("Cannot open file: "+fn+"\n");
    } 
       
}

// init
function loadbang() {
    clear();
}
Fora's icon

Thanks for your input KLSDIZ!

I am trying out the code but I am still struggling to find a way to replace a file with a smaller one in size, the file size doesn't change. the savedialog warns before replacing but it doesn't seem to delete the previous, instead bytes are written on top of the previous file.

Any ideas in what I could try?

Fora's icon

Wanted to also say thanks, I was struggling to work out how to contain the data inside the JS object.
much appreciated!

Fora's icon

ok, adding "f.eof = f.position;" before f.close(); has sorted out the resizing issue.

I have one more issue which I am trying to get my head around, its to do with filenames, specifically when writing over the same file with the same name, because I have added a .syx prefix to the JS code, when I click on the file I wish to overwrite (using the savedialog object) the filename's extension is duplicated.

I was wondering if you know of a way to strip the extensions of a file?

Source Audio's icon

write file name with extension, like write MySysex-1.syx
and remove
f = new File(fn+".syx", "Midi");

Fora's icon

Here is a working solution. I think it's pretty much done, I'm sure it can be optimised but for now it does the job. I hope it's useful to somebody!

I would really like to say thanks to everybody that helped me along the way!

--- here's the JS---

var data;

// reset data
function clear() {
    data = [];
}

// append to data
function append() {
    for(i in arguments)
        data.push(arguments[i]);
}

// replace data
function list() {
    clear();
    for(i in arguments)
        data.push(arguments[i]);
}

// write data to file
function write(fn) {
    var f = new File(fn);
    if (f.filetype == "Midi") {
        //    post("extension = .syx");
            var f = new File(fn, "write", "Midi");
    }else{
        //    post("extension = blank");
            var f = new File(fn+".syx", "write", "Midi");
         }
    if (f.isopen) {
        f.writebytes(data);
        f.eof = f.position;
        f.close();
    } else {
        error("Cannot open file: "+fn+"\n");
    }
}

// init
function loadbang() {
    clear();
}

Dan Nigrin's icon

Nice work and thank you to all who helped.

Fora's icon

@Dan.. I was hoping I could get your email as I have some Max for Live project ideas and wondered if you would be interested in collaborating on?

Dan Nigrin's icon

Of course - dan@defectiverecords.com , and take a look at https://defectiverecords.com if you've not already...

Fora's icon

Awesome! yeah defective, I'm familiar with it. Didn't realise you were behind it! Cool I'll drop you an email

Fora's icon

Calling @Dan & @KLSDIZ (or anybody else with JS experience) !

I am facing an issue with the current code, which is working swell for small files but I'm working with
larger data sets now (on a different project) and it doesn't write any bytes to disk, I just get zero byte files.

I tried using the append (function in JS) and [zl group] but it crashes the patch.

Does anybody know what the size limitations for large arrays, lists are and if there's a way to work around that?

Cheers

Roman Thilenius's icon

for zl you must set an argument for maximum list elements (up to 32,000), otherwise it defaults to 256 (like it used to be 15 years ago.)

Fora's icon

Thanks for the quick reply Roman, regarding the zl objects, I know this but I am trying to figure out in how to get around 32K limit of the writeline() method in JS

Dan Nigrin's icon

Sorry Fora, I'm afraid I don't have any knowledge of the JS limits....

Fora's icon

I think I am getting a little closer to finding a solution, I found this post which has a work around but I'm trying to figure out how to get my data in..

Dan Nigrin's icon

Aha! Nice that Peter Nyboer posted his solution - thanks for linking to that for future searches of the forum...

Fora's icon

Yeah it’s a great little find!

It definitely puts me within the ball park but I’m still scratching my head.

Any ideas in what would need changing for it to store a dump from the capture object and write those instead?

Dan Nigrin's icon

Sorry, you're far ahead of me on this one!

Fora's icon

No prob! I will defo post a solution as soon as I come across one :)