Performance problem with writing lots of real-time data to a text file
I am debugging a Max crash. I need to log all the real-time data to disk so that when Max crashes not even a single line is missing.
Currently, I am sending data via UDP and capture it with a PowerShell script so that Max doesn't have to do any heavy lifting. I have tried [capture] and [text] objects but writing each new value immediately to disk totally overwhelmed Max to the point of not responding to any user input in the patcher (I was able to turn off the scheduler from the alt menu though).
The PowerShell workaround works fine but I am wondering if I am missing something obvious that I can setup directly in Max and not use workarounds for this.
Any tips?
You could use node.js for this, using the builtin fs.appendFile or a fs stream (which I believe is more efficient as the file is opened once).
For those who don't download .zip, here is the code (to be saved as data_stream_record.js):
const Max = require('max-api');
const fs = require('fs');
const filePath = 'my-file.txt';
const stream = fs.createWriteStream(filePath, {flags: 'a'});
Max.addHandler("append", (msg) => {
stream.write(Date.now() + ' | ' + msg + '\n', 'utf8');
});And the patch
I guess for more accurate timestamps it would make more sense to prepend them in Max directly rather than in the script.
Thanks TFL, but wouldn't the node script get killed when Max crashes to desktop?
Yes but that doesn't matter because incoming data gets written on the file as it comes.
I guess what doesn't work with [text] (but would be the same with [coll] or [dict]) is that the entire text is saved over and over, even though there is just one more line.
The advantage of stream.write() is that it just appends the text you pass to it in the file, and that's all.
Normally you would need to stream.end() to properly "close" the file, but from my testings crashing Max didn't corrupt the text file, so seems all good!
I can imagine a theoretical event-order scenario where the last line isn't written because of a crash. That's precisely why I went with Powershell intuitively, because I could be sure Powershell does not crash together with Max. But I haven't analyze the theory properly yet. Just thinking aloud.
The problem in first instance is the crash itself! That last line being registered or not really depends on what makes your patch to crash and when.
If you know the culprit then you can make sure the line to store arrives to node.script before the crashing part of the patch, and it should be good. I can verify that by causing crash on purpose:
The crash happens right after the 'precrash' message has been sent, and yet it is correctly recorded in the file.
Otherwise, if you don't know when the patch crashes, I would say there is no more guaranty that the last line is being sent via UDP than written on disk with node.script.