Help! Sudden Intermittent Serial Unstability
Hey All,
I have a big project using an Arduino Mega 2560. It's been up several times before with little to no issues however this time, Max seems to freeze or crash (this of course happened on opening night). What's stranger is Max seemed to crash like clock work, crashing after about one minute, which I feel might make it less likely if it was an electronics issue (but I could be wrong).
I'm using an Arduino Mega with a modified version of the ArduinoForDummies library for the Mega 2560, and some tweaks pertaining to this specific project. I've tested most of the hardware (I have a custom shield, and swapped various cables /components, etc but the electronics is fairly complex). however, I ran a couple tests and strongly believe the issue is Max. I tried two Arduino Megas, with different USB cables, two different computers and without the system hooked up. In one case, I had Max freezing / crashing after one minute of just connecting the Arduino with no messages being really sent. I've read on the forums that some people have had serial stability issues before with Max, which I haven't.
I tried earlier versions of the patch and same thing happened (perhaps my patch was corrupt?). I wrote just the serial portion of the the patch from scratch and while it seemed to fix the one minute crash, the system still crashed from about 5 - 10 minutes after running. I also tried Max 6.1, same issue.
I tried installing the FTDI driver from the site vs using Apples built in one, this did not fix it.
I tried different baud rates, I tried the chunk argument. I have 16 inputs and 16 outputs, but I'm just testing the outputs as that seems to be what's crashing. Once again, set this up several times before, no issues.
I have a truncated patch as the final patch is huge and it's essentially a 16 channel output blink sketch. The outputs are actually timed with MIDI notes and the note~ but I removed that part of the code for simplification and my own debugging. One thing I notice is that even though the outputs are on / off for 1000 ms sometimes they hang for longer than 1000 ms before tuning off (sometimes up to 4000 ms).
Macbook Pro (I think mid 2010)
OSX 10.10.5
Processor: 2.8 GHz Intel Core 2 Duo
Memory: NVIDIA GeForce 9600 GT 512 MB
500GB SSD of available space
Max 7.0.6
Any ideas? My rep is crumbling as we speak.
Attached is my Arduino sketch and patch. Any help would be greatly appreciated as the situation is pretty dire and there's a chunk of money in all of this.
Hi Duffield,
it could be an OS X 10.10 issue. Do you have a 10.9 machine to test on ?
Also check memory usage, maybe something is leaking.
That's only indirecly related, but could you move from serial to ethernet connection? in general i find serial connections to unreliable for live use and banned them from all my works...
jan
intriguing question.
do you have handshake serial ?
or
could be a hardware power issue.
try to go wireless from mac to arduino
to debug current draw from the usb port.
Does it still crash if you close then immediately reopen the serial port shortly before the crash happens (eg. you mention ten minutes - what happens if you close then open every 9 minutes - maybe not a solution, but it might help to diagnose the problem). I used to have problems like this all the time with Max (c. 2011), and I found this was one of its strange quirks, More recent versions have been fine for me.
Have you tried a scaled down patch with a smaller/normal Arduino - both one of the FTDI ones and the newer Unos?
I too have had excellent stability with an ethernet shield on the MEGA.
... these days i am prototyping some ideas for a new controller using an arduino leonardo and a serial connection on Max 7.0.6. As i am trying out different algorythms to convert the data from 6 sensors i ended up with sending around quite a lot of Max messages at a rate of about 10 ms. I also experienced thax Max freezes. I did not go into deep debugging, because i am still prototyping. but when I moved most data processing into jitter matices and replaced the data polling metro objects with qmetro the problem dissapeared.
btw - for me it was an almost embarrasing gotcha-heureka moment moving sensor data processing to matrices: much better performance and results! Why didn't I had this idea in the past - seems to be so basic now... !!! Only at the end of the chain, when I need them i.e. to control MIDI I grab what I need from the matrix and convert them back to Max messages.
j
Update: So it tends to not crash with nothing connected lately, which is good. However, when I run my original patch (which worked fine before!) It does crash. Looks like somehow my patch got corrupt so I gotta recode it...which is a major pain. Also doesn't explain why the Arduino crashed with nothing connected, but at least things might be turning up (hopefully). Some things are off with the performance of this all lately, so I'm in the process of working with an expert and he's making some significant changes (all of which I'll share if the results are positive and will be a significant uphaul to the ArdiunoInOutForDummies Library :) ) I just gotta take out the code that's more specialized to my specific project. Admittedly, I know a some Arduino on the C level, but I am at the mercy of a self modification of ArduinoInOutForDummies as a library in this case a this point (I couldn't have written something better from scratch)...which I've used in many installations for long periods of time (museum grade, 6 months running) with no problems.
@Jan - Nope didn't test on a 10.9 machine. Don't have access to one at the moment :/ Unfortunately both my machines are 10.10.5 and I worry about rolling back just with the amount of projects I have and things crapping out. That said, I found that I've had better results in general on 10.10 vs 10.9 in terms of performance. Also, in terms of dataprocessing into Jitter matrices, do you mean your just sending Jitter Matrices and something like [jit.spill] to the Arduino? The processing of this data isn't too intense, it's literally MIDI note messages converted to simple on/off switches. I will try this out though and see my results :) Also, I will try [qmetro], despite that timing is slightly important. I thought about trying this but was having some timing issues so didn't (which I should have anyways...why not).
@Ironside I haven't scaled down and done the tests...mainly because I need this specific project up and I NEED a Mega. I have a custom shield that was created so to go Ethernet, I don't know if I would have to totally redesign this shield to accommodate that :/ Definitely something to consider for the future if this project progresses (which it has to work...like it did before first :) )
Thanks for your input guys. Any other suggestions in the meantime are always appreciated of course.
In My case I am currently not sending any data to the Arduino, I am just receiving a list of sensor values in Max. First thing after receiving the list of data in Max I use [jit.fill] to write them into a named jitter matrix. All the smoothing, filtering and converting (i.e. changes of position data in to speed/velocity, direction of movement, angle... ) I do in jitter. I write those data into named matrices (for the position data, for speed/velocity, direction of the movement.).
Where i need the data to control things I pull them out of the matrices with [jit.submatrix] (only fetching the cells i am interested in) -> [jit.spill].
@ Jan
do you mind sharing the patch.
sounds a great idea to cue data in a matrix.
Hi Balam, it all quite basic, here is the part of the patch where i write the data into matrices:
I created several abstraction to read out the data. Here is one example for the positions-matrix:
J
UPDATE:
Hey guys, so I think I found the problem...which is reaaaaaaally strange and potentially disturbing and not directly related to the Arduino. I did two things and it seems more stable.
1) The Arduino code did in fact change, I'm trying to edit out the things that are project specific when I have a chance and will post it :) The timing of events has drastically improved!
2)
Constant crashes:
Believe it or not I have a sound card using FireWire and I think a port got rotten and may have electrical issues. I also don't know if the whole card was causing electrical problems in my system (I saw a couple blue sparks from my hand!). If I recall correctly, I was listening to music while debugging so my sound card was connected and I think something to do with it / it's drivers or some electrical issue was causing Max to crash. I noticed that the system was stable when I was working late and had no audio so as to not piss off the neighbours. I turned it off and the system worked!
I then did various longevity tests to confirm that it was stable with good results. As a final test, my audio interface has two firewire outputs, I switched which output the laptop was connected to...seems stable. When I go back to the suspect port part of my electronics don't even turn on. Additionally, after finding this out I touched the back and saw a blue spark between my finger and the interface! Of course this is the last place I would think to look...then again it's a Presonus which aren't the most durable and I've had it for 10 years.
Does this make sense that it could be affecting the Arduino / crashing Max? Any electronics gurus?
I also uninstalled / reinstalled Max 7 and deleted all preferences. I don't know if this affected it.
I don't know if this is a bit of a ghost in the machine as I did a couple things, but this certainly did help! I'll also try and post that code to the project sections of the site!
Also awesome Matrix stuff @Jan !
back in the days when max 5 was new I had a sound card which would crash max intermittently and unexpectedly-- even if it wasn't being used (ie no msp objects, no audio turned on, no signal routed to the card)-- in the end the only thing i could do was remove the card-- so sound cards can crash max