Unit tests for Max/MSP objects (and where's the Max 5 SDK, eh?)

Quote: tom@swirly.com wrote on Sat, 21 June 2008 09:21
----------------------------------------------------
> I also have issues with breakage in general; I find I make "tiny" changes to a patch and some other patch somewhere else breaks. Of course, this is due to my mistaken perception of what "tiny" really is but if I had testing I could make a small change, verify it, commit my work, and continue.
>
> ...
> how to unit test Max objects?
>
> Current plan:
> ...

I think you are entering unexplored territory here. Keep in mind most Max users don't have a software engineering background. If you come up with a good system, please share.

If I have really complicated logic I tend to drop into java or ruby where I can unit test my code. Personally, I don't want to add extra stuff to my patches just for testing purposes.

For the patch breakage issue, that's something you get a better handle on over time. The most important part is figuring out how to properly modularize your patches into abstractions. If the abstractions are simple and focused on doing one thing well, they have less bugs. Remember the KISS principle!

I make help files for my important abstractions that demonstrate all the features. These help files more or less serve as my unit tests, but it's a manual testing effort.

When you reuse abstractions for different projects, it's often a good idea to create a new copy of the abstraction for the current project. That way, if you need to modify anything you won't break all your old patches. This also helps stay organized, you can copy all the abstractions into the project folder and keep it all together.

Max will make you good at debugging ;)

Well I am a software engineer and a musician. And god knows
the last thing I want to be doing is writing unit tests for my
patches. I spend far too much time coding/patching than making
music as it is, I don't see how making unit tests would help
that. Sounds like a lot of extra work.

And by the way, where is the new SDK?

Quote: Anthony Palomba wrote on Mon, 30 June 2008 18:07
----------------------------------------------------
> And by the way, where is the new SDK?
----------------------------------------------------

We recently had a very productive workshop in Paris, where a preliminary version of the still-being-finalized SDK was unveiled. So fear not, the new SDK is coming. I don't believe that we announced a release date, though, so, well, thanks for your patience as we finish it up. It'll come out when it's ready.

Jeremy

Quote: Anthony Palomba wrote on Mon, 30 June 2008 18:07
----------------------------------------------------
> Well I am a software engineer and a musician. And god knows
> the last thing I want to be doing is writing unit tests for my
> patches. I spend far too much time coding/patching than making
> music as it is, I don't see how making unit tests would help
> that. Sounds like a lot of extra work.
>
> And by the way, where is the new SDK?
----------------------------------------------------

I believe Tom is one of the few people (like me) that want to use max as a platform for actual software engineering (i.e. patches complex enough to benefit from known software engineering techniques).

On a smaller scale I always make some test data for small pieces of logic in Max and see if I agree with the output, but so far never for pieces of my patch that depend on a lot of communication to the outside. But with a server side Java project, I recently started doing a lot of unit testing. So Tom, I am very interested to see where you're going with this.

In your scheme I think I currently miss the equivalent of assert statements. But I'm quite sure that the case you put up could very well be made to work in Max.

Furthermore:
* Creating mocks appears to be impossible.

True, unless you introduced nifty semi-automatic renaming of receives/forwards. I guess the patch you are testing would need to use special abstractions instead of send/receives.

* Testing user interfaces appears difficult or impossible.

But this is always the case with unit tests, no?

* Simulating time (so unit tests don't run in real time) seems difficult or impossible.

Yeah, but here too, with multiple threads, unit tests generally don't apply or get really difficult.

* This appears to be best for testing logic or edge cases.

Agreed.

* However, most of my bugs at least are logic bugs or edge cases.

Agreed, too.

Best,
Mattijs

Even in C or C++ unit tests are really hard for things that have output that is in the form of audio or video. I worked on an image editor for a few years, and automated testing of sections of that code was really hard.

You can try and come up with a "fingerprint" for audio buffers, but unless you're dealing with fairly simple audio paths, and keeping _everything_ in the audio domain, there is enough uncertainty in scheduling time to make comparing audio buffers non-trivial. Throw in any random, or hard to determine randomish control source, and things get really hard to test.

-C

Yawn.

Quote: barry threw wrote on Tue, 01 July 2008 00:30
----------------------------------------------------
> Yawn.
>

Dude..

Quote: barry threw wrote on Mon, 30 June 2008 16:30
----------------------------------------------------
> Yawn.
>

Senior member, eh? So why act like a junior?

Thanks for a lot of good comments.

Let's start with

0. I had no idea of doing this for audio, I quake with fear at the very thought. Actually, I simply never do audio with Max/MSP, I'm really interested in routing and processing MIDI for performance.

1. I should also point out that this wasn't "me looking for things to waste my time" (not that I don't do that :-D ).

It's "me doing Max development and having all sorts of subtle logic bugs that suck up a whole evening with nothing to show for it."

Also, since I'm just in the first couple of months of (re)starting development I'm contemplating moderate investments that might pay off over a year or so.

I want to spend *less* time programming and more time playing.

Within my time programming I want to spend *less* time doing stressful debugging and more time making things.

2. My thought was that I might be able to do a lot of good by testing the 15% of my Max code with the most logic.

3. I'm also thinking of doing the complex logic in another language as mentioned by others too. However, I haven't yet advanced on looking into native Python (aside from downloading the project that might do this).

4. I already have a set of .maxhelp patches for pretty well everything I wrote; I was wondering how to extend them to make at least some automated testing without work.

5. The system I proposed before doesn't involve changing the patches you're testing: you can think of it as adding a receive and a send to your maxhelp for those patches.

It might be several weeks before I have time to continue with this because I'm away for a lot of this time, but I'll report any progress here.

My guess is that I'll make a small unit test framework (two new patches with only light complexity inside) and even only test two or three of the most problematic and logic-oriented of my patches to date and hopefully this will take much less time than all this writing did (but that's why it's called "design" :-D).

(Classic story: we had a terrible problem with the flagship product in a previous company; it was an applet that did animation under all versions of Java back to 1.1 and we had an intermittent crashing bug under some IE/JVM combinations. We had a series of emergency meetings, came up with theories and ran a bunch of experiments, narrowed down the cause, and eventually came up with a fix, which was in fact quite short. Management, who were extremely ignorant of software development, were pissed: "If you hadn't spent all that time yakking, you could have fixed this last week.")