Tom Swirly

Hello again. As you can probably guess, I'm setting up for what looks like quite a few months of development (though I already have a nice 2-way driver for the Frontier Tranzport 

http://www.frontierdesign.com/Products/TranzPort,

 a truly fantastic and cheap little wireless unit, which I intend to contribute in the next few weeks...)

I realize that I'm deeply suspicious of any code I write that does not have unit tests, that I frankly don't believe it works.

I also have issues with breakage in general; I find I make "tiny" changes to a patch and some other patch somewhere else breaks. Of course, this is due to my mistaken perception of what "tiny" really is but if I had testing I could make a small change, verify it, commit my work, and continue.

* There need to be two levels of unit testing, one at the C/C++ level, one at the Max level.

* The C++ unit testing depends on the C++ framework for development. I downloaded the Max 4.6 SDK (any ETA for the Mac 5 SDK?) and will peruse it. I'm favourably impressed with the simplicity and clarify of its documentation so far; on the other hand, I have nothing yet to test.

* C++ unit testing I can work out: but how to unit test Max objects?

1. Create two well-known send/receive (s/r) pairs called UnitTest and UnitTestResult

2. Patches that need to be tested need to add one or more "unit test patches" to a single central unit test patch ("main test") for the project.

3. A unit test patch for a patch can be that patch, its .maxhelp, or some new patch.

4. A unit test patch must receive UnitTest and send to UnitTestResult in at least one place.

5. A unit test patch has a name that must be unique over all unit tests in the same main test.

6. To start the test, the main test sends a start message to UnitTest.

7. Each unit test must first reply with a message to UnitTestResult:

8. Each unit test must subsequently send exactly one of two other messages to UnitTestResult

9. A unit test that has sent a start message to UnitTestResult but hasn't sent an ok or fail message is said to be "stalled".

10. On the original page, our unit test indicator is red if any fail message has been received since the last start was sent, yellow if some tests are stalled or green if all tests that were started had replied "OK".

* Creating mocks appears to be impossible.

* Testing user interfaces appears difficult or impossible.

* Simulating time (so unit tests don't run in real time) seems difficult or impossible.

* This appears to be best for testing logic or edge cases.

* However, most of my bugs at least are logic bugs or edge cases.

Hello again.  As you can probably guess, I'm setting up for what looks like quite a few months of development (though I already have a nice 2-way driver for the Frontier Tranzport http://www.frontierdesign.com/Products/TranzPort, a truly fantastic and cheap little wireless unit, which I intend to contribute in the next few weeks...)

I also have issues with breakage in general;  I find I make "tiny" changes to a patch and some other patch somewhere else breaks.  Of course, this is due to my mistaken perception of what "tiny" really is but if I had testing I could make a small change, verify it, commit my work, and continue.

* The C++ unit testing depends on the C++ framework for development.  I downloaded the Max 4.6 SDK (any ETA for the Mac 5 SDK?) and will peruse it.  I'm favourably impressed with the simplicity and clarify of its documentation so far;  on the other hand, I have nothing yet to test.

* C++ unit testing I can work out:  but how to unit test Max objects?

4. A unit test patch must receive UnitTest and send to UnitTestResult in at least one place.  

* Creating mocks appears to be impossible.
* Testing user interfaces appears difficult or impossible.
* Simulating time (so unit tests don't run in real time) seems difficult or impossible.
* This appears to be best for testing logic or edge cases.
* However, most of my bugs at least are logic bugs or edge cases.


unit-tests-for-maxmsp-objects-and-wheres-the-max-5-sdk-eh

Quote: tom@swirly.com wrote on Sat, 21 June 2008 09:21

----------------------------------------------------

> I also have issues with breakage in general; I find I make "tiny" changes to a patch and some other patch somewhere else breaks. Of course, this is due to my mistaken perception of what "tiny" really is but if I had testing I could make a small change, verify it, commit my work, and continue.

I think you are entering unexplored territory here. Keep in mind most Max users don't have a software engineering background. If you come up with a good system, please share.

If I have really complicated logic I tend to drop into java or ruby where I can unit test my code. Personally, I don't want to add extra stuff to my patches just for testing purposes.

For the patch breakage issue, that's something you get a better handle on over time. The most important part is figuring out how to properly modularize your patches into abstractions. If the abstractions are simple and focused on doing one thing well, they have less bugs. Remember the KISS principle!

I make help files for my important abstractions that demonstrate all the features. These help files more or less serve as my unit tests, but it's a manual testing effort.

When you reuse abstractions for different projects, it's often a good idea to create a new copy of the abstraction for the current project. That way, if you need to modify anything you won't break all your old patches. This also helps stay organized, you can copy all the abstractions into the project folder and keep it all together.

Quote: tom@swirly.com wrote on Sat, 21 June 2008 09:21
----------------------------------------------------
> I also have issues with breakage in general;  I find I make "tiny" changes to a patch and some other patch somewhere else breaks.  Of course, this is due to my mistaken perception of what "tiny" really is but if I had testing I could make a small change, verify it, commit my work, and continue.
>
> ...
> how to unit test Max objects?
>
> Current plan:
> ...

If I have really complicated logic I tend to drop into java or ruby where I can unit test my code. Personally, I don't want to add extra stuff to my patches just for testing purposes. 

Well I am a software engineer and a musician. And god knows

the last thing I want to be doing is writing unit tests for my 

patches. I spend far too much time coding/patching than making 

music as it is, I don't see how making unit tests would help

Well I am a software engineer and a musician. And god knows
the last thing I want to be doing is writing unit tests for my 
patches. I spend far too much time coding/patching than making 
music as it is, I don't see how making unit tests would help
that. Sounds like a lot of extra work.

Quote: Anthony Palomba wrote on Mon, 30 June 2008 18:07

We recently had a very productive workshop in Paris, where a preliminary version of the still-being-finalized SDK was unveiled. So fear not, the new SDK is coming. I don't believe that we announced a release date, though, so, well, thanks for your patience as we finish it up. It'll come out when it's ready.

Quote: Anthony Palomba wrote on Mon, 30 June 2008 18:07
----------------------------------------------------
> And by the way, where is the new SDK?
----------------------------------------------------

> Well I am a software engineer and a musician. And god knows

> the last thing I want to be doing is writing unit tests for my 

> patches. I spend far too much time coding/patching than making 

> music as it is, I don't see how making unit tests would help

I believe Tom is one of the few people (like me) that want to use max as a platform for actual software engineering (i.e. patches complex enough to benefit from known software engineering techniques).

On a smaller scale I always make some test data for small pieces of logic in Max and see if I agree with the output, but so far never for pieces of my patch that depend on a lot of communication to the outside. But with a server side Java project, I recently started doing a lot of unit testing. So Tom, I am very interested to see where you're going with this.

In your scheme I think I currently miss the equivalent of assert statements. But I'm quite sure that the case you put up could very well be made to work in Max.

True, unless you introduced nifty semi-automatic renaming of receives/forwards. I guess the patch you are testing would need to use special abstractions instead of send/receives.

But this is always the case with unit tests, no?

Yeah, but here too, with multiple threads, unit tests generally don't apply or get really difficult.

Quote: Anthony Palomba wrote on Mon, 30 June 2008 18:07
----------------------------------------------------
> Well I am a software engineer and a musician. And god knows
> the last thing I want to be doing is writing unit tests for my 
> patches. I spend far too much time coding/patching than making 
> music as it is, I don't see how making unit tests would help
> that. Sounds like a lot of extra work.
>
> And by the way, where is the new SDK?
----------------------------------------------------

Furthermore:
* Creating mocks appears to be impossible.

Even in C or C++ unit tests are really hard for things that have output that is in the form of audio or video. I worked on an image editor for a few years, and automated testing of sections of that code was really hard.

You can try and come up with a "fingerprint" for audio buffers, but unless you're dealing with fairly simple audio paths, and keeping _everything_ in the audio domain, there is enough uncertainty in scheduling time to make comparing audio buffers non-trivial. Throw in any random, or hard to determine randomish control source, and things get really hard to test.

You can try and come  up with a "fingerprint" for audio buffers, but unless you're dealing with fairly simple audio paths, and keeping _everything_ in the audio domain, there is enough uncertainty in scheduling time to make comparing audio buffers non-trivial. Throw in any random, or hard to determine randomish control source, and things get really hard to test.

Quote: barry threw wrote on Tue, 01 July 2008 00:30

Quote: barry threw wrote on Tue, 01 July 2008 00:30
----------------------------------------------------
> Yawn.
>

Quote: barry threw wrote on Mon, 30 June 2008 16:30

Senior member, eh? So why act like a junior?

Quote: barry threw wrote on Mon, 30 June 2008 16:30
----------------------------------------------------
> Yawn.
>

Senior member, eh? So why act like a junior?


0. I had no idea of doing this for audio, I quake with fear at the very thought. Actually, I simply never do audio with Max/MSP, I'm really interested in routing and processing MIDI for performance.

1. I should also point out that this wasn't "me looking for things to waste my time" (not that I don't do that :-D ).

It's "me doing Max development and having all sorts of subtle logic bugs that suck up a whole evening with nothing to show for it."

Also, since I'm just in the first couple of months of (re)starting development I'm contemplating moderate investments that might pay off over a year or so.

I want to spend *less* time programming and more time playing.

Within my time programming I want to spend *less* time doing stressful debugging and more time making things.

2. My thought was that I might be able to do a lot of good by testing the 15% of my Max code with the most logic.

3. I'm also thinking of doing the complex logic in another language as mentioned by others too. However, I haven't yet advanced on looking into native Python (aside from downloading the project that might do this).

4. I already have a set of .maxhelp patches for pretty well everything I wrote; I was wondering how to extend them to make at least some automated testing without work.

5. The system I proposed before doesn't involve changing the patches you're testing: you can think of it as adding a receive and a send to your maxhelp for those patches.

It might be several weeks before I have time to continue with this because I'm away for a lot of this time, but I'll report any progress here.

My guess is that I'll make a small unit test framework (two new patches with only light complexity inside) and even only test two or three of the most problematic and logic-oriented of my patches to date and hopefully this will take much less time than all this writing did (but that's why it's called "design" :-D).

(Classic story: we had a terrible problem with the flagship product in a previous company; it was an applet that did animation under all versions of Java back to 1.1 and we had an intermittent crashing bug under some IE/JVM combinations. We had a series of emergency meetings, came up with theories and ran a bunch of experiments, narrowed down the cause, and eventually came up with a fix, which was in fact quite short. Management, who were extremely ignorant of software development, were pissed: "If you hadn't spent all that time yakking, you could have fixed this last week.")

0. I had no idea of doing this for audio, I quake with fear at the very thought.  Actually, I simply never do audio with Max/MSP, I'm really interested in routing and processing MIDI for performance.

1. I should also point out that this wasn't "me looking for things to waste my time" (not that I don't do that :-D ).  

I want to spend *less* time programming and more time playing.  

3. I'm also thinking of doing the complex logic in another language as mentioned by others too.  However, I haven't yet advanced on looking into native Python (aside from downloading the project that might do this).

4. I already have a set of .maxhelp patches for pretty well everything I wrote;  I was wondering how to extend them to make at least some automated testing without work.

5. The system I proposed before doesn't involve changing the patches you're testing:  you can think of it as adding a receive and a send to your maxhelp for those patches.

It might be several weeks before I have time to continue with this because I'm away for a lot of this time, but I'll report any progress here.  

(Classic story:  we had a terrible problem with the flagship product in a previous company;  it was an applet that did animation under all versions of Java back to 1.1 and we had an intermittent crashing bug under some IE/JVM combinations.  We had a series of emergency meetings, came up with theories and ran a bunch of experiments, narrowed down the cause, and eventually came up with a fix, which was in fact quite short.  Management, who were extremely ignorant of software development, were pissed:  "If you hadn't spent all that time yakking, you could have fixed this last week.")


I don't know if this is helpful or not, but we have just begun 

implementing unit tests for the Jamoma project.

1. a Ruby script starts an OSC server (rosc)

2. the Ruby script copies a patcher into the max-startup folder

4. and then starts communicating with Max, whose patcher (from the max- 

startup folder) has a udpsend and udpreceive object in it.

The current version of the ruby script (testrunner.rb) is here:

http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Tools/

Some components used by the system (need to be in the searchpath) are 

http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Jamoma/library/test-components/

http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Tests/

Hopefully this is somewhat relevant. The system in Jamoma is not 

mature, so if there are ways that people would like to improve upon 

I don't know if this is helpful or not, but we have just begun  
implementing unit tests for the Jamoma project.

The way it works is this:
1. a Ruby script starts an OSC server (rosc)
2. the Ruby script copies a patcher into the max-startup folder
3. the script launches Max
4. and then starts communicating with Max, whose patcher (from the max- 
startup folder) has a udpsend and udpreceive object in it.

The current version of the ruby script (testrunner.rb) is here:
	http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Tools/

Some components used by the system (need to be in the searchpath) are  
here:
	http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Jamoma/library/test-components/

Some example test patches are here:
	http://jamoma.svn.sourceforge.net/viewvc/jamoma/branches/active/Tests/

Hopefully this is somewhat relevant.  The system in Jamoma is not  
mature, so if there are ways that people would like to improve upon  
it, etc. then please let us know...

No, what I'm saying is that its not interesting for at least some 

people that work on patches with more than 10,000 objects.

No, what I'm saying is that its not interesting for at least some  
people that work on patches with more than 10,000 objects.

I would also like to say that I didn't necessarily mean to sound as 

Indeed, I don't feel strongly against unit tests, I guess one of the 

things that makes me scratch my head about this is the fact that its 

rare that I patch things with a single point of entry to which I could 

It seems that Jamoma uses one faucet, with udp send and receive...many 

of my "units" have a lot more doors than that.

Mattijs, do you generally structure your patches with one inlet and 

outlet? If there was a good solution to the communication with this 

kind of testing it might be useful in other situations...

So, sorry for the noise, which got carried on longer than it might have.

> But perhaps you are trying to say that this discussion is not 

> interesting for people that work on patches with less than 10,000 

> objects, and I mostly agree with that. I'd say those people are free 

I would also like to say that I didn't necessarily mean to sound as  
discouraging as I might have.

Indeed, I don't feel strongly against unit tests, I guess one of the  
things that makes me scratch my head about this is the fact that its  
rare that I patch things with a single point of entry to which I could  
pass messages.

It seems that Jamoma uses one faucet, with udp send and receive...many  
of my "units" have a lot more doors than that.

Mattijs, do you generally structure your patches with one inlet and  
outlet? If there was a good solution to the communication with this  
kind of testing it might be useful in other situations...

> But perhaps you are trying to say that this discussion is not  
> interesting for people that work on patches with less than 10,000  
> objects, and I mostly agree with that. I'd say those people are free  
> to skip this thread.
>
> Mattijs


Quote: barry threw wrote on Wed, 02 July 2008 01:45

> I would also like to say that I didn't necessarily mean to sound as 

Happy to hear that, I think Tom's initiative is great. And it applies to my current situation, maybe that's why seeing it criticized in a blunt way might have gotten me carried away. Anyway, happy to have you aboard, barry.

Tim, cool to hear that you guys keep venturing into larger-scale software engineering with Max. You reminded me that I still have to give Jamoma some proper attention and send you the feedback.

> Mattijs, do you generally structure your patches with one inlet and 

> outlet? If there was a good solution to the communication with this 

> kind of testing it might be useful in other situations...

An example of how I structure my patches can be found on my user page, see the MPC Studio patch.

For smaller algorithms (one network with a few small subpatchers at the most and clearly defined in- and outputs), I highly prefer patch cords. But for sharing data between larger subsystems patch cords get messy so I use wireless connections. The problem with send/receives is that they are global, but we found a beautiful solution for that, as you can see ;)

Quote: barry threw wrote on Wed, 02 July 2008 01:45
----------------------------------------------------
> I would also like to say that I didn't necessarily mean to sound as  
> discouraging as I might have.

> Mattijs, do you generally structure your patches with one inlet and  
> outlet? If there was a good solution to the communication with this  
> kind of testing it might be useful in other situations...

An example of how I structure my patches can be found on my user page, see the MPC Studio patch. 

> But I've been impressed before. I hope to be again.

I have a debug abhaXion in my collection, though I still use usually 

prints... debug has the advantage to switch debuging on/off without the 

need to trash prints or their connnections....

I am sure the new debuggin' modes of max 5 will reveal some extra 

insights as well... I am so much used to do it the old way, that I still 

But what am I talking, I first have to look up these coders buzz words 

like "unit testing" - I am sure in the Max world this is called "help 

file". (Just verified it with the Wikipedia article... ;-) If I create a 

new abhaXion, I create a help file to test it... That way I have a help 

file in the same moment. And I need to include all possible tests in the 

help file anyway, that's why I often enhance the existing help files, as 

they often do not cover all possible cases...

As I am not the oo expert, I will appreciate any explanation what the 

difference between a unit test and a help file would be, or what a help 

Barry Threw schrieb:
> But I've been impressed before. I hope to be again.

I have a debug abhaXion in my collection, though I still use usually 
prints... debug has the advantage to switch debuging on/off without the 
need to trash prints or their connnections....

I am sure the new debuggin' modes of max 5 will reveal some extra 
insights as well... I am so much used to do it the old way, that I still 
didn't uncover all of its beauty...

But what am I talking, I first have to look up these coders buzz words 
like "unit testing" - I am sure in the Max world this is called "help 
file". (Just verified it with the Wikipedia article... ;-) If I create a 
new abhaXion, I create a help file to test it... That way I have a help 
file in the same moment. And I need to include all possible tests in the 
help file anyway, that's why I often enhance the existing help files, as 
they often do not cover all possible cases...

As I am not the oo expert, I will appreciate any explanation what the 
difference between a unit test and a help file would be, or what a help 
file could not cover...

-- 
Stefan Tiedje------------x-------
--_____-----------|--------------
--(_|_ ----|-----|-----()-------
-- _|_)----|-----()--------------
----------()--------www.ccmix.com


Quote: Stefan Tiedje wrote on Fri, 04 July 2008 13:27

> As I am not the oo expert, I will appreciate any explanation what the 

> difference between a unit test and a help file would be, or what a help 

Unit tests are automated. Usually there is a whole suite of tests that checks most of the code in an entire system after someone changes something. This is also known as "regression testing" and is really important if you work on a big project (think hundreds of thousands or millions of lines of code) with a lot of developers. This is how open source development for Linux, etc, can function with people from all over the world hacking on it.

So I think the real discussion here is, given a good help file that tests the functionality of an object/abstraction, how can we wrap that file with some scripting so that the test can be automated as much as possible.

For most Max users, this isn't something they'll ever need to worry about. But as Mattijs points out, maybe more "serious" developers will start using Max and be concerned about these things. If I was selling an application made with Max, I would start worrying about it.

Oh, and it has nothing to do with OO. It's general software engineering process. Like inspections for civil engineering.

Quote: Stefan Tiedje wrote on Fri, 04 July 2008 13:27
----------------------------------------------------
> As I am not the oo expert, I will appreciate any explanation what the 
> difference between a unit test and a help file would be, or what a help 
> file could not cover...

Oh, and it has nothing to do with OO. It's general software engineering process. Like inspections for civil engineering.


> Unit tests are automated. Usually there is a whole suite of tests

> that checks most of the code in an entire system after someone

Ah, that makes sense, you change an object and test its functionality 

(probably with its help file), but you still don't know if all other 

parts of a big project who use that object still do what they supposed 

to do, so you need to set up a test for all these dependencies...

Seems tricky, usually you just know if the main application fails there 

This creates a lot of overhead in coding for setting up the tests, don't 

know if that wouldn't create more work than just normal systematic 

But it would allow to make more substantial changes to objects which are 

often avoided because of "backward compatibility" issues which is 

Adam Murray schrieb:
> Unit tests are automated. Usually there is a whole suite of tests
> that checks most of the code in an entire system after someone
> changes something.

Ah, that makes sense, you change an object and test its functionality 
(probably with its help file), but you still don't know if all other 
parts of a big project who use that object still do what they supposed 
to do, so you need to set up a test for all these dependencies...

Seems tricky, usually you just know if the main application fails there 
is something wrong... ;-)

This creates a lot of overhead in coding for setting up the tests, don't 
know if that wouldn't create more work than just normal systematic 
debugging if something fails...
But it would allow to make more substantial changes to objects which are 
often avoided because of "backward compatibility" issues which is 
another word for "don't touch it"... ;-)

-- 
Stefan Tiedje------------x-------
--_____-----------|--------------
--(_|_ ----|-----|-----()-------
-- _|_)----|-----()--------------
----------()--------www.ccmix.com

I just want to give another point of view about testing in general.

I'm working on a project who use some robot percussionists on oil can, controlled by a mixed software environment (the show control system is written in Max, the robot control system is open source for robotics).

Being the robots heavy and powerful , I need to be sure that everything is ok and I need some kind of automatic testing both in development and in deployment, possibly executable by a non . Max guru.

I'm working on a project who use some robot percussionists on oil can, controlled by a mixed software environment (the show control system is written in Max, the robot control system is open source for robotics).
Being the robots heavy and powerful , I need to be sure that everything is ok and I need some kind of automatic testing both in development and in deployment, possibly executable by a non . Max guru.
It's a basic safety need.

Hello, all. As you can see, I have some time for Max/MSP again :-D and so I'm following up on old threads.

I did in fact write a unit testing framework for Max programs, but as we all suspected it probably wasn't worth the effort.

http://code.google.com/p/max-swirly/source/browse/#svn/trunk/max-swirly/testing

Warning: it might need other things from the max-swirly patches (an informal place for me to put things people might want).

If there were interest, I'd clean it up or explain it.

Hello, all.  As you can see, I have some time for Max/MSP again :-D and so I'm following up on old threads.

You can see it here:  http://code.google.com/p/max-swirly/source/browse/#svn/trunk/max-swirly/testing

If there were interest, I'd clean it up or explain it.


Unit tests for Max/MSP objects (and where's the Max 5 SDK, eh?)