I'm not saying this from personal experience, but I would imagine that Jitter has some really interesting applications for GUI design.........I've been meaning to delve into it but it's still on my to-do list unfortunately.
Although, in my opinion, a simple, well thought out UI combined with an impressive sounding patch/synth/etc.. would always be more enjoyable to use than a complicated clutter of controls.
A good sounding synth trumps eye candy any day. Thoughtful interface design also trumps eye candy. IMHO, it's the last question to ask - but *definitely* ask it - the good thing about Max is that if you're the person who created what's under the hood, you're also the best person to make decisions about useability (especially if you go out and work with the thing for a while to make sure that the decisions you made favor working with the thing instead of trying to look cool).
I have to agree with Tim on the "less is more" approach. Creating custom UI controls might also be an idea: have a look at [pictctrl], [pictslider] and [matrixctrl] where you can load in your own images. The "dialmode" attribute of the latter is pretty useful if you haven't noticed it yet. If you want to test the ideas you have then perhaps post something to the forum as a sort of beta test. I'm sure people will be more than happy to provide constructive criticism on the usability as well as the appearance.
haha, I'm a little hesitant to suggest this.........but If you're on a Mac (I've got no windows knowledge unfortunately), you can open the package contents of an app that you like the look of, and usually can nick all the images that make up their UI and use them with fpic+ubutton and/or pictctrl.................it screams copyright infringement....but can make for some veery nice looking patches.
+1 to the Photoshop approach. Don't forget also that you can make your Max controls transparent and overlay them on the background image(s). With creative use of [hidden] you can flip between graphical states based on the invisible max controls, so combining that with graphical knobs and customized sliders, effectively you can make a Max patch look like anything you can imagine.
I agree with less is more, as well as well thought out interface. Examples of some of my work include the Klee Sequencer, Major Malfunction, and MC-202 Hack, all viewable here: http://defectiverecords.com
--I use semi-transparent white multisliders on top of my signal meters---it's intuitive, saves space, and looks good.
--if you have a [filtergraph] or other object whose motion is spread out over time, you can a second object under the semi-transparent interactive one which shows where the actual response is.
--multiple [ksliders] on top of each other, using transparent white/black keys, can highlight multiple colors on the "same" keyboard. So you can show different tracks, voices, or transpositions. The bottom one can have opaque keys if you want.
--a global color chooser/send for certain parts of common objects: button, LED, dial, slider. Gives some user options while keeping things consistent. The "saturation" message to [swatch] is important to not have everything look too colorful.