Stefano Fasciani

The TSAM is tool for the analysis, modeling, and mapping of the timbre of sound synthesizers.

Synthesis algorithms often have a large number of adjustable parameters that determine the generated sound and its psychoacoustic features. The relationship between parameters and timbre is valuable for end users, but it is generally unknown, complex, and difficult to analytically derive. The TSAM includes a strategy for the analysis of the sonic response of synthesizers subject to the variation of an arbitrary set of parameters, and it uses an extensive set of sound descriptors ranked using a novel metric based on statistical analysis. The TSAM can be used for:

- study of the sonic response of synthesizers;

- explicit control of timbral characteristics;

- dimensionality reduction of the synthesis control space.

tsam-timbre-space-analyzer-and-mapper

TSAM – Timbre Space Analyzer And Mapper

Christos Michalakos

The brain of the electronic part of the instrument, which consists of motors, solenoids, DMX lights, microphones, embedded speakers and live electronics is entirely programmed using Max/MSP and custom externals objects.

augmented-drum-kit-trrraction

Augmented Drum Kit - Trrraction

Sam Tarakajian

Mira. It's coming. In fact, if you find yourself in Daejeon on Wednesday the 29th, then you'll be able to play with it in person. I'll be demoing Mira as part of NIME, the 13th international conference for New Interfaces for Musical Expression. Hope to see you there!


mira-at-nime

Mira at NIME

When most people get ready to make a presentation they turn to Powerpoint or Keynote. If they’re of the fixed-gear bicycle persuasion, maybe they reach for something like Prezi. As would become increasingly obvious over the course of his keynote address, Bill Verplank is not most people. The NIME community is known for enthusiastically embracing new technologies. For them, anything older than a Leap Motion is a relic from the Stone Age. Faced with the challenge of holding the attention of such a techno-ravenous group, Bill opted for a revolutionary piece of high-resolution, force feedback presentation hardware. You may have heard of it: it’s called the pen.

Bill’s talk, called Motors and Music, covers all the things that you would never expect to hear about during a discussion of computer music. Things like physicality. Things like emotive content and embodiment. Things like the ‘80’s. He starts by taking us on a tour of early experiments in computer music interaction. Bill narrates over videos of Max Mathews wiggling a string to trigger scanned synthesis and Perry Cook shaking a chain of bend sensors. It’s amazing to see just how advanced the interfaces were then, and how little has changed since. Sure, these days we might use a MacBook Air instead of a tape reel, but at the end of the day it’s almost as if we’ve lost more than we’ve gained. When I sit down to interact with a computer today, I’m lucky if I’m given a real keyboard as opposed to a simulation under glass. Watching Max Mathews and Perry Cook dancing in front of a wall of mainframes, creating music out of a chain, a string, a pen, and a coffee cup, it’s hard not to feel like we’ve lost something.

Bill now moves from archival footage to the present day. He introduces us to the Plank, one of many haptic toys crafted at the Copenhagen Institute of Interaction Design. Think of it as the atomic unit of force feedback interaction. Built out of a strip of wood and a re-purposed hard drive, it’s a piano key that pushes back.

He goes on to show us some of the projects his students have worked on using the Plank. You might think that there wouldn’t be much you could accomplish with one sensor and a single degree of freedom, but then you’d be wrong. Clearly, you’ve never played Angry Birds with a force-feedback slingshot, touched a quantum-entangled pendulum, or played Prosthetic Golf.

http://ciid.dk/education/portfolio/idp12/courses/motors-music/projects/

As Bill brings his presentation to a close, I fight the urge to rush the stage and take the Plank home for myself. I can’t remember the last time I got so excited about a piece of hardware. I didn’t feel this way when the iPad came out, instead I remember sinking into a fog of disappointment as I realized computer interfaces were moving away from physical interaction, not towards it. We’ve lost a lot of ground since the days of Max Mathews. When the iPhone first appeared people expressed frustration at having to type by tapping on a piece of glass. It doesn’t feel real, they said. I miss having buttons I can touch, they said. Now, Siri and spellcheck have taught our fingers laziness, complacency.

    So what happened to our nuanced, multimodal interaction paradigms? If you ask me, the same thing happened to interface design that happened to digital music: convenience beat quality. Given the choice between watching a poorly encoded YouTube clip and patiently downloading a high quality, DRM-laden audio file, most people would rather not wait. In the same way, people are more interested in checking their email 500 times a day from their smartphone than they are in having a two-hour jam session with a force feedback joystick. Researchers like Bill may push advances in interaction design, but it seems to me that makers of consumer electronics will always be more focused on portability and power consumption than on haptic feedback.

    Of course, I can’t help myself but wonder what would happen if some company suddenly decided to throw their whole weight behind a new gestural controller. What would they come up with? II feel the first step would be to fortify the woefully impoverished language that we currently use to talk about gesture. Think about it: when it comes to sound we’re able to address all the nuances of spectrum, waveform, frequency domain, timbre, loudness, pitch, attack, envelope and decay, just to name a few. What language do we have for talking about gestures? Slow versus fast, maybe?

    Before he leaves the stage, Bill offers us one last quote:

"Grab a hold of something and feel it push back at you and make music"

    At this point one of the audience members, inspired by Bill’s august presence, asks a question about toilets.
New Interfaces for Musical Excretion

nime-day-2

NIME, Day 2

This system allows to insert physical objects in audio digital signal processing chains, filtering the sound with the acoustic impulse response of any solid measured in real-time. Physical objects are modeled as a linear time-invariant system, which is then used as an audio filter. By interacting with the object or with the measuring hardware it is posible to dynamically modify the characteristics of the filter. The impulse response is obtained correlating a noise signal injected in the object through an acoustic actuator with the signal received from an acoustic sensor placed on the object. The system is implemented in Max using native objects only. Correlation and convolution are implemented in the frequency domain (using poly and pfft). This enable running multiple physical filters of maximum lenght (4096 samples) in parallel without overloading the CPU. The system support further creative use beyond audio filtering, such as tangible signal patching and sound spatialization.

This system allows to insert physical objects in audio digital signal processing chains, filtering the sound with the acoustic impulse respo

physical-audio-digital-filters

Physical Audio Digital Filters

The system is implemented in Max, which hosts VST-Plugin software synthesizer, performs the analysis and the real-time mapping. Max communicates via OSC with MATLAB, which computes the mapping using a unsupervised dimensionality reduction and machine learning algorithms. IRCAM FTM externals are extensively used in this project.

timbre-space-mapping-for-vst-synthesizers

Timbre Space Mapping for VST Synthesizers

On my last day in Korea, I wouldn't even have noticed. Nothing about staring at a heaping bowl of pickled cabbage first thing in the morning would bother me. After one week of eating kimchee three meals a day, every day, the part of me that needed two chocolate croissants and a double americano just to feel normal would finally have acclimated. On day one, however, I'm not quite there yet. After sputtering out a weak, black broth, the coffee machine advises me to “Have a nice time”. I’m giving it my best, but looking down at my bowl of rice with seaweed broth and trying to see oatmeal is more that I can manage. To any outside observer I'm sure I look like what I am: a coffee starved software engineer very much out of his element.

“Excuse me,” I hear someone say, “you must be here for NIME.”

Getting to NIME from the Toyoko Inn requires a fifteen minute taxi ride, circling the government complex at the center of Daejeon and crossing the river into KAIST campus. Staring out the window on the way over, I’m not entirely sure what to make of my surroundings. Based on the gnashing juxtapositions all around me, I’d say the city of Daejeon seems to think it can shock me out of sleep deprivation with bewildering choices in urban planning. The humble government complex, for example, rises no more than three stories high in the center of a large public park, yet towering apartment blocks housing thousands flank the complex to the east and the west. I decide that the squat government building must be nothing more than a gateway, and that beneath the park extends a labyrinth sprawling hundreds of miles underground. Also, peering into the distance beyond two apartment buildings, I notice a strange, metallic spire. It looks like a spaceship from my vantage point, but that would be crazy, my mind must be playing tricks on me. Of course, as we get closer it looks more and more like a spaceship, until it turns out that’s exactly what it is. In an attempt to clash maximally with the drab apartment units on the south side of the river, northern Daejeon sports a giant amusement park.

I give up on trying to understand the city and opt for conversation instead. My first traveling companion is Simon Hutchinson, who when he complains about being fatigued and confused does so in a voice both energetic and lucid. He explains that he’s come to NIME to perform a piece called Shin no Shin, using an iPad to turn touch and acceleration into music. I’m tempted to spout off about Mira, but I decide that there will be plenty of time for that later at my poster session. We exchange a few notes about the iPad as a performance instrument.  I wonder if Mira will be useful for musicians like Simon, or if the tools that already exist are good enough.

I’m also fortunate enough to ride with me Adam and Liam of Alphasphere, who tell me about their spherical music making gadget of the same name. My description can’t do it justice, but you can think of the Alphasphere as an overgrown buckyball with aftertouch. For a more precise picture, imagine plastic rings arranged in a ball, with flexible, pressure sensitive fabric stretched over each one. You play the instrument by distorting the fabric, which Alphasphere translates into MIDI and OSC data. As Adam describes the hardware I notice a strange tension in my fingers, my first taste of what I’m now calling NIME Complex Sigma. It’s a debilitating condition that I will encounter several times throughout the conference, characterized by acute mental anxiety and muscle twitching. The cause comes from listening to the description of a revolutionary new instrument; really, really wanting to play it and then not getting to play it.

NIME doesn’t officially start until the next day, but people like me who chose to show up early get to attend one of several workshops. I want to go to all six, but somehow the conference organizers expect me to pick just two. Of course, the whole question of which workshop to choose becomes moot when it turns out that none of us can find the building where we’re supposed to register. Each of the buildings on KAIST campus has a letter and number associated with it, which would in theory make finding a given building an easy task. However, at the center of campus the correlation between number, letter and proximity approaches zero–building E16 is right next to N4. Naturally, asking for directions is an exercise in futility, as what little Korean I know comes from watching Arrested Development. Eventually by walking in ever-widening circles we manage to find the right building. We’re a bit worried about showing up several minutes late, until we notice two crucial facts. Fact one: there is a giant mob of not-at-all-Korean looking people standing outside the building. Fact two: the man who is supposed to lead the first workshop is among them.

Sometimes people who make NIMEs forget to bring keys

When at last we manage to enter, the first thing I discover is that black coffee is not as easy to find as I would have hoped. Canned coffee drinks come easy, with vending machines at every corner offering sugary, undrinkable swill with names like Joy and Yes. But it seems I’m going to have to wait a bit longer to get a taste of something fresh roasted. Not having caffeine impairs my decision making process, which makes my second discovery all the more significant. As it turns out, two of the workshops are free, whereas the other four very much are not. So in the end, I opt for the NIME orientation workshop and for the one on making music with Web Audio.

Michael Lyons, a NIME veteran and researcher in musical interaction, leads the first workshop. His presentation does a great job of filling in the gaps in my knowledge on NIME related topics, subjects like primary versus secondary feedback (secondary feedback is the sound an instrument makes, primary feedback is everything else it does). He also provides a thought provoking overview of why people make NIMEs in the first place, which I find particularly interesting. Beyond techno-fetishism and fascination with the human-machine relationship, he posits that the #1 reason that people are interested in building new instruments is because of an insistence on cultural fluidity. People want new ways to make sound because they want their own tools–they don’t just accept what’s given to them. No wonder so many NIME builders use Max.

As Michael brings the presentation to a close, my mind is humming with new ideas to take back to the Cycling ‘74 think tank:

Mapping (between input gestures and sound output) is the heart of NIME, and indeed of instrument building in general.

MIDI is plug and play, OSC isn’t because there’s no standard

Programmability is a curse, and it’s important to have long-term versions of things

Primary feedback (lights, vibrations) is critical for intimacy

Music is becoming increasingly process oriented as opposed to artifact oriented. People who are not virtuosos are willing to go out in public and make music, and are eager to find a forum to do so.

My second workshop is with Charlie Roberts, the furiously talented man behind the Control app for iPad and iPhone that was in many ways the inspiration for Mira. This workshop was advertised as an introduction to the Web Audio API, so I’m imagining that we’re going to spend the afternoon talking about the complexities of working with audio in a high-level language like Javascript, and the challenges of getting audio to run in the browser on multiple platforms. As it turns out, Charlie has basically solved all those problems already, and so instead he takes on a three hour tour of Gibber. Gibber is a Supercollider-like wrapper around Web Audio that lets you build sample-accurate sequencer and synthesizers in Javascript. Oh, did I mention it’s runtime re-configurable? Anyway, it nearly melts my brain to think about how Charlie’s work could fit together with Max. Imagine a Max-like program running in the browser, with something like Gibber providing the backend to a patchable interface. Imagine using that interface to build and deploy interactive audio to the web. Or, switch your brain with me to Totally Unwarranted Speculation mode and imagine being able to turn any webpage into a programmable patch. It’s a bit of a pipe dream, to be sure, but why bother coming to NIME if you aren’t going to entertain impossible ideas?

After the workshop, we stumble back outside on legs made weak from six hours of sitting. Since I’ve gone over ten minutes without complaining, I seize on the opportunity to mope about the cloudy weather. No one around me seems to pay any attention, perhaps because they wisely understand that overcast and humid is probably an overture for rainy and chilly. For now, we take advantage of the warm weather to restore circulation to our feet and converse about all things NIME. As we chat, we’re treated to the first appearance of the chair of KAIST 2013, Woon Seung Yeo, better known as Woony. I had been told at some point (by someone very foolish) that Koreans have a cultural lacuna when it comes to sarcasm. I suspect that Woony knew this and made it his personal mission to wipe away my misconception. “I encourage you to visit the famous KAIST goose crossing, especially since I know NIME participants are all great lovers of animals,” he says. “Not in that way,” he adds.  Woony’s dry and biting wit would only desiccate in the days to come.

    Drawing his short opening remarks to a close, Woony directs our attention to the area behind us, where a seemingly infinite quantity of food seems to have materialized out of nowhere. “And now, enjoy the banquet,” he says. “And of course the free beer.”
Well, there you have it. Free beer and unlimited food. No points for guessing how long it took me to fall asleep after that one.

nime-day-1

NIME, Day 1

Recently, I went to the NIME (New Interfaces for Musical Expression) conference in at KAIST in Daejeon, South Korea. Over the course of five days, I attended workshops in Web Audio, absorbed paper presentations on digital laughter and watched what could only be described as a pneumatic zombie duet. I also attended not one but three banquets. For those interested in the gaps between banquets, I offer this story.

I step off the plane. Location: Incheon. Body: Exhausted. Mind: Blank. Between the 12 hour flight, the 15 hour time difference and repeated exposure to the in flight movie, A Werewolf Boy, I can already feel my grip on reality starting to slip away. I make my way through the airport, down to baggage claim and onto the express train for Seoul. As far as I can tell the train was constructed in the year 2040 and brought back in time to the present day. The oleophobic seats conform exactly to every contour of my exhausted body. A flatscreen television unfolds from the ceiling above, presenting a promotional ad for a nearby civic development project. BUILDING, it promises, in blaring, positivist capitals. CIVIL. PLANT. HOUSING. Depictions of enormous glass and steel buildings, assembled by swarms of tiny robots, rise before me. Outside my window, we pass row upon row of small scale farms, sometimes running all the way up to the train tracks. Eventually the train comes to a small bridge connecting Incheon to the mainland. Rising up out of the water I can see huge mounds of dirt and grass, looking like the backs of giant turtles lumbering towards Seoul. I am very sleepy. I decide that they probably are turtles, and I write the following poem:

The fog helps me see the tortoises
Grinding out low channels
And the speculative egrets on long stalks

The tortoises are my cold cows
Ruminating on the countryside
And other fictions
They roar silently
Like old men, or magma

Train tracks are humming
The sound of soft gray wool
And my eyes are as heavy as the tortoises

I decide that this poem is very good, then I fall asleep. When I wake up, we’ve arrived in Seoul, where I must have boarded another train for Daejeon, though I honestly can’t remember. Neither do I remember arriving in Daejeon, finding my hotel, or making my way up to my room. Probably all these things happened, but whether they happened to me or to someone who looks a lot like me I will never know. In the morning a straight line connects my backpack to my suitcase, to a pair of shoes, to where I fell asleep, face down on a still-made bed. 

Search Results