(Muscle) Audio to sensor/control data
I did a quick search but didn't find anything on the forum, which I guess makes sense as it's built in PD.
Basically I've been looking at the Xth Sense sensor:
http://res.marcodonnarumma.com/projects/xth-sense/
http://marcodonnarumma.com/works/xth-sense/
Basically an electret microphone that you attach to your forearms.
Seems quite ingenious as that's a very cheap way to generate complex data.
Now the patch is written in PD, and I don't know PD at all, but I'm curious if anyone has come across this, or something like this, in terms of turning what I'm imagining is very quiet and complex audio into meaningful data (streams).
Hi
has Marco made the PD patch available; could you "translate" from there? I spoke to him at a conference last year regarding the broader issues and he did concede that custom microphones and some hardcore filtering were required to reduce the abundant noise - a problem common to any "biometric" input I guess. At that time he was using actual audio from muscle/tendon against bone/each other, as both control data and audio sources. Not sure this helps though??
Brendan
That's the thing, I wouldn't even be able to translate a simple PD patch, much less something more complicated. I was speaking to some colleagues at my school and they were thinking of working out a port for the patch (for Max), but had similar problems. None of them knew PD.
Hey Rodrigo, I m following Marco's work with the biosensor's (I'm trying to have a third working channel i hi s software). As in a first moment I wasn't familiar with pd I've used sound flower (or another similar) to get the audio from pd to max (the only problem is that if u are working with more than 1 audio channel, pd will send all the channels mixed together so in max u will have just one channel. I was trying to solve this in pd, and I didn't have time to finish the work, I hope I'll do it soon)
Hope it helps
Serena
I have an Xth sense kit (but haven't done much with it). Have you looked at PD? It's very similar to max. Wouldn't be hard to translate one to the other. Though I think there's custom objects/externals in the loop, now that I think of it. Building an audio/control data bridge between the PD patch and Max would be the way to go I guess.
(Double post)
I wouldn't want to have both environments open as my max patch hits the CPU pretty hard as it is. It would also be great to use the control data for my own processes.
@dtr
Did you not find it useful or why did you not use it?
I had a quick look to Marco's software and what it's doing is mostly to emphasize the low frequencies (under 80 Hz or even less, u can set the value in pd) so what u hear (and see, because u can see the wave forms in the software) are only the low frequencies (he used pitch switching to do this). then Marco's made some libraries for the muscles tracking (so for example the software understand when u keep contracting the muscle or not), but I haven't looked at all at the libraries. Then everything is sent to a mixer so u can control the volume of each channel. Doing some pitch switching with max is not that hard (and also I've tried to use instead of the Xth sensors some normal pickup/contact microphones and the signal was pretty good).
I was thinking to translate the software in pd to max as well (because testing Marco's software I found different issues and I would like to work with something that I can totally control)..
Interesting.
What is pitch switching? Do you mean pitch shifting?
If so, is it doing add-overlap shifting?
It would be amazing if you could translate the software, or at least the input/library software!
> Did you not find it useful or why did you not use it?
I played around with it very briefly and didn't instantly get something useful for my current work. Then got too busy with stuff to delve in deeper. I really should give it another try sometime. (Marco just gave a workshop at STEIM here in Holland but too bad I was too busy for that)
Did you buy the kit complete? (as in all the components) I hate having to order individual things from different manufacturers.
yes sorry SHIFTING (blame my telephone corrector!)... I'm thinking about translating the software for my final project at uni...and I have bought a compete kit but I'm still waiting for it (i ordered it in november, it was ready last month and it will take another month to be delivered) so in the meantime I've built a couple on my own (they don't look very nice but they work)...also u can try with contact microphones the signal it's not the same but they work pretty well too...
> Did you buy the kit complete? (as in all the components) I hate having to order individual things from different manufacturers.
I got a complete kit. Takes 15 minutes or less to assemble.
Hi lads,
I can shed some light on what's going on in the Xth-sense and help solve some of your immediate problems.
The software is actually an impressive piece of patching and porting it is not trivial. It is conceived as an instrument so even though the methods are simple the way it all ties together is quite complex. Basically you have your audio inputs that are downsampled to the message domain so that you can use several extracted features to control processing parameters that affect the incoming audio signals. There is a sequence creator so you can re-route control parameters to different objects during any given piece and store presets. You also have a preset friendly audio mixer so you can fade in/out different sound streams. The main patch generates different canvases which are blank PD patches. There, you patch at a bit of a high level since every effect and feature stream gets named automagically (it is a bit like patching using Jamoma modules). A simple patch workflow would be as follows:
1 create a new canvas/preset.
2 Select incoming audio,
3 create a processing object (say that you have one arm's sound to be pitch shifted using one of the included modules.)
4 create an incoming extracted feature (let's say we use the amplitude envelope of the other arm to control the transposition)
Now you have a simple patch where you generate very low frequency sound with one arm and you can transpose it with the other.
You can now create another preset with some crazy delay parameters as a different stream that fades in after you've passed a threshold on one of the arms a number of times and so on until you have a piece.
Marco has planned to set up a team to do a proper port of the software but it will take time and more than I port I reckon it will need to be a fork as there are significant differences in the way PD and Max work. Also, he doesn't know Max so it is difficult to get much done without physically meeting and going module by module. However, on his working version of the patch there is OSC streaming of the features he is using. I do not know if this is included on the current 'release' version or if it is only on his personal patch.
In any case, help is at hand and if you have FTM installed (and everyone should) you can do the feature extraction in Max like so:
It is still up to you what sound processing modules you develop and how you map the filtered signals but it is a head start.
I'm hoping to meet Marco in London in a few weeks and see where he is going with the Xth-Sense. I know he has been working on a wireless version and I have a wee PCB design to simplify manufacturing. As for components, I got mine from farnell:
Battery
http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=2065171
Battery Holder
http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=2064726
Electret mic.
http://uk.farnell.com/jsp/search/productdetail.jsp?SKU=1502742
I had resistors, capacitors, female jack connector and project boxes lying around so I didn't purchase those but you can get at Farnell as well.
I have not done the sewing and have been using mine with electrical tape.
Last but not least, I've found a better option for the silicon cases that will be included in the next official version of the Xth-sense
They're 11 quid for a sheet with 72. You can find them at: http://www.fingameblobs.co.uk/index.php?id=buy (Part: W. PD.2210 recessed) available in clear and Black! These provide better signal to noise ratio and you only have to drill a hole making them easier to make.
Hope you find this info useful.
Cheers
- Miguel
@Miguel
Tons of info!
It was mainly the hardware and feature extraction that I was interested in, to use in my own patch/processes. I only saw the PD patch (screens of it anyways) recently. I had no idea it had the spawning/reconfiguration stuff, which is great.
Interesting to hear about a wireless version being developed. I would imagine, especially since it's working with messages, that the downsampling can happen on an embedded circuit, and pipe it over using an XBee (or bluetooth if you're fancier (or that new TI all-in-one wifi chip that's supposed to come out soon, if you're even fancier!)).
The posted patch looks interesting/useful. I didn't even think I had ftm installed, but as it turns out I do. I just hate developing dependencies, as I never want to have to completely refactor or abandon code when eventual deprecation comes.
Rodrigo,
I know the guys at Goldsmiths were looking at both bluetooth and XBee but not sure what the next official version will be. I have to say though that I quite like the simplicity of audio and long cables. I've worked with the BioMuse EMG bands and a Bluetooth Arduino for some years and have run on a few problems now and then. Also, the Xth-sense going directly to a digitech whammy pedal is a blast!
I generally share your point on dependencies but have fallen for the ftm objects. Having matrices that are not assumed to be video and having static/dynamic named variables makes for very clean patching.
looking forward to see what you create with the x sense.
- Miguel
I've messed with XBee a bunch, and although the xbee communication itself seems fine/reliable, I REALLY hate serial connectivity in Max.
I've never had anything crash my computer (much less Max) so hard. Like hard locked up.
ftm is also scary in how big it is. As far as I know, I don't need anything in it, so it would take me a while to learn what I need, then more to implement it....
Thanks for your help with all of this!
(off-topic) I'm using SenseStage's Minibee (Arduino+Xbee variant). The lightest version of their supporting software suit uses a simple GUI-less Python app that handles Xbee communication and sends OSC to Max (or whatever). Receiving data at 60Hz without a hitch.
You never have a problem with the python app getting hung up on serial?
I ended up abandoning my XBee experiments when I got funding to buy an x-IMU (which blows my xbee/arduino system out of the water), but do want to use xbee stuff for other projects.
I don't recall having any issues with it. I initially had some communication with SenseStage about getting high frequency data transmission stable, with some firmware updates etc. After that, no problem. Except for when the battery runs out of juice ;)
I'm using two senders and one receiving node.
Sorry for the noise on this topic , but I see this link https://getmyo.com/ on electromyogram sensor so of course I used in the past bioflex sensor http://infusionsystems.com/catalog/product_info.php/products_id/199
But I hope a developpement of specific arduino will come....
hi all! this is my first post here :)
first, thanks (and to Miguel specially)! I'm glad of such interest in the instrument.
second, please, note that the Xth Sense (XS) uses muscle sounds or MMG (mechanomyogram). This is a quite different signal than what the Biomuse and the Myo are based on. The XS captures muscle sounds, i.e. mechanical vibration, with a custom mic and no skin contact; Biomuse and Myo capture electrical voltage from skin contact electrodes. Therefore, the XS amplifies sounds from your body, and produce control data from them. The Biomuse and the Myo only produce control data.
Unfortunately, a piezo mic won't report the frequency range of the electret condenser mic used by the XS (which is indicated in the publicly available parts list at the XS website, along with tutorials, etc. http://res.marcodonnarumma.com/projects/xth-sense).
Max/MSP:
That said, a small team of Max/MSP developers emerged spontaneously, and we have started the porting of the XS software, and Miguel is with us too. The progress will be slow. As Miguel said, the instrument is way more complex than it looks like; there's lots going on under the hood so to keep a very high usability also for beginners. Besides, nobody of us is paid to do this, so ppl have other priorities. We do plan to have it ready around autumn, but that's only a plan, cannot guarantee :)
We are looking for expert Max/MSP programmers that would like to spend a little time collaborating with us on the porting. Please contact me in pv if you're interested or know someone who might be.
Another side of the story is the license. The XS software is GPL licensed (free software http://www.gnu.org/licenses/quick-guide-gplv3.html), and this won't change. The GPL license poses some issues in terms of porting to a commercial software, such as Max. If anybody knows well Max license, we could use some further info.
Wireless:
We do have a different team working on this. A working prototype exists, and will be tested soon with a large-scale biophysical dance project; we are all quite excited about it. So expect an open source, DIY wireless Xth Sense sometime this year!
Also, don't want to do propaganda here, but for all XS related questions, etc, feel free to post at our Users Forum (kindly hosted by Create Digital Music):
http://createdigitalnoise.com/categories/xth
cheers!
M
Because I'm working sometimes with Atau, I'm fully interested to give some help.
@Marco
Thanks for chiming in.
It's cool that there's a port happening as there are quite a lot of Max users who would be into it (assumption). It would be great just to have the feature extraction side of things ported over, as the core idea (muscle audio to control/audio data) is interesting, and useful beyond the scope of the XS project in specific.
My max is ok, but not good enough to help with something like this (hence the starting of this post), otherwise I'd be into it.
Thanks @mortimer59! will write you in pv.
@rodrigo, I see your point. It is my belief though that the Xth Sense needs to be ported in the most transparent way, and "in full". Which means bringing simply making the Xth Sense as a musical instrument available to the Max community. This means porting also the custom mapping methods I developed, and more generally, the way by which a musician compose music with it, that is dictated by the design of the GUI, and the way you interact with the software side.
Of course, once the port is complete people is free to reverse engineering, and take what they need the most. Which is what happens as we speak with the Pure Data version, and the hardware components.
Glad of this discussion!
Hi all,
I'm probably not an expert in Max programming, nor in Pd~ but if I can help for this wonderful project (for example, French translations of a user guide or something like this), it would be a pleasure !
Best,
Nicolas Canot,
France.