Re: live audio – video synchronization

Sep 22, 2008 at 12:48pm

Re: live audio – video synchronization

Hello Nick,

For many obvious reasons I did not initially develop a Midi reader
for Antescofo, the most basic one being that you can not even write
trills correctly and things become very messy on the time level if
the Midi file is not correctly prepared! On the development agenda of
Antescofo however is the capability of atleast reading Midi and
MusicXML, but this won’t be a priority! My development priorities
these days are ensemble coordination, augmenting the FWD semantics
and also making the system robust for heavy polyphonic stuff
(orchestra etc.).

Up to now, whenever I was confronted with a Midi score I did one of
the following:
- Use NoteAbilityPro (gotta buy! I understand your reluctance!)
- Use an intermediate parser that reads MIDI and spits out Antescofo
scores in text. Serge Lemouton of Ircam (in CC here) has already done
this on the FTM Scoob editor that can read and parse MIDI, and then
he parses it out for Antescofo using his java script program. I used
his system for a whole piece of about 45 minutes long and used it in
concert. It dramatically reduces edit time! Maybe you can ask him
directly for that… .
- If you have patience, write down the score by hand!!! Time
consuming but for some situations the best solution so far.. .
Moreover, you can make use of full odd event functionalities of
Antescofo semantics (Trills, Multis etc.) and also put your cues
directly in the score using beat-time delays.

The current version can be used in a polyphonic situation (as the
examples suggest). But for ‘heavy’ polyphonic pieces (piano music
with pedal, orchestra etc.) I am in the process of making it as
robust as possible suitable for concert situations. This should be
ready before the end of November.

Also, soon you’ll be able to synchronize pitched events (like midi
scores!) as well as video (or anything else) concurrently. The main
dilemma for that today is not the synchronization system itself but
HOW to represent such data together in a coherent (multi-purpose,
multi-dimensional) container in Max… . Non existing up to now!

Hope this helps… .

Arshia Cont

http://cosmal.ucsd.edu/arshia/

#39951
Sep 22, 2008 at 1:03pm

Hi Arshia,

sounds good to me :-).

I already prepared the score file manually, which isn’t too problematic for short pieces. For the actual sound, I used some inbuilt drums from Ableton. The wav file was used as audio input together with the score text file.

For some odd reason, the recognition wasn’t too good. Is there any specific hints/documentation when using antescofo for percussive stuff (drums etc.) ? maybe some pointers what could cause issues ?

and another question: What’s the relevance of the beat duration if the tones are not sustained ?

thx, nick

#140916
Sep 23, 2008 at 7:50am

Hi again Nick,

Regarding recognition:

Note that Antescofo is a reactive system based on recognition. Therefore the AUDIO QUALITY of the analysis is of extreme importance and has nothing to do with the audio quality you’d want to achieve (for example) when using a harmonizer! Therefore to be able to answer your question:
- What kind of mic are you using (Cardio, Omni)? and in what environment (noisy with lots of background electronics etc)?
- What is the instrument in question? How do you mic it for score following?
- Do you use any hardware/software audio compressor?
To be more useful, in every concert situation we have a separate adc for any recognition algorithm such as Antescofo. Sometimes you’d need to ‘reduce’ audio quality (by compressing etc) to achieve more robust results. If you give me some hints on the above questions I can be more useful!

Your second question was “and another question: What’s the relevance of the beat duration if the tones are not sustained?”

Realtime tempo detection in Antescofo is based on IOI (or Inter-Onset-Interval). So as far as you have the same score in Antescofo as for the musician, things should go with no problem.

Cheers,

Arshia

#140917
Sep 23, 2008 at 8:17am

Hi Arshia,

we are not sure yet on what drums to use exactly, potentially the bass drum and something else (to be more robust).

In terms of mic’ing it, again the option to use standard mic’s (I am not a specialist, don’t know about the specifics) or trigger microphones. If I am not mistaken,trigger mic’s don’t provide audio signals but rather impulses (on/off). In that case, the whole observation system wouldn’t need to be tuned regarding noisiness, compressors etc. The question is whether antescofo would then be appropriate as there wouldn’t be any pitch or midi information anymore to analyse.

How robust is antescofo in terms of missed notes/drum hits etc. ?

nick

#140918
Sep 26, 2008 at 3:57pm

Hey Nick..

It seems to me that you don’t really need a score follower! What you really need is a tempo tracker that works smoothly on onsets (that you get from Bonk~ or others).

Ofcourse you can always use Antescofo in MIDI mode in order to get a pretty fair tempo curve. But you’d go to the hassle of having a score for it!
If you don’t care about PITCH, then try putting fake notes (let’s say all with pitch 60) but correct durations (in beats) and then simply use NEXTEVENT during the performance.. You’ll notice that the tempo is being updated.

We have used Antescofo Midi synchronized with an orchestra just weeks ago with BBC Scotish orchestra and it worked smoothly.

Antescofo is pretty smooth in terms of missed notes/jumps etc. If you notice anything strange let me know!

But once again.. you gotta figure out whether your problem is really a synch problem or just tempo tracking… .

Cheers,

Arshia

#140919
Sep 27, 2008 at 6:33pm

Hey Arshia,

the requirements are shifting quite a bit, the more one finds out :-). As it doesn’t matter whether sync’ing visual effects or firing off audio effects at specific times, it should be more than just tempo “following”.

I used bonk the other day, loading it with 8 different drum sounds (bd, tom, snare, etc.). Then I streamed some audio from my sequencer, and bonk successfully recognized the different instruments, returning the respective id.

I shifted the values by 60, thereby making it appear like midi notes to antescofo (using standard mode converting midi to frequency).

For some reason, if there was the same note (instrument) played twice after another, it didn’t recognize the second occurence.

Any ideas ?

nick

#140920
Sep 28, 2008 at 9:11pm

Hi Nick!

I’m happy that things are starting to move… .

For the case of “repeated notes”, make sure you somehow send in a NOTE OFF (pitch ZERO) to Antescofo, otherwise it’ll think you’re still on the same event without moving to the next event with the same pitch tag. That’s all!

For firing stuff, try the FWD, PFWD and LFWD commands.. . They couple with tempo in realtime and have proven to be quite musical and useful in performances… .

Let me know if there’s any other issues.

Cheers,

Arshia

#140921
Sep 29, 2008 at 9:26am

Hi arshia,

I already used noteoff’s between repeated notes/drum hits. Still, at one point, antescofo recognizes the repeated note, the second occurence though is not recognized.

It’s hard to “debug” over the forum, I could send you a zip with :

- the wav file
- the bonk patterns
- the antescofo score
- the patch
- (the mid file for the sequence)

Let me know whether I shall send it, don’t want to occupy you too much.

cheers, nick

#140922
Oct 7, 2008 at 10:24pm

Just to follow up Nick’s problem with Antescofo… The problem was mysteriously solved! So the current distribution is for now solid. Do not hesitate to bug me with bugs!

A note: You do not need to put fake silences between notes! Antescofo automatically puts dummy silences between notes even if there’s no silence indicated in the score… .

#140923

You must be logged in to reply to this topic.