could I write my own jit_qt_movie_matrix_to_image?

Oct 17, 2007 at 1:35pm

could I write my own jit_qt_movie_matrix_to_image?

So that’s the question – I’d really like to make my own version of jit.qt.movie, but I’m having trouble conceptualizing where the movie data becomes max-land matrix data. Or possibly even jit.gl.texture data. I don’t really know if any of this is possible, but a movie-direct-to-texture object would be very useful, as would a stripped-down jit.qt.movie.

I know you guys are probably super-busy with Max 5 and all that, but I’d really appreciate any advice on this subject. Perhaps I should just do this all using Cocoa / Objective C / QT?

Thanks
Evan

#34211
Oct 17, 2007 at 1:46pm

Nuts, didn’t read properly before I posted. I don’t mean “jit_qt_movie_matrix_to_image” because that exports a movie to an image – I meant something more along the lines of taking a quicktime movie file, reading it, decompressing it, possibly changing it, and transferring it to jitter for further processing. Forgive the hasty initial post, please.

#115065
Oct 17, 2007 at 3:01pm

On 07-10-17, at 0635, evan.raskob wrote:
>
> So that’s the question – I’d really like to make my own version of
> jit.qt.movie, but I’m having trouble conceptualizing where the
> movie data becomes max-land matrix data. Or possibly even
> jit.gl.texture data. I don’t really know if any of this is
> possible, but a movie-direct-to-texture object would be very
> useful, as would a stripped-down jit.qt.movie.
>
> I know you guys are probably super-busy with Max 5 and all that,
> but I’d really appreciate any advice on this subject. Perhaps I
> should just do this all using Cocoa / Objective C / QT?
>
It is possible (movie-direct-to-texture), but it is a _lot of work.
Writing your own movie-to-matrix object probably wouldn’t give you
much of a win over what is currently available from jit.qt.movie. If
you want a stripped down version, couldn’t you just use the [movie]
Max object?

r.

#115066
Oct 17, 2007 at 4:54pm

Could you please elaborate more? What I’m thinking has to do with multi-threading, something which I don’t think jit.qt.movie does (because multi-threading is mainly a 10.4 thing), but I’d love to be proved wrong about that. The theory is that this might get around frame skipping issues that jit.qt.movie has right now. But again, if I’m wrong, I’d really like to know.

Thanks
Evan

#115067
Oct 17, 2007 at 5:24pm

On 07-10-17, at 0954, evan.raskob wrote:
>
> Could you please elaborate more? What I’m thinking has to do with
> multi-threading, something which I don’t think jit.qt.movie does
> (because multi-threading is mainly a 10.4 thing), but I’d love to
> be proved wrong about that. The theory is that this might get
> around frame skipping issues that jit.qt.movie has right now. But
> again, if I’m wrong, I’d really like to know.
>
As far as I know QuickTime is not yet multithreaded for movie
playback, so threading won’t help too much to prevent skipping. You
can thread some operations so that loading a 2nd movie (for instance)
doesn’t affect an already playing one, but as for eliminating skips
completely I haven’t yet found a solution.

Consider looking at the CoreVideo stuff, specifically the
LiveVideoMixer sample code from Apple, which details how to set up a
CVDisplayLink to drive movie playback from the video card.

r.

#115068
Oct 17, 2007 at 9:09pm

yes, especially: http://developer.apple.com/technotes/tn/tn2125.html

“ThreadsImportMovie demonstrates importing and displaying QuickTime Movies on separate threads.”

but how to hook this up to max? i’m having trouble figuring out how to go about this.

and this opens the possibility of opening up multiple movies on different threads, i think?

thanks
evan

#115069
Oct 17, 2007 at 9:26pm

On 07-10-17, at 1409, evan.raskob wrote:
>
> yes, especially: http://developer.apple.com/technotes/tn/tn2125.html
>
> “ThreadsImportMovie demonstrates importing and displaying QuickTime
> Movies on separate threads.”
>
> but how to hook this up to max? i’m having trouble figuring out how
> to go about this.
>
> and this opens the possibility of opening up multiple movies on
> different threads, i think?
>

http://developer.apple.com/technotes/tn/tn2125.html#TNTAG3

“The Movie Toolbox provides the functionality that allows you to
import, play, render, create, edit, and store time-based data. Movies
cannot be played on background threads, therefore calling Movie
Toolbox playback APIs such as StartMovie or SetMovieRate should only
be done from the main thread.”

Opening movies on different threads is possible (although not for all
codecs) but this will not address your frame dropping issue. Movie
playback must happen on the main (UI) thread.

This is not a straightforward project you are undertaking. I would
recommend that you really familiarize yourself with both jit.gl.*
sample code and the Apple sample code before you embark. If you have
troubles understanding any of the sample code you’ll have a lot of
trouble. If you _do understand all of the sample code, then you
should be able to figure out how to hook it all up. Good luck =)

r.

#115070
Oct 17, 2007 at 11:14pm

On Oct 17, 2007, at 10:26 PM, Ritchie Argue wrote:

> On 07-10-17, at 1409, evan.raskob wrote:
>>
>> yes, especially: http://developer.apple.com/technotes/tn/tn2125.html
>>
>> “ThreadsImportMovie demonstrates importing and displaying
>> QuickTime Movies on separate threads.”
>>
>> but how to hook this up to max? i’m having trouble figuring out
>> how to go about this.
>>
>> and this opens the possibility of opening up multiple movies on
>> different threads, i think?
>>
> http://developer.apple.com/technotes/tn/tn2125.html#TNTAG3
>
> “The Movie Toolbox provides the functionality that allows you to
> import, play, render, create, edit, and store time-based data.
> Movies cannot be played on background threads, therefore calling
> Movie Toolbox playback APIs such as StartMovie or SetMovieRate
> should only be done from the main thread.”

> Opening movies on different threads is possible (although not for
> all codecs) but this will not address your frame dropping issue.
> Movie playback must happen on the main (UI) thread.

I see. That makes sense. Still, I can’t be sure where the frame
dropping occurs – is it in the jit.window at display time, or are
frames lost in the pipeline on the way there? Can I do some caching
at some point to get around this? I searched a bit harder and found
this page that seems to spell out a possible method for caching
ahead: http://www.cycling74.com/twiki/bin/view/ProductDocumentation/
JitterSdkUsingQTObjects

But the more I look at this, the more I think its the scheduler
itself that is the limitation. You can see similar effects with
metro skipping beats (there was a thread on this awhile back). I’d
need to go with the apple examples and forego max/jitter in order to
get truly seamless playback, it seems.

> This is not a straightforward project you are undertaking. I would
> recommend that you really familiarize yourself with both jit.gl.*
> sample code and the Apple sample code before you embark. If you
> have troubles understanding any of the sample code you’ll have a
> lot of trouble. If you _do understand all of the sample code, then
> you should be able to figure out how to hook it all up. Good luck =)
>
> r.

Yes, certainly not straightforward… but now that I found the online
jitter API documentation (I didn’t realize it was separate from the
max universal binary docs) I have a lot to work with. Still, it
would be interesting to see some snippets of what jit.qt.movie
actually does under the hood.

Thanks
Evan

#115071
Oct 17, 2007 at 11:32pm

On 07-10-17, at 1625, evan.raskob [lists] wrote:
>
> I see. That makes sense. Still, I can’t be sure where the frame
> dropping occurs – is it in the jit.window at display time, or are
> frames lost in the pipeline on the way there? Can I do some caching
> at some point to get around this? I searched a bit harder and
> found this page that seems to spell out a possible method for
> caching ahead: http://www.cycling74.com/twiki/bin/view/
> ProductDocumentation/JitterSdkUsingQTObjects
>
You could try the loadram argument to jit.qt.movie to see if that
helps any. That’s about as much as you can hope for in terms of
caching QT, as it does a lot of caching and preloading behind the
scenes already. Look at the various threads regarding prerolling
movies, what that does, and how easy it is to accidentally invalidate
the effects of prerolling.

> But the more I look at this, the more I think its the scheduler
> itself that is the limitation. You can see similar effects with
> metro skipping beats (there was a thread on this awhile back). I’d
> need to go with the apple examples and forego max/jitter in order
> to get truly seamless playback, it seems.
>
I would recommend rendering a really obvious test clip that
demonstrates the smoothness you are looking for, and play it back in
some of the sample Apple QT projects. I’ve found even outside of Max,
it’s frustratingly difficult to never drop frames. This is just a
limitation of the multitasking nature of consumer OSs these days. If
you need guaranteed playback, something like realtime linux might be
a better place to start (though good luck finding a decent media
framework there).

r.

#115072
Oct 18, 2007 at 9:28am

On Oct 18, 2007, at 12:32 AM, Ritchie Argue wrote:

> On 07-10-17, at 1625, evan.raskob [lists] wrote:
>>
>> I see. That makes sense. Still, I can’t be sure where the frame
>> dropping occurs – is it in the jit.window at display time, or are
>> frames lost in the pipeline on the way there? Can I do some
>> caching at some point to get around this? I searched a bit harder
>> and found this page that seems to spell out a possible method for
>> caching ahead: http://www.cycling74.com/twiki/bin/view/
>> ProductDocumentation/JitterSdkUsingQTObjects
>>
> You could try the loadram argument to jit.qt.movie to see if that
> helps any. That’s about as much as you can hope for in terms of
> caching QT, as it does a lot of caching and preloading behind the
> scenes already. Look at the various threads regarding prerolling
> movies, what that does, and how easy it is to accidentally
> invalidate the effects of prerolling.

Yes, tried a few variations (and realized just what you’re saying
about how easy it is to invalidate those prerolling effects!). Still
skippage.

>> But the more I look at this, the more I think its the scheduler
>> itself that is the limitation. You can see similar effects with
>> metro skipping beats (there was a thread on this awhile back).
>> I’d need to go with the apple examples and forego max/jitter in
>> order to get truly seamless playback, it seems.
>>
> I would recommend rendering a really obvious test clip that
> demonstrates the smoothness you are looking for, and play it back
> in some of the sample Apple QT projects. I’ve found even outside of
> Max, it’s frustratingly difficult to never drop frames. This is
> just a limitation of the multitasking nature of consumer OSs these
> days. If you need guaranteed playback, something like realtime
> linux might be a better place to start (though good luck finding a
> decent media framework there).

I have a very smooth test clip in full PAL DVD resolution/frame rate,
compressed in photojpeg, played back off a FW800 RAID drive on an 8-
core Mac pro tower with 16GB RAM, and I see occasional skips in Max/
Jitter but NOT in the various apple QT example applications (all the
ones where they show you basic video effects) and quicktime player.
(Examples from http://developer.apple.com/samplecode/QuickTime/
idxMovieBasics-date.html – QTCarbonCoreImage101 is an interesting one
that uses some overlaid effects)

I completely agree that its frustratingly difficult to never drop
frames, but now I’m trying to overcome that frustration :) There
must be some way. Even if it’s difficult or near-impossible I’d
still like to know what the possibilities are!

Thanks
Evan

#115073
Oct 18, 2007 at 4:37pm

it would certainly be worth a shot.

my suggestion (which might be what you’re already thinking) is to use corevideo to drive the video (and not use any of jitters-qt objects). let the external have its own timing mechanism and just render the video as an opengl texture, allowing it to use the opengl context of gl.render. i can send you the basics of retrieving the aglcontext if you need.

the livevideomixer seems like a great place to start, and shouldn’t take much work to test out the theory.
if you haven’t already, just build that example and see if it actually does handle the video playback better than jit-qt.
-rob

#115074

You must be logged in to reply to this topic.