could I write my own jit_qt_movie_matrix_to_image?


    Oct 17 2007 | 1:35 pm
    So that's the question - I'd really like to make my own version of jit.qt.movie, but I'm having trouble conceptualizing where the movie data becomes max-land matrix data. Or possibly even jit.gl.texture data. I don't really know if any of this is possible, but a movie-direct-to-texture object would be very useful, as would a stripped-down jit.qt.movie.
    I know you guys are probably super-busy with Max 5 and all that, but I'd really appreciate any advice on this subject. Perhaps I should just do this all using Cocoa / Objective C / QT?
    Thanks
    Evan

    • Oct 17 2007 | 1:46 pm
      Nuts, didn't read properly before I posted. I don't mean "jit_qt_movie_matrix_to_image" because that exports a movie to an image - I meant something more along the lines of taking a quicktime movie file, reading it, decompressing it, possibly changing it, and transferring it to jitter for further processing. Forgive the hasty initial post, please.
    • Oct 17 2007 | 3:01 pm
      On 07-10-17, at 0635, evan.raskob wrote:
      >
      > So that's the question - I'd really like to make my own version of
      > jit.qt.movie, but I'm having trouble conceptualizing where the
      > movie data becomes max-land matrix data. Or possibly even
      > jit.gl.texture data. I don't really know if any of this is
      > possible, but a movie-direct-to-texture object would be very
      > useful, as would a stripped-down jit.qt.movie.
      >
      > I know you guys are probably super-busy with Max 5 and all that,
      > but I'd really appreciate any advice on this subject. Perhaps I
      > should just do this all using Cocoa / Objective C / QT?
      >
      It is possible (movie-direct-to-texture), but it is a _lot of work.
      Writing your own movie-to-matrix object probably wouldn't give you
      much of a win over what is currently available from jit.qt.movie. If
      you want a stripped down version, couldn't you just use the [movie]
      Max object?
      r.
    • Oct 17 2007 | 4:54 pm
      Could you please elaborate more? What I'm thinking has to do with multi-threading, something which I don't think jit.qt.movie does (because multi-threading is mainly a 10.4 thing), but I'd love to be proved wrong about that. The theory is that this might get around frame skipping issues that jit.qt.movie has right now. But again, if I'm wrong, I'd really like to know.
      Thanks
      Evan
    • Oct 17 2007 | 5:24 pm
      On 07-10-17, at 0954, evan.raskob wrote:
      >
      > Could you please elaborate more? What I'm thinking has to do with
      > multi-threading, something which I don't think jit.qt.movie does
      > (because multi-threading is mainly a 10.4 thing), but I'd love to
      > be proved wrong about that. The theory is that this might get
      > around frame skipping issues that jit.qt.movie has right now. But
      > again, if I'm wrong, I'd really like to know.
      >
      As far as I know QuickTime is not yet multithreaded for movie
      playback, so threading won't help too much to prevent skipping. You
      can thread some operations so that loading a 2nd movie (for instance)
      doesn't affect an already playing one, but as for eliminating skips
      completely I haven't yet found a solution.
      Consider looking at the CoreVideo stuff, specifically the
      LiveVideoMixer sample code from Apple, which details how to set up a
      CVDisplayLink to drive movie playback from the video card.
      r.
    • Oct 17 2007 | 9:09 pm
      "ThreadsImportMovie demonstrates importing and displaying QuickTime Movies on separate threads."
      but how to hook this up to max? i'm having trouble figuring out how to go about this.
      and this opens the possibility of opening up multiple movies on different threads, i think?
      thanks
      evan
    • Oct 17 2007 | 9:26 pm
      On 07-10-17, at 1409, evan.raskob wrote:
      >
      > yes, especially: http://developer.apple.com/technotes/tn/tn2125.html
      >
      > "ThreadsImportMovie demonstrates importing and displaying QuickTime
      > Movies on separate threads."
      >
      > but how to hook this up to max? i'm having trouble figuring out how
      > to go about this.
      >
      > and this opens the possibility of opening up multiple movies on
      > different threads, i think?
      >
      "The Movie Toolbox provides the functionality that allows you to
      import, play, render, create, edit, and store time-based data. Movies
      cannot be played on background threads, therefore calling Movie
      Toolbox playback APIs such as StartMovie or SetMovieRate should only
      be done from the main thread."
      Opening movies on different threads is possible (although not for all
      codecs) but this will not address your frame dropping issue. Movie
      playback must happen on the main (UI) thread.
      This is not a straightforward project you are undertaking. I would
      recommend that you really familiarize yourself with both jit.gl.*
      sample code and the Apple sample code before you embark. If you have
      troubles understanding any of the sample code you'll have a lot of
      trouble. If you _do understand all of the sample code, then you
      should be able to figure out how to hook it all up. Good luck =)
      r.
    • Oct 17 2007 | 11:14 pm
      On Oct 17, 2007, at 10:26 PM, Ritchie Argue wrote:
      > On 07-10-17, at 1409, evan.raskob wrote:
      >>
      >> yes, especially: http://developer.apple.com/technotes/tn/tn2125.html
      >>
      >> "ThreadsImportMovie demonstrates importing and displaying
      >> QuickTime Movies on separate threads."
      >>
      >> but how to hook this up to max? i'm having trouble figuring out
      >> how to go about this.
      >>
      >> and this opens the possibility of opening up multiple movies on
      >> different threads, i think?
      >>
      > http://developer.apple.com/technotes/tn/tn2125.html#TNTAG3
      >
      > "The Movie Toolbox provides the functionality that allows you to
      > import, play, render, create, edit, and store time-based data.
      > Movies cannot be played on background threads, therefore calling
      > Movie Toolbox playback APIs such as StartMovie or SetMovieRate
      > should only be done from the main thread."
      > Opening movies on different threads is possible (although not for
      > all codecs) but this will not address your frame dropping issue.
      > Movie playback must happen on the main (UI) thread.
      I see. That makes sense. Still, I can't be sure where the frame
      dropping occurs - is it in the jit.window at display time, or are
      frames lost in the pipeline on the way there? Can I do some caching
      at some point to get around this? I searched a bit harder and found
      this page that seems to spell out a possible method for caching
      ahead: http://www.cycling74.com/twiki/bin/view/ProductDocumentation/
      JitterSdkUsingQTObjects
      But the more I look at this, the more I think its the scheduler
      itself that is the limitation. You can see similar effects with
      metro skipping beats (there was a thread on this awhile back). I'd
      need to go with the apple examples and forego max/jitter in order to
      get truly seamless playback, it seems.
      > This is not a straightforward project you are undertaking. I would
      > recommend that you really familiarize yourself with both jit.gl.*
      > sample code and the Apple sample code before you embark. If you
      > have troubles understanding any of the sample code you'll have a
      > lot of trouble. If you _do understand all of the sample code, then
      > you should be able to figure out how to hook it all up. Good luck =)
      >
      > r.
      Yes, certainly not straightforward... but now that I found the online
      jitter API documentation (I didn't realize it was separate from the
      max universal binary docs) I have a lot to work with. Still, it
      would be interesting to see some snippets of what jit.qt.movie
      actually does under the hood.
      Thanks
      Evan
    • Oct 17 2007 | 11:32 pm
      On 07-10-17, at 1625, evan.raskob [lists] wrote:
      >
      > I see. That makes sense. Still, I can't be sure where the frame
      > dropping occurs - is it in the jit.window at display time, or are
      > frames lost in the pipeline on the way there? Can I do some caching
      > at some point to get around this? I searched a bit harder and
      > found this page that seems to spell out a possible method for
      > caching ahead: http://www.cycling74.com/twiki/bin/view/
      > ProductDocumentation/JitterSdkUsingQTObjects
      >
      You could try the loadram argument to jit.qt.movie to see if that
      helps any. That's about as much as you can hope for in terms of
      caching QT, as it does a lot of caching and preloading behind the
      scenes already. Look at the various threads regarding prerolling
      movies, what that does, and how easy it is to accidentally invalidate
      the effects of prerolling.
      > But the more I look at this, the more I think its the scheduler
      > itself that is the limitation. You can see similar effects with
      > metro skipping beats (there was a thread on this awhile back). I'd
      > need to go with the apple examples and forego max/jitter in order
      > to get truly seamless playback, it seems.
      >
      I would recommend rendering a really obvious test clip that
      demonstrates the smoothness you are looking for, and play it back in
      some of the sample Apple QT projects. I've found even outside of Max,
      it's frustratingly difficult to never drop frames. This is just a
      limitation of the multitasking nature of consumer OSs these days. If
      you need guaranteed playback, something like realtime linux might be
      a better place to start (though good luck finding a decent media
      framework there).
      r.
    • Oct 18 2007 | 9:28 am
      On Oct 18, 2007, at 12:32 AM, Ritchie Argue wrote:
      > On 07-10-17, at 1625, evan.raskob [lists] wrote:
      >>
      >> I see. That makes sense. Still, I can't be sure where the frame
      >> dropping occurs - is it in the jit.window at display time, or are
      >> frames lost in the pipeline on the way there? Can I do some
      >> caching at some point to get around this? I searched a bit harder
      >> and found this page that seems to spell out a possible method for
      >> caching ahead: http://www.cycling74.com/twiki/bin/view/
      >> ProductDocumentation/JitterSdkUsingQTObjects
      >>
      > You could try the loadram argument to jit.qt.movie to see if that
      > helps any. That's about as much as you can hope for in terms of
      > caching QT, as it does a lot of caching and preloading behind the
      > scenes already. Look at the various threads regarding prerolling
      > movies, what that does, and how easy it is to accidentally
      > invalidate the effects of prerolling.
      Yes, tried a few variations (and realized just what you're saying
      about how easy it is to invalidate those prerolling effects!). Still
      skippage.
      >> But the more I look at this, the more I think its the scheduler
      >> itself that is the limitation. You can see similar effects with
      >> metro skipping beats (there was a thread on this awhile back).
      >> I'd need to go with the apple examples and forego max/jitter in
      >> order to get truly seamless playback, it seems.
      >>
      > I would recommend rendering a really obvious test clip that
      > demonstrates the smoothness you are looking for, and play it back
      > in some of the sample Apple QT projects. I've found even outside of
      > Max, it's frustratingly difficult to never drop frames. This is
      > just a limitation of the multitasking nature of consumer OSs these
      > days. If you need guaranteed playback, something like realtime
      > linux might be a better place to start (though good luck finding a
      > decent media framework there).
      I have a very smooth test clip in full PAL DVD resolution/frame rate,
      compressed in photojpeg, played back off a FW800 RAID drive on an 8-
      core Mac pro tower with 16GB RAM, and I see occasional skips in Max/
      Jitter but NOT in the various apple QT example applications (all the
      ones where they show you basic video effects) and quicktime player.
      (Examples from http://developer.apple.com/samplecode/QuickTime/
      idxMovieBasics-date.html - QTCarbonCoreImage101 is an interesting one
      that uses some overlaid effects)
      I completely agree that its frustratingly difficult to never drop
      frames, but now I'm trying to overcome that frustration :) There
      must be some way. Even if it's difficult or near-impossible I'd
      still like to know what the possibilities are!
      Thanks
      Evan
    • Oct 18 2007 | 4:37 pm
      it would certainly be worth a shot.
      my suggestion (which might be what you're already thinking) is to use corevideo to drive the video (and not use any of jitters-qt objects). let the external have its own timing mechanism and just render the video as an opengl texture, allowing it to use the opengl context of gl.render. i can send you the basics of retrieving the aglcontext if you need.
      the livevideomixer seems like a great place to start, and shouldn't take much work to test out the theory.
      if you haven't already, just build that example and see if it actually does handle the video playback better than jit-qt.
      -rob