jit.net.send/recv limitations


    Aug 31 2006 | 4:05 pm
    Hi,
    i'm developing a video-mixer patch that takes in up to three quicktime movies and two cameras. The final output is a long triple-screen (16:3 ratio) matrix that is sent to 3 projectors.
    since this is a quite intensive patch i created two subpatches that send data from the first to the second patch through the jet.net.send/recv objects.
    the first patch reads in the single-screen movies/cameras and creates two decks (each a triple-screen matrix - 16:3). So in numbers this is:
    - sources: reading 5x (800x600) matrix
    - processing: glueing them together to 2x (2400x600) matrix [i select for each deck three of the five sources to be glueed in an order i decide using the router and jit.glue objects]
    - output: sending out 2x (2400x600) 4 plane char matrix through jit.net.send/recv.
    if i want to be able to send this over UTP with 25 fps, i would need a 10Gbit LAN interface on both machines (pc + laptop running WinXp). Gigabit LAN should get me up to 10 fps.
    (more numbers: 10fps = little less than 1 Gbit/s - 15fps = 1,38 Gbit/s - 25 fps = 2,30 Gbit/s - this if my calculations are correct offcourse)
    I was wondering if, apart from redesigning my patch to sending just one deck over lan, if there is a hardware configuration which could allow me to send data from my laptop to the pc so that at the desired speed. Or am I just really wrong trying to accomplish this?
    Maybe i should notice that my laptop is approximately running at 75 to 80 pct of ist CPU usage, so adding a lot more functionality to my first patch is not really what i'm looking for,...au contraire...
    thanks for all advice
    d

    • Aug 31 2006 | 9:35 pm
      > i'm developing a video-mixer patch that takes in up to three
      > quicktime movies and two cameras.
      ...
      > Or am I just really wrong trying to accomplish this?
      i just love to give out hi-res multi-projector advice.. so here goes
      A) it is most definitely unrealistic to send those huge matrices over
      a network
      B) you probably need to redesign your mixer architecture, but the
      final result is achievable.
      C) with a proper design you should be able to do it on a single
      desktop machine
      I am assuming by your numbers that each projector resolution is
      800x600 SVGA, there is no actual rule that says that your movies must
      be at the native projector resolution, unless you are steadfast on
      having a 1to1 correlation in pixel resolution. the scalars on either
      the gpu or the hardware scalars built into the projectors themselves
      can be quite good at scaling...
      The perception of resolution is in fact very subjective, primarily a
      factor of audience distance to the screen:
      The larger the screen, the more likely the audience is to be viewing
      at a distance, the farther you are from screen, the resolution
      matters much less, because ours eyes 'mix' the pixles for us, and are
      unable to perceive fine detail anyway. therefore the less likely it
      becomes that you actually need 1to1 pixels to achieve an excellent
      result.
      sit in the front row of any movie theater, and then move to the back
      row and you'll see exactly what i mean.
      just a small side example, 4 years ago i did an 8 channel piece on a
      single SXGA projector in a long 'strip' as you suggest, each channel
      was only 160x120, but the end result was a hyper long 1280x120
      resolution image with all those channels playing... the perception
      for the audience was as exhilarating as if they watched an omnimax
      panoramic display -- my point being, that you are on the right track,
      but you probably do not need to push the outer limits of resolution/
      processing power to achieve it.
      i would suggest using 400x300 source/process resolution, (because
      scalars do the best job when the source is exactly 1/4 (1/2 depending
      on you look at it) the resolution -- indeed, your cameras are most
      likely 640x480 (perhaps not if you've gone for a machine vision
      solution) , so you would already be scaling up those two sources up
      before mixing.
      if you accept this law of scale and perception, than you could easily
      perform everything on a single machine with two graphics cards.
      source and mixing all happens at 400x300 (or 640x480 if you like),
      then allow the scalars to do the hard work for you.
      if you are not already, you will achieve much more efficient results
      with mixing using jit.gl.slab, and also your sources in uyvy mode...
      it might not be out of the question to use 800x600 given this
      powerful combination
      > The final output is a long triple-screen (16:3 ratio) matrix that
      > is sent to 3 projectors.
      > - processing: glueing them together to 2x (2400x600) matrix [i
      > select for each deck three of the five sources to be glueed in an
      > order i decide using the router and jit.glue objects]
      > - output: sending out 2x (2400x600) 4 plane char matrix through
      > jit.net.send/recv.
      this part is slightly confusing-- not sure why you are 'glueing' the
      three streams together for mixing, AFAIK you still need to send
      800x600 stream to three vga/dvi outputs -- are you trying to achieve
      a 'seamless' image? are you using 'edge blending' to achieve this?
      otherwise you WILL have seams :)
      how are you mixing the 5 channels into 3 (that is, 5x 800 =4000, not
      2400) if i am missing something here let me know.
      //
      there is one limitation i should mention, you have 3 projectors, so
      you you need 3 outputs, and as far as i can see, you want one long
      matrix to 'span' across 3 desktops. unfortunately opengl cards do
      very poorly at spanning images ACROSS CARDS. they do perfectly fine
      spanning across 2 outputs of the same card though. so you may want
      to consider redesigning the system to use 2 SXGA projectors instead
      of 3 SVGA. 2x SXGA = 2560x1024 -- you do not need to utilize the
      full horizontal resolution, just leave your desktop black and display
      a 2400x600 window. now you only need one dedicated graphics card for
      display, and greatly simplify your architecture in terms of 'gluing'
      lastly, if you want to explore hardware solutions, check out http://
      folsomresearch.com/ -- the screenpro and/or blendpro might be of
      interest to you, and can be rented from most video/tradeshow
      production houses. this is typically how its done in the 'pro' world
      - not to say that its not achievable in jitter (it is, as several
      here including myself will attest)
      by the way, your aspect ratio is 12x3 not 16x3 ; )
      good luck!
      --DeKam
      >
      >
      > since this is a quite intensive patch i created two subpatches that
      > send data from the first to the second patch through the
      > jet.net.send/recv objects.
      >
      > the first patch reads in the single-screen movies/cameras and
      > creates two decks (each a triple-screen matrix - 16:3). So in
      > numbers this is:
      > - sources: reading 5x (800x600) matrix
      >
      >
      > if i want to be able to send this over UTP with 25 fps, i would
      > need a 10Gbit LAN interface on both machines (pc + laptop running
      > WinXp). Gigabit LAN should get me up to 10 fps.
      > (more numbers: 10fps = little less than 1 Gbit/s - 15fps = 1,38
      > Gbit/s - 25 fps = 2,30 Gbit/s - this if my calculations are correct
      > offcourse)
      >
      > I was wondering if, apart from redesigning my patch to sending just
      > one deck over lan, if there is a hardware configuration which could
      > allow me to send data from my laptop to the pc so that at the
      > desired speed.
      >
      > Maybe i should notice that my laptop is approximately running at 75
      > to 80 pct of ist CPU usage, so adding a lot more functionality to
      > my first patch is not really what i'm looking for,...au contraire...
      >
      > thanks for all advice
      > d
      >
      >
      >
    • Sep 06 2006 | 12:01 am
      Hi DeKam,
      First off, sorry for the late reply and thanks for the input!
      ... I tried replying earlier but just now i noticed that the message bounced since i tried attaching an image using my mail client... o well, here it is anyway
      (my comments to your post are between /* and */...)
      > i'm developing a video-mixer patch that takes in up to three
      > quicktime movies and two cameras.
      ...
      > Or am I just really wrong trying to accomplish this?
      i just love to give out hi-res multi-projector advice.. so here goes
      A) it is most definitely unrealistic to send those huge matrices over
      a network
      B) you probably need to redesign your mixer architecture, but the
      final result is achievable.
      C) with a proper design you should be able to do it on a single
      desktop machine
      I am assuming by your numbers that each projector resolution is
      800x600 SVGA, there is no actual rule that says that your movies must
      be at the native projector resolution, unless you are steadfast on
      having a 1to1 correlation in pixel resolution. the scalars on either
      the gpu or the hardware scalars built into the projectors themselves
      can be quite good at scaling...
      The perception of resolution is in fact very subjective, primarily a
      factor of audience distance to the screen:
      The larger the screen, the more likely the audience is to be viewing
      at a distance, the farther you are from screen, the resolution
      matters much less, because ours eyes 'mix' the pixles for us, and are
      unable to perceive fine detail anyway. therefore the less likely it
      becomes that you actually need 1to1 pixels to achieve an excellent
      result.
      sit in the front row of any movie theater, and then move to the back
      row and you'll see exactly what i mean.
      just a small side example, 4 years ago i did an 8 channel piece on a
      single SXGA projector in a long 'strip' as you suggest, each channel
      was only 160x120, but the end result was a hyper long 1280x120
      resolution image with all those channels playing... the perception
      for the audience was as exhilarating as if they watched an omnimax
      panoramic display -- my point being, that you are on the right track,
      but you probably do not need to push the outer limits of resolution/
      processing power to achieve it.
      i would suggest using 400x300 source/process resolution, (because
      scalars do the best job when the source is exactly 1/4 (1/2 depending
      on you look at it) the resolution -- indeed, your cameras are most
      likely 640x480 (perhaps not if you've gone for a machine vision
      solution) , so you would already be scaling up those two sources up
      before mixing.
      /*
      This is indeed a good argument to start working with smaller matrices; allthough, when i first tested the patch on the screens (which are three screens -3m x 4m- standing next to each other in an open-cube-like configuration), i was using 640 x 480 matrices which resulted in quite pixelated images. Anyhow, i will investigate this issue further and try to work with smaller matrices, which will definitely result in more breathing space for me and my pc and laptop.
      */
      if you accept this law of scale and perception, than you could easily
      perform everything on a single machine with two graphics cards.
      source and mixing all happens at 400x300 (or 640x480 if you like),
      then allow the scalars to do the hard work for you.
      if you are not already, you will achieve much more efficient results
      with mixing using jit.gl.slab, and also your sources in uyvy mode...
      it might not be out of the question to use 800x600 given this
      powerful combination
      /*
      As mentioned before, for now I am only using processing power. It is definitely not an option that the final mixer will work completely on the CPU, so i will dig deeper in the gl objects to get better performance. the reason why i'm starting out based on the CPU is because I thought it might be a good 'lower border' for performance. So if get reasonable results using only the CPU i can go a step further and make an even better patch using the GPU. Considering your arguments, and my instinct, i think i'd better start using the GPU straight away since, i'll need its performance anyhow.
      */
      > The final output is a long triple-screen (16:3 ratio) matrix that
      > is sent to 3 projectors.
      > - processing: glueing them together to 2x (2400x600) matrix [i
      > select for each deck three of the five sources to be glueed in an
      > order i decide using the router and jit.glue objects]
      > - output: sending out 2x (2400x600) 4 plane char matrix through
      > jit.net.send/recv.
      this part is slightly confusing-- not sure why you are 'glueing' the
      three streams together for mixing, AFAIK you still need to send
      800x600 stream to three vga/dvi outputs -- are you trying to achieve
      a 'seamless' image? are you using 'edge blending' to achieve this?
      otherwise you WILL have seams :)
      how are you mixing the 5 channels into 3 (that is, 5x 800 =4000, not
      2400) if i am missing something here let me know.
      //
      /*
      actually, what i'm doing is, I have 5 inputs of which i only use 3 for each deck. to select which sources to use, i use a the max router object, more precise as ' router 5 3 '. afterwards i glue these three output matrices (from the router) together with jit.glue.
      */
      there is one limitation i should mention, you have 3 projectors, so
      you you need 3 outputs, and as far as i can see, you want one long
      matrix to 'span' across 3 desktops. unfortunately opengl cards do
      very poorly at spanning images ACROSS CARDS. they do perfectly fine
      spanning across 2 outputs of the same card though. so you may want
      to consider redesigning the system to use 2 SXGA projectors instead
      of 3 SVGA. 2x SXGA = 2560x1024 -- you do not need to utilize the
      full horizontal resolution, just leave your desktop black and display
      a 2400x600 window. now you only need one dedicated graphics card for
      display, and greatly simplify your architecture in terms of 'gluing'
      /*
      You should know that i am using a Matrox Triplehead2Go 'screensplitter' This little box shows up to my pc as being a 12x3 (thanks for the correction :) screen, and has three outputs; for three screens or projectors.
      I attached a little basic sketch of the hardware setup which should make things a bit more clear.
      */
      lastly, if you want to explore hardware solutions, check out http://
      folsomresearch.com/ -- the screenpro and/or blendpro might be of
      interest to you, and can be rented from most video/tradeshow
      production houses. this is typically how its done in the 'pro' world
      - not to say that its not achievable in jitter (it is, as several
      here including myself will attest)
      /*
      nice hardware products, but i'm afraid they're an itsypitsy out of mine and the project's price range ;)
      */
      by the way, your aspect ratio is 12x3 not 16x3 ; )
      /*
      indeedio
      */
      good luck!
      --DeKam
      /*
      Thanks
      */
      >
      >
      > since this is a quite intensive patch i created two subpatches that
      > send data from the first to the second patch through the
      > jet.net.send /recv objects.
      >
      > the first patch reads in the single-screen movies/cameras and
      > creates two decks (each a triple-screen matrix - 16:3). So in
      > numbers this is:
      > - sources: reading 5x (800x600) matrix
      >
      >
      > if i want to be able to send this over UTP with 25 fps, i would
      > need a 10Gbit LAN interface on both machines (pc + laptop running
      > WinXp). Gigabit LAN should get me up to 10 fps.
      > (more numbers: 10fps = little less than 1 Gbit/s - 15fps = 1,38
      > Gbit/s - 25 fps = 2,30 Gbit/s - this if my calculations are correct
      > offcourse)
      >
      > I was wondering if, apart from redesigning my patch to sending just
      > one deck over lan, if there is a hardware configuration which could
      > allow me to send data from my laptop to the pc so that at the
      > desired speed.
      >
      > Maybe i should notice that my laptop is approximately running at 75
      > to 80 pct of ist CPU usage, so adding a lot more functionality to
      > my first patch is not really what i'm looking for,...au contraire...
      >
      > thanks for all advice
      > d
      >
      > _______________________________________________
      > jitter mailing list
      > jitter@cycling74.com
      > http://www.cycling74.com/mailman/listinfo/jitter
      >
      >
    • Sep 06 2006 | 12:27 am
      Just so you know,
      i've been altering my patch to make use of the GPU. This results in much better performance... however it's still not optimal.
      I can fairly say that the bottleneck now, is no longer the jit.net.send/recv objects, so i'll be moving my lamentation to another topic :)
      cheers!
      d
    • Sep 06 2006 | 9:38 am
      A smart little solution for a wide-screen 3-projector setup we use in some situations is to render to a 1080 x 288 window (on one 1152 x 864 resolution output). We send this to a video splitter (DC VS-814) and to 3 beamers that support zooming (e.g. the Sharp Notevision PG-A20X Projector). Each beamer zooms to another portion of the incoming signal.
      So that makes 360 x 288 per screen.
      > The perception of resolution is in fact very subjective
      I agree. We've been performing with this resolution for fairly large crowds, with fairly large screens, both projections and led-screens and I never heard complaints about resolution. Of course there are better solutions for this but the priority is not very high.. the content is much more important.
      Cheers,
      Mattijs
    • Sep 06 2006 | 2:46 pm
      > /*
      > This is indeed a good argument to start working with smaller
      > matrices; allthough, when i first tested the patch on the screens
      > (which are three screens -3m x 4m- standing next to each other in
      > an open-cube-like configuration), i was using 640 x 480 matrices
      > which resulted in quite pixelated images. Anyhow, i will
      > investigate this issue further and try to work with smaller
      > matrices, which will definitely result in more breathing space for
      > me and my pc and laptop.
      > */
      it boils down to your own subjective decision, as its is often
      dependent on the content itself (for example 'photographic' content
      shot with a camera may scale better than things generated digitally)
      > Considering your arguments, and my instinct, i think i'd better
      > start using the GPU straight away since, i'll need its performance
      > anyhow.
      > */
      good!
      > You should know that i am using a Matrox Triplehead2Go
      > 'screensplitter' This little box shows up to my pc as being a 12x3
      > (thanks for the correction :) screen, and has three outputs; for
      > three screens or projectors.
      > I attached a little basic sketch of the hardware setup which should
      > make things a bit more clear.
      > */
      ok - personally i haven't used this, but i've heard good things from
      friends on the list :)
      > i'm experiencing low framerates when i'm loading in three large
      > quicktime movies simultaneously. Each of the files is about 1.7GB
      > and i paste em all three together using the jit.glue object. This
      > is all rendered on the GPU.
      > I'm pretty sure that it's my Harddisk (7200 rpm) that cannot keep
      > up,... so i was wondering if there's a way to work around this issue.
      you are correct intuitively, the movie datarate is a large bottleneck
      to overcome. -- we have discussed this issue on the list previously,
      if you search on other threads i've posted before - essentially, the
      codec you compress the movie in will have the big effect, and
      naturally you should also have an adequate disk subsystem capable of
      handling the load.
      The best work around is simply to add more hardrives(!) -- a low end
      RAID 0, such as eSATA, internal SATA, or even a firewire 800 'dual
      disk' raid will greatly improve things. the main thing is to have
      multiple disks reading the movies in. I've had good luck with a 3
      drive internal SATA raid, which can be had for as little as $500.
      For codecs, the most favored codec generally is PhotoJPEG at 70-75%
      quality - it offers a great balance of quality, datarate and cpu
      load. (keep in mind that different codecs require different CPU
      loads to decompress) Others use 'Animation' codec, which yields 2:1
      and is very cheap on the cpu, but the datarate is still quite high.
      Most shy away from modern 'h264', 'mp4', 'pixlet' and the like
      because they are expensive on the cpu. MPEG2 generally sucks for
      quality, but works for some. If you invest in a wicked wicked disk
      subsystem, you might opt for the 'Component' codec which is
      temporally and spatially uncompressed, great quality, zero CPU,
      native for jitter's uyvy mode, but also very high datarate (thus you
      need STRONG disks for this)
      I would buy more hardrives before you spend any time on 'buffering'
      schemes in Max. at the very least you can read 'one movie per disk'
      without a raid, even on cheap FW400 external disks, and have better
      luck than trying to read all 3 from one 7200 disk.
      --deKam
    • Sep 09 2006 | 8:07 pm
      Thanks for the tip!
      using more disks is indeed the best solution... once again i was searching answers too far.
      A nice disk subsystem is on my list!