making a 'vector outline' from a silhouette.


    Feb 15 2006 | 4:38 am
    I'm wanting to turn a live video feed of the silhouette of a body into a vector outline.
    I'm grabbing a live video image and converting into a single plane high-contrast image. So the body is black and the surrounding area is white.
    I want to do some kind of edge detection that gives me the co-ordinates of the black-white edge. But I want to list them in order so that if I joined them up using the lineto command in gl.sketch I would get a nice continuous shape.
    A bit like 'join-the-dots'. So that an outline of the figure can be drawn live.
    I've been pointed towards Cyclops, but I'm not sure this is right.
    All ideas most appreciated.

    • Feb 15 2006 | 5:44 am
      In a perfect world, the following would work:
      1) Take your silhohuette, threshold it to get a binary image
      2) take the gradient of the binary image (you can probably do just
      fine with jit.sobel, if not, you can use the finite difference
      method..if you do use this method, I can send you an object that does
      it)
      3) Threshold the gradient to get just the edge pixels
      4) spit out the cell coordinates of the edge pixels ( you can use
      xray.jit.cellcoord)
      5) convert from pixel to opengl coordinates
      6) draw
      Problems you might encounter:
      1)Image has noise, need to smooth it first. Try a median filter. I
      can send you one of these too if you need it.
      2)too many coordinates (oversampling)-->try out jit.demultiplex, but
      this might cause you to lose smaller shapes like fingers and feet.
      3) probably more issues, but no more come to mind.
      good luck,
      wes
    • Feb 15 2006 | 5:59 am
      > 2)too many coordinates (oversampling)-->try out jit.demultiplex, but
      > this might cause you to lose smaller shapes like fingers and feet.
      it would be worthwhile to try downsampling the image (pre-edge detect
      would be more cpu efficient, but could give better quality?) to a low
      resolution - play with the dim message that works best...no need to be
      a slave to 4x3!
      it is nice to know about the xray.jit.cellcoord. i started to reply to
      this email with a gedanken patch, but my brain got lazy/stumped at how
      to pull just the white coords w/ jitter's obejct set. Looks like wes
      solved that one, too.
      p.
    • Feb 15 2006 | 7:14 am
      You guys are working way too hard... Happy Valentine's.
      Andrew B.
    • Feb 15 2006 | 7:47 am
      That's pretty cool. You love that isosurf object don't you!
      wes
    • Feb 15 2006 | 3:45 pm
      *blush*
      I love all of Jitter's children.
      ;)
      AB
    • Feb 15 2006 | 8:27 pm
      And here's a little toy begat from Andrew's example:
      www.killingfrenzy.com/Jitter/Vectorizer.zip
      Sloppy, but fun to play with.
      Beware the "detail" setting on slower machines.
    • Feb 16 2006 | 2:10 am
      Marvellous help so far.
      Couldn't get Andrew's thing to open. (could see foo window but couldn't see the patch window). But the vectorizer patch is definitely a step in the right direction.
      But I want to be able to access the individual points that make up the shape. So isosurf is doing exactly the right thing in that it's making a shape from the silhouette. But I haven't got access to the data that makes up the shape.
      I started off making a patch like wesley's suggestion (listed underneath).
      I can get all of the coordinates and get them to draw a shape, but they aren't nicely sorted as they are in the isosurf examples. They just come out in matrix order. So when you use lineto in jit.gl.sketch, it draws the lines in matrix order, rather than the elegant outline manner of the isosurf example.
      How do I get the isosurf object to spit out the nicely sorted list of opengl coordinates it is using to draw the shape?
      The reason I need to do this is that I want to use the convert the co-ordinates into control voltages to send to an oscilloscope-type device.
      Currently I'm doing this through a soundcard that allows DC offset.
      Here's my crap patch that doesn't work at all well, just to show you where I've been....
      again, thanks for the help so far....
      max v2;
    • Feb 16 2006 | 3:06 am
      Hello Marcus,
      Can you elaborate this (if you don't mind of course) since I am kind
      of curious what you mean by oscilloscope-type device.
      Thank you so much!
      ilteris.
    • Feb 16 2006 | 4:23 am
      Well. it's an oscilloscope at the moment. But it might end up being a dangerously modified CRT projector.
    • Feb 16 2006 | 4:57 am
      and I forgot to join jit.robcross to jit.iter in my patch.
    • Feb 16 2006 | 5:23 am
      Hi Marcus,
      Here's some things I noticed about your patch. If possible, do math
      operations in matrix form. They're much faster than doing them in max
      messages. Second, jit.iter is always a bottleneck for large matrices
      (meaning > say 20 cells). Instead, hook your matrix up to jit.gl.mesh
      and render lines that way.
      I also noticed you had a jit.bsort object in your patch. If you
      decide you need to sort your matrix, I would recommend against using
      jit.bsort since it's quite a slow sorting algorithm. I would instead
      use xray.jit.quicksort. It's several times faster. The the patch
      below, I used xray.jit.cellcoords to grab white cells. You can
      Here's a crude response:
    • Feb 16 2006 | 5:56 am
      BTW, you can always do @matrixoutput 1 on jit.gl.isosurf.
      wes
    • Feb 16 2006 | 6:03 am
      It's funny that only text is shown when I open the saved patch.
      Am I missing any module not installed ?
      William
    • Feb 16 2006 | 7:59 am
      The X-ray objects are well worth having.
      xray.jit.cellcoord
      is in the patch.
    • Feb 16 2006 | 8:23 am
      I have been playing with X-ray objects happily now, but the problem on
      loading the saved patch presist.
      I guess the patch missed a few lines like,
      -- max v2;
      -- #N vpatcher 211 99 758 452;
      right ?
      By the way, I have looked into another project Wesley has done,
      "A-Contrario". The description of what it does is very interesting, but the
      project file downloadable doesn't work in my PC. Wesley, any chance we may
      get a windows version of the project ? We are missed "xray.jit.probsegment"
      and "xray.jit.median" to have it run.
      ^_^
      William
    • Feb 16 2006 | 8:30 am
      > I guess the patch missed a few lines like,
      > -- max v2;
      > -- #N vpatcher 211 99 758 452;
      > right ?
      What version of Max are you running?? The patch code pasted was from
      the clipboard and only works with the newer versions on Max (I think
      4.5 and >, but I could be wrong). Basically, it's not a full patcher
      but a patcher chunk.
      > By the way, I have looked into another project Wesley has done,
      > "A-Contrario". The description of what it does is very interesting, but the
      > project file downloadable doesn't work in my PC. Wesley, any chance we may
      > get a windows version of the project ? We are missed " xray.jit.probsegment"
      > and "xray.jit.median" to have it run.
      >
      True, these 2 objects are not compiled on a PC yet. I'll see if I
      can't find time this weekend to do it.
      wes
    • Feb 16 2006 | 10:43 am
      > What version of Max are you running?? The patch code pasted was from
      > the clipboard and only works with the newer versions on Max (I think
      > 4.5 and >, but I could be wrong). Basically, it's not a full patcher
      > but a patcher chunk.
      I am using the latest version of Max/Jitter. Maybe I am too green on using
      the program, I don't know the different between patcher and patcher chunk.
      I always copy and paste the texts into a text file and save with .pat
      extension. Usually it will work when I open it, but sometimes it fails with
      only text displayed. _|~~|O
      > True, these 2 objects are not compiled on a PC yet. I'll see if I
      > can't find time this weekend to do it.
      great to hear that~ thanks first~
      ^_^,
      William
    • Feb 16 2006 | 4:06 pm
      On 2/16/06, William Wong wrote:
      > > What version of Max are you running?? The patch code pasted was from
      > > the clipboard and only works with the newer versions on Max (I think
      > > 4.5 and >, but I could be wrong). Basically, it's not a full patcher
      > > but a patcher chunk.
      >
      Sorry, those weren't technical terms. If you're using the latest max,
      you should be able to open a new patcher window and paste the text in
      there. The text will magically become a patcher.
      wes
    • Feb 17 2006 | 1:27 am
      Thanks for all the excellent advice so far.
      For any newbies (like myself) following this thread, I've run into a couple of issues.
      I installed the xray objects but couldn't see them in max.
      So I built an object list and put it into the cycling74/init folder. Bingo!
      Couldn't find one with the installer. This one isn't very elegant (sorry Wes) but seems to do the trick.
      But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself.
      Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears.
      I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy.....
      I'm probably missing something very obvious here.....
    • Feb 17 2006 | 1:52 am
      > I installed the xray objects but couldn't see them in max.
      >
      Do you mean they didn't show up in the New Object List? Or when you
      right click on the patcher background? If you put the objects in
      /jitter-externals/xray then they are definitely in your search path
      and you should be able to use them without having to make a list.
      Personally, I don't use the object lists in max so I've never bothered
      to make one.
      >
      > But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself.
      >
      > Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears.
      >
      > I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy.....
      >
      > I'm probably missing something very obvious here.....
      >
      What are the precise steps you do to take text and turn it into a
      patcher? I'm working at 1280x854. This shouldn't make a difference.
      Do you take the text and paste it into a new max patcher? This is the
      way you should go. I've never had a problem with this technique but
      maybe your system doesn't take to it somehow. If you paste the text
      into a new patcher, the patcher won't be offscreen as it's right there
      where you created it.
      best,
      wes
    • Feb 17 2006 | 8:34 am
      still don't see them in the right-click context.
      but with the list they show up in the new object window.
      But.....on the other matter....
      I've been opening a text file, copying the posted patch text into that and then saving it. Then opening as a patch. Dumbkopf!
      This is more like it!
      Please excuse my ignorance. Will examine patch and let you know how I get on.
      xray.jit.cellcoords looks like the daddy....
      Many thanks
    • Feb 18 2006 | 3:39 am
      So using matrixoutput on jit.gl.isosurf seems to be spitting out the gl co-ordinates in order. Which is fantastic.
      The only thing is that there are rather a lot of them.
      The dimension of the matrix seem much bigger than they should be. ie, if the dimensions given to the isosurf object are 80 x 60 x 2, then the resulting 'matrixoutput' matrix is
      1 x 144000.
      Most of this matrix is just zeroes.
      My ideal scenario would be to filter this matrixoutput data so that I get a 2d matrix at around the same resolution as what I'm feeding the isosurf object. So if isosurf res is 80 x 60, I get a matrixoutput 1 x 480.
      I don't need the third dimension as I'm only drawing in 2d.
      Trying jit.submatrix, etc but no consistent joy yet.
      The reason I need to filter the information is that it will eventually be fed as a 2d 1 x 1920 matrix to jit.release @ 25fps to create a 2 channel 48k audio signal.
      Sorry for the length of this saga. Here's an amalgam patch.
    • Feb 23 2006 | 8:10 am
      The patch below outputs a single line matrix of about 5-6000 points.
      But if I try to reduce it down to my target of a fixed matrix 1920 points, the drawing order gets messed up.
      Damn!
      I've bastardised the cellvalue help file to show what I mean.
      Try pressing the adapt switch.
      bball works well as an example.
      Any tips on maintaining the drawing order or am I trying to square a circle?
    • Feb 23 2006 | 8:33 am
      ok,
      You can get less point and better efficiency by using a smaller
      matrix. I set the dims to 80 x 60 and the contours basically look the
      same. About the contouring algorithm. It doesn't follow curves at
      all. It works by scanning through a matrix of values one row after
      the other. If you imagine the video image in grayscale format as a
      heightmap with bright values as peaks and black values as valleys, the
      contouring algorithm is looking for the location of particular
      heights. Obviously, this is going to be between pixels for the most
      part. So, when the algorithm encounters a change in value from one
      pixel to another that passes through the level it is looking for, it
      creates a line at that location. The result of this is that the
      output is sets of pairs of points without regard for order. When you
      force the output into a fixed size matrix, you are mucking up the
      carfully chosen pairs of points and getting bad connections between
      lines. If you absolutely must have a fixed number of points, I'm
      afraid this approach will only cause many headaches. If you can
      handle a certain number of points below a certain maximum, this
      approach can work.
      I would suggest playing with the dimensions of the matrix you apply
      the contouring algorithm to as that will determine for the most part
      about how many points you get back. then,I would add a safety that
      clips the size of the matrix to a certain value.
      If you absolutely must have the points in order, I'm afraid you're out
      of luck as both the contourmap and the isolevel algorithms say nothing
      about connectivity. This requires quite a bit of extra logic that is
      a pain in th arse to implement, which is why no one has done it for a
      jitter object so far (as far as I know). I did a rudimentary
      connectivity-based object a few months ago, and it was really
      difficult to say the least.
      best,
      wes
    • Feb 24 2006 | 1:35 am
      Right. Got it. The lines just appear to be joined up. But they aren't.
      I spoke to a technological friend about this issue. Here's how he thought it could be done.
      1) Make the silhouette completely black and white. (object is white)
      2) Scan the matrix from the top for the first pixel containing white. Record its position.
      3) Make this pixel the centre of a 3x3 region. Search the region clockwise for the next white pixel. If you find one, check that it's not one that you've already recorded.
      4) If you can't find a white pixel, expand the region to 5x5, 7x7 until you do. If you've hit the edge of frame, move along the edge of frame until you find another white pixel.
      5) Record the newly found pixel and make it the centre of a 3x3 region.
      6) Repeat the process until you get back to the first pixel you started with.
      Obviously this means you'll only get a solid shape around one object, but this would be sufficient for my needs.
      So, great in theory. But obviously not too easy.
      Does it sound far too difficult to implement this with real-time video?
      Are there any existing objects or code that already do some of this?
      Looks like something to do in js or code? Any ideas of what resources it would need to make something like this work?
      If it was reasonably economical,I would be more than happy to commission someone with superior coding skills. I have realised that coding is not my thing.
    • Nov 13 2011 | 8:58 pm
      I'm reviving this thread as there was a lot of good work in it that pertains exactly to what i'm trying to do.
      Lyallmarcus, did you ever get this going as you wished?
      I've been struggling the last few days trying to find a way to output the contour of in a sequential order (in order of connectedness).
      as you mentioned in your last post, you have to implement an algorithm: find the first "on" pixel from the edge of the matrix, and then follow its connected neighbors until you get to the end of the line.
      but i see no way of doing this with the objects i know of.
    • Jan 24 2012 | 11:54 pm
      A variant on this is the marching squares algorithm (marching cubes for 3D shapes).