making a 'vector outline' from a silhouette.


    Feb 15 2006 | 4:38 am
    I'm wanting to turn a live video feed of the silhouette of a body into a vector outline.
    I'm grabbing a live video image and converting into a single plane high-contrast image. So the body is black and the surrounding area is white.
    I want to do some kind of edge detection that gives me the co-ordinates of the black-white edge. But I want to list them in order so that if I joined them up using the lineto command in gl.sketch I would get a nice continuous shape.
    A bit like 'join-the-dots'. So that an outline of the figure can be drawn live.
    I've been pointed towards Cyclops, but I'm not sure this is right.
    All ideas most appreciated.

    • Feb 15 2006 | 5:44 am
      In a perfect world, the following would work:
      1) Take your silhohuette, threshold it to get a binary image 2) take the gradient of the binary image (you can probably do just fine with jit.sobel, if not, you can use the finite difference method..if you do use this method, I can send you an object that does it) 3) Threshold the gradient to get just the edge pixels 4) spit out the cell coordinates of the edge pixels ( you can use xray.jit.cellcoord) 5) convert from pixel to opengl coordinates 6) draw
      Problems you might encounter: 1)Image has noise, need to smooth it first. Try a median filter. I can send you one of these too if you need it. 2)too many coordinates (oversampling)-->try out jit.demultiplex, but this might cause you to lose smaller shapes like fingers and feet. 3) probably more issues, but no more come to mind.
      good luck, wes
    • Feb 15 2006 | 5:59 am
      > 2)too many coordinates (oversampling)-->try out jit.demultiplex, but > this might cause you to lose smaller shapes like fingers and feet.
      it would be worthwhile to try downsampling the image (pre-edge detect would be more cpu efficient, but could give better quality?) to a low resolution - play with the dim message that works best...no need to be a slave to 4x3!
      it is nice to know about the xray.jit.cellcoord. i started to reply to this email with a gedanken patch, but my brain got lazy/stumped at how to pull just the white coords w/ jitter's obejct set. Looks like wes solved that one, too.
      p.
    • Feb 15 2006 | 7:14 am
      You guys are working way too hard... Happy Valentine's.
      Andrew B.
    • Feb 15 2006 | 7:47 am
      That's pretty cool. You love that isosurf object don't you!
      wes
    • Feb 15 2006 | 3:45 pm
      *blush*
      I love all of Jitter's children.
      ;) AB
    • Feb 15 2006 | 8:27 pm
      And here's a little toy begat from Andrew's example: www.killingfrenzy.com/Jitter/Vectorizer.zip Sloppy, but fun to play with. Beware the "detail" setting on slower machines.
    • Feb 16 2006 | 2:10 am
      Marvellous help so far.
      Couldn't get Andrew's thing to open. (could see foo window but couldn't see the patch window). But the vectorizer patch is definitely a step in the right direction.
      But I want to be able to access the individual points that make up the shape. So isosurf is doing exactly the right thing in that it's making a shape from the silhouette. But I haven't got access to the data that makes up the shape.
      I started off making a patch like wesley's suggestion (listed underneath).
      I can get all of the coordinates and get them to draw a shape, but they aren't nicely sorted as they are in the isosurf examples. They just come out in matrix order. So when you use lineto in jit.gl.sketch, it draws the lines in matrix order, rather than the elegant outline manner of the isosurf example.
      How do I get the isosurf object to spit out the nicely sorted list of opengl coordinates it is using to draw the shape?
      The reason I need to do this is that I want to use the convert the co-ordinates into control voltages to send to an oscilloscope-type device.
      Currently I'm doing this through a soundcard that allows DC offset.
      Here's my crap patch that doesn't work at all well, just to show you where I've been....
      again, thanks for the help so far....
    • Feb 16 2006 | 3:06 am
      Hello Marcus,
      Can you elaborate this (if you don't mind of course) since I am kind of curious what you mean by oscilloscope-type device. Thank you so much!
      ilteris.
    • Feb 16 2006 | 4:23 am
      Well. it's an oscilloscope at the moment. But it might end up being a dangerously modified CRT projector.
    • Feb 16 2006 | 4:57 am
      and I forgot to join jit.robcross to jit.iter in my patch.
    • Feb 16 2006 | 5:23 am
      Hi Marcus, Here's some things I noticed about your patch. If possible, do math operations in matrix form. They're much faster than doing them in max messages. Second, jit.iter is always a bottleneck for large matrices (meaning > say 20 cells). Instead, hook your matrix up to jit.gl.mesh and render lines that way.
      I also noticed you had a jit.bsort object in your patch. If you decide you need to sort your matrix, I would recommend against using jit.bsort since it's quite a slow sorting algorithm. I would instead use xray.jit.quicksort. It's several times faster. The the patch below, I used xray.jit.cellcoords to grab white cells. You can download it here: http://www.mat.ucsb.edu/~whsmith/ .
      Here's a crude response:
    • Feb 16 2006 | 5:56 am
      BTW, you can always do @matrixoutput 1 on jit.gl.isosurf.
      wes
    • Feb 16 2006 | 6:03 am
      It's funny that only text is shown when I open the saved patch. Am I missing any module not installed ?
      William
    • Feb 16 2006 | 7:59 am
      The X-ray objects are well worth having. http://www.mat.ucsb.edu/~whsmith/
      xray.jit.cellcoord is in the patch.
    • Feb 16 2006 | 8:23 am
      I have been playing with X-ray objects happily now, but the problem on loading the saved patch presist.
      I guess the patch missed a few lines like, -- max v2; -- #N vpatcher 211 99 758 452; right ?
      By the way, I have looked into another project Wesley has done, "A-Contrario". The description of what it does is very interesting, but the project file downloadable doesn't work in my PC. Wesley, any chance we may get a windows version of the project ? We are missed "xray.jit.probsegment" and "xray.jit.median" to have it run.
      ^_^ William
    • Feb 16 2006 | 8:30 am
      > I guess the patch missed a few lines like, > -- max v2; > -- #N vpatcher 211 99 758 452; > right ?
      What version of Max are you running?? The patch code pasted was from the clipboard and only works with the newer versions on Max (I think 4.5 and >, but I could be wrong). Basically, it's not a full patcher but a patcher chunk.
      > By the way, I have looked into another project Wesley has done, > "A-Contrario". The description of what it does is very interesting, but the > project file downloadable doesn't work in my PC. Wesley, any chance we may > get a windows version of the project ? We are missed " xray.jit.probsegment" > and "xray.jit.median" to have it run. >
      True, these 2 objects are not compiled on a PC yet. I'll see if I can't find time this weekend to do it.
      wes
    • Feb 16 2006 | 10:43 am
      > What version of Max are you running?? The patch code pasted was from > the clipboard and only works with the newer versions on Max (I think > 4.5 and >, but I could be wrong). Basically, it's not a full patcher > but a patcher chunk.
      I am using the latest version of Max/Jitter. Maybe I am too green on using the program, I don't know the different between patcher and patcher chunk.
      I always copy and paste the texts into a text file and save with .pat extension. Usually it will work when I open it, but sometimes it fails with only text displayed. _|~~|O
      > True, these 2 objects are not compiled on a PC yet. I'll see if I > can't find time this weekend to do it.
      great to hear that~ thanks first~
      ^_^, William
    • Feb 16 2006 | 4:06 pm
      On 2/16/06, William Wong wrote: > > What version of Max are you running?? The patch code pasted was from > > the clipboard and only works with the newer versions on Max (I think > > 4.5 and >, but I could be wrong). Basically, it's not a full patcher > > but a patcher chunk. >
      Sorry, those weren't technical terms. If you're using the latest max, you should be able to open a new patcher window and paste the text in there. The text will magically become a patcher.
      wes
    • Feb 17 2006 | 1:27 am
      Thanks for all the excellent advice so far.
      For any newbies (like myself) following this thread, I've run into a couple of issues.
      I installed the xray objects but couldn't see them in max.
      So I built an object list and put it into the cycling74/init folder. Bingo!
      Couldn't find one with the installer. This one isn't very elegant (sorry Wes) but seems to do the trick.
      But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself.
      Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears.
      I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy.....
      I'm probably missing something very obvious here.....
    • Feb 17 2006 | 1:52 am
      > I installed the xray objects but couldn't see them in max. > Do you mean they didn't show up in the New Object List? Or when you right click on the patcher background? If you put the objects in /jitter-externals/xray then they are definitely in your search path and you should be able to use them without having to make a list. Personally, I don't use the object lists in max so I've never bothered to make one. > > But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself. > > Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears. > > I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy..... > > I'm probably missing something very obvious here..... >
      What are the precise steps you do to take text and turn it into a patcher? I'm working at 1280x854. This shouldn't make a difference. Do you take the text and paste it into a new max patcher? This is the way you should go. I've never had a problem with this technique but maybe your system doesn't take to it somehow. If you paste the text into a new patcher, the patcher won't be offscreen as it's right there where you created it.
      best, wes
    • Feb 17 2006 | 8:34 am
      still don't see them in the right-click context. but with the list they show up in the new object window.
      But.....on the other matter....
      I've been opening a text file, copying the posted patch text into that and then saving it. Then opening as a patch. Dumbkopf!
      This is more like it!
      Please excuse my ignorance. Will examine patch and let you know how I get on.
      xray.jit.cellcoords looks like the daddy....
      Many thanks
    • Feb 18 2006 | 3:39 am
      So using matrixoutput on jit.gl.isosurf seems to be spitting out the gl co-ordinates in order. Which is fantastic.
      The only thing is that there are rather a lot of them.
      The dimension of the matrix seem much bigger than they should be. ie, if the dimensions given to the isosurf object are 80 x 60 x 2, then the resulting 'matrixoutput' matrix is
      1 x 144000.
      Most of this matrix is just zeroes.
      My ideal scenario would be to filter this matrixoutput data so that I get a 2d matrix at around the same resolution as what I'm feeding the isosurf object. So if isosurf res is 80 x 60, I get a matrixoutput 1 x 480.
      I don't need the third dimension as I'm only drawing in 2d.
      Trying jit.submatrix, etc but no consistent joy yet.
      The reason I need to filter the information is that it will eventually be fed as a 2d 1 x 1920 matrix to jit.release @ 25fps to create a 2 channel 48k audio signal.
      Sorry for the length of this saga. Here's an amalgam patch.
    • Feb 23 2006 | 8:10 am
      The patch below outputs a single line matrix of about 5-6000 points.
      But if I try to reduce it down to my target of a fixed matrix 1920 points, the drawing order gets messed up.
      Damn!
      I've bastardised the cellvalue help file to show what I mean. Try pressing the adapt switch.
      bball works well as an example.
      Any tips on maintaining the drawing order or am I trying to square a circle?
    • Feb 23 2006 | 8:33 am
      ok, You can get less point and better efficiency by using a smaller matrix. I set the dims to 80 x 60 and the contours basically look the same. About the contouring algorithm. It doesn't follow curves at all. It works by scanning through a matrix of values one row after the other. If you imagine the video image in grayscale format as a heightmap with bright values as peaks and black values as valleys, the contouring algorithm is looking for the location of particular heights. Obviously, this is going to be between pixels for the most part. So, when the algorithm encounters a change in value from one pixel to another that passes through the level it is looking for, it creates a line at that location. The result of this is that the output is sets of pairs of points without regard for order. When you force the output into a fixed size matrix, you are mucking up the carfully chosen pairs of points and getting bad connections between lines. If you absolutely must have a fixed number of points, I'm afraid this approach will only cause many headaches. If you can handle a certain number of points below a certain maximum, this approach can work.
      I would suggest playing with the dimensions of the matrix you apply the contouring algorithm to as that will determine for the most part about how many points you get back. then,I would add a safety that clips the size of the matrix to a certain value.
      If you absolutely must have the points in order, I'm afraid you're out of luck as both the contourmap and the isolevel algorithms say nothing about connectivity. This requires quite a bit of extra logic that is a pain in th arse to implement, which is why no one has done it for a jitter object so far (as far as I know). I did a rudimentary connectivity-based object a few months ago, and it was really difficult to say the least.
      best, wes
    • Feb 24 2006 | 1:35 am
      Right. Got it. The lines just appear to be joined up. But they aren't.
      I spoke to a technological friend about this issue. Here's how he thought it could be done.
      1) Make the silhouette completely black and white. (object is white)
      2) Scan the matrix from the top for the first pixel containing white. Record its position.
      3) Make this pixel the centre of a 3x3 region. Search the region clockwise for the next white pixel. If you find one, check that it's not one that you've already recorded.
      4) If you can't find a white pixel, expand the region to 5x5, 7x7 until you do. If you've hit the edge of frame, move along the edge of frame until you find another white pixel.
      5) Record the newly found pixel and make it the centre of a 3x3 region.
      6) Repeat the process until you get back to the first pixel you started with.
      Obviously this means you'll only get a solid shape around one object, but this would be sufficient for my needs.
      So, great in theory. But obviously not too easy.
      Does it sound far too difficult to implement this with real-time video?
      Are there any existing objects or code that already do some of this?
      Looks like something to do in js or code? Any ideas of what resources it would need to make something like this work?
      If it was reasonably economical,I would be more than happy to commission someone with superior coding skills. I have realised that coding is not my thing.
    • Nov 13 2011 | 8:58 pm
      I'm reviving this thread as there was a lot of good work in it that pertains exactly to what i'm trying to do.
      Lyallmarcus, did you ever get this going as you wished?
      I've been struggling the last few days trying to find a way to output the contour of in a sequential order (in order of connectedness).
      as you mentioned in your last post, you have to implement an algorithm: find the first "on" pixel from the edge of the matrix, and then follow its connected neighbors until you get to the end of the line.
      but i see no way of doing this with the objects i know of.
    • Jan 24 2012 | 11:54 pm
      A variant on this is the marching squares algorithm (marching cubes for 3D shapes).