making a 'vector outline' from a silhouette.
I'm wanting to turn a live video feed of the silhouette of a body into a vector outline.
I'm grabbing a live video image and converting into a single plane high-contrast image. So the body is black and the surrounding area is white.
I want to do some kind of edge detection that gives me the co-ordinates of the black-white edge. But I want to list them in order so that if I joined them up using the lineto command in gl.sketch I would get a nice continuous shape.
A bit like 'join-the-dots'. So that an outline of the figure can be drawn live.
I've been pointed towards Cyclops, but I'm not sure this is right.
All ideas most appreciated.
In a perfect world, the following would work:
1) Take your silhohuette, threshold it to get a binary image
2) take the gradient of the binary image (you can probably do just
fine with jit.sobel, if not, you can use the finite difference
method..if you do use this method, I can send you an object that does
it)
3) Threshold the gradient to get just the edge pixels
4) spit out the cell coordinates of the edge pixels ( you can use
xray.jit.cellcoord)
5) convert from pixel to opengl coordinates
6) draw
Problems you might encounter:
1)Image has noise, need to smooth it first. Try a median filter. I
can send you one of these too if you need it.
2)too many coordinates (oversampling)-->try out jit.demultiplex, but
this might cause you to lose smaller shapes like fingers and feet.
3) probably more issues, but no more come to mind.
good luck,
wes
> 2)too many coordinates (oversampling)-->try out jit.demultiplex, but
> this might cause you to lose smaller shapes like fingers and feet.
it would be worthwhile to try downsampling the image (pre-edge detect
would be more cpu efficient, but could give better quality?) to a low
resolution - play with the dim message that works best...no need to be
a slave to 4x3!
it is nice to know about the xray.jit.cellcoord. i started to reply to
this email with a gedanken patch, but my brain got lazy/stumped at how
to pull just the white coords w/ jitter's obejct set. Looks like wes
solved that one, too.
p.
You guys are working way too hard... Happy Valentine's.
Andrew B.
That's pretty cool. You love that isosurf object don't you!
wes
*blush*
I love all of Jitter's children.
;)
AB
And here's a little toy begat from Andrew's example:
www.killingfrenzy.com/Jitter/Vectorizer.zip
Sloppy, but fun to play with.
Beware the "detail" setting on slower machines.
Marvellous help so far.
Couldn't get Andrew's thing to open. (could see foo window but couldn't see the patch window). But the vectorizer patch is definitely a step in the right direction.
But I want to be able to access the individual points that make up the shape. So isosurf is doing exactly the right thing in that it's making a shape from the silhouette. But I haven't got access to the data that makes up the shape.
I started off making a patch like wesley's suggestion (listed underneath).
I can get all of the coordinates and get them to draw a shape, but they aren't nicely sorted as they are in the isosurf examples. They just come out in matrix order. So when you use lineto in jit.gl.sketch, it draws the lines in matrix order, rather than the elegant outline manner of the isosurf example.
How do I get the isosurf object to spit out the nicely sorted list of opengl coordinates it is using to draw the shape?
The reason I need to do this is that I want to use the convert the co-ordinates into control voltages to send to an oscilloscope-type device.
Currently I'm doing this through a soundcard that allows DC offset.
Here's my crap patch that doesn't work at all well, just to show you where I've been....
again, thanks for the help so far....
Hello Marcus,
Can you elaborate this (if you don't mind of course) since I am kind
of curious what you mean by oscilloscope-type device.
Thank you so much!
ilteris.
Well. it's an oscilloscope at the moment. But it might end up being a dangerously modified CRT projector.
and I forgot to join jit.robcross to jit.iter in my patch.
Hi Marcus,
Here's some things I noticed about your patch. If possible, do math
operations in matrix form. They're much faster than doing them in max
messages. Second, jit.iter is always a bottleneck for large matrices
(meaning > say 20 cells). Instead, hook your matrix up to jit.gl.mesh
and render lines that way.
I also noticed you had a jit.bsort object in your patch. If you
decide you need to sort your matrix, I would recommend against using
jit.bsort since it's quite a slow sorting algorithm. I would instead
use xray.jit.quicksort. It's several times faster. The the patch
below, I used xray.jit.cellcoords to grab white cells. You can
download it here: http://www.mat.ucsb.edu/~whsmith/ .
Here's a crude response:
BTW, you can always do @matrixoutput 1 on jit.gl.isosurf.
wes
It's funny that only text is shown when I open the saved patch.
Am I missing any module not installed ?
William
The X-ray objects are well worth having.
http://www.mat.ucsb.edu/~whsmith/
xray.jit.cellcoord
is in the patch.
I have been playing with X-ray objects happily now, but the problem on
loading the saved patch presist.
I guess the patch missed a few lines like,
-- max v2;
-- #N vpatcher 211 99 758 452;
right ?
By the way, I have looked into another project Wesley has done,
"A-Contrario". The description of what it does is very interesting, but the
project file downloadable doesn't work in my PC. Wesley, any chance we may
get a windows version of the project ? We are missed "xray.jit.probsegment"
and "xray.jit.median" to have it run.
^_^
William
> I guess the patch missed a few lines like,
> -- max v2;
> -- #N vpatcher 211 99 758 452;
> right ?
What version of Max are you running?? The patch code pasted was from
the clipboard and only works with the newer versions on Max (I think
4.5 and >, but I could be wrong). Basically, it's not a full patcher
but a patcher chunk.
> By the way, I have looked into another project Wesley has done,
> "A-Contrario". The description of what it does is very interesting, but the
> project file downloadable doesn't work in my PC. Wesley, any chance we may
> get a windows version of the project ? We are missed " xray.jit.probsegment"
> and "xray.jit.median" to have it run.
>
True, these 2 objects are not compiled on a PC yet. I'll see if I
can't find time this weekend to do it.
wes
> What version of Max are you running?? The patch code pasted was from
> the clipboard and only works with the newer versions on Max (I think
> 4.5 and >, but I could be wrong). Basically, it's not a full patcher
> but a patcher chunk.
I am using the latest version of Max/Jitter. Maybe I am too green on using
the program, I don't know the different between patcher and patcher chunk.
I always copy and paste the texts into a text file and save with .pat
extension. Usually it will work when I open it, but sometimes it fails with
only text displayed. _|~~|O
> True, these 2 objects are not compiled on a PC yet. I'll see if I
> can't find time this weekend to do it.
great to hear that~ thanks first~
^_^,
William
On 2/16/06, William Wong wrote:
> > What version of Max are you running?? The patch code pasted was from
> > the clipboard and only works with the newer versions on Max (I think
> > 4.5 and >, but I could be wrong). Basically, it's not a full patcher
> > but a patcher chunk.
>
Sorry, those weren't technical terms. If you're using the latest max,
you should be able to open a new patcher window and paste the text in
there. The text will magically become a patcher.
wes
Thanks for all the excellent advice so far.
For any newbies (like myself) following this thread, I've run into a couple of issues.
I installed the xray objects but couldn't see them in max.
So I built an object list and put it into the cycling74/init folder. Bingo!
Couldn't find one with the installer. This one isn't very elegant (sorry Wes) but seems to do the trick.
But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself.
Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears.
I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy.....
I'm probably missing something very obvious here.....
> I installed the xray objects but couldn't see them in max.
>
Do you mean they didn't show up in the New Object List? Or when you
right click on the patcher background? If you put the objects in
/jitter-externals/xray then they are definitely in your search path
and you should be able to use them without having to make a list.
Personally, I don't use the object lists in max so I've never bothered
to make one.
>
> But my main issue is that if I open the patches that have been posted, I only see a jit.window and not the patch itself.
>
> Same problem on Andrew and Wes's patches. Thought it had something to do with the lines at the top of the text. I'm assuming these refer to where the patcher window appears.
>
> I thought that because I'm on a little powerbook, maybe the patcher window was out of sight. So edited the text so that it's the same as the patch I posted. Still no joy.....
>
> I'm probably missing something very obvious here.....
>
What are the precise steps you do to take text and turn it into a
patcher? I'm working at 1280x854. This shouldn't make a difference.
Do you take the text and paste it into a new max patcher? This is the
way you should go. I've never had a problem with this technique but
maybe your system doesn't take to it somehow. If you paste the text
into a new patcher, the patcher won't be offscreen as it's right there
where you created it.
best,
wes
still don't see them in the right-click context.
but with the list they show up in the new object window.
But.....on the other matter....
I've been opening a text file, copying the posted patch text into that and then saving it. Then opening as a patch. Dumbkopf!
This is more like it!
Please excuse my ignorance. Will examine patch and let you know how I get on.
xray.jit.cellcoords looks like the daddy....
Many thanks
So using matrixoutput on jit.gl.isosurf seems to be spitting out the gl co-ordinates in order. Which is fantastic.
The only thing is that there are rather a lot of them.
The dimension of the matrix seem much bigger than they should be. ie, if the dimensions given to the isosurf object are 80 x 60 x 2, then the resulting 'matrixoutput' matrix is
1 x 144000.
Most of this matrix is just zeroes.
My ideal scenario would be to filter this matrixoutput data so that I get a 2d matrix at around the same resolution as what I'm feeding the isosurf object. So if isosurf res is 80 x 60, I get a matrixoutput 1 x 480.
I don't need the third dimension as I'm only drawing in 2d.
Trying jit.submatrix, etc but no consistent joy yet.
The reason I need to filter the information is that it will eventually be fed as a 2d 1 x 1920 matrix to jit.release @ 25fps to create a 2 channel 48k audio signal.
Sorry for the length of this saga. Here's an amalgam patch.
The patch below outputs a single line matrix of about 5-6000 points.
But if I try to reduce it down to my target of a fixed matrix 1920 points, the drawing order gets messed up.
Damn!
I've bastardised the cellvalue help file to show what I mean.
Try pressing the adapt switch.
bball works well as an example.
Any tips on maintaining the drawing order or am I trying to square a circle?
ok,
You can get less point and better efficiency by using a smaller
matrix. I set the dims to 80 x 60 and the contours basically look the
same. About the contouring algorithm. It doesn't follow curves at
all. It works by scanning through a matrix of values one row after
the other. If you imagine the video image in grayscale format as a
heightmap with bright values as peaks and black values as valleys, the
contouring algorithm is looking for the location of particular
heights. Obviously, this is going to be between pixels for the most
part. So, when the algorithm encounters a change in value from one
pixel to another that passes through the level it is looking for, it
creates a line at that location. The result of this is that the
output is sets of pairs of points without regard for order. When you
force the output into a fixed size matrix, you are mucking up the
carfully chosen pairs of points and getting bad connections between
lines. If you absolutely must have a fixed number of points, I'm
afraid this approach will only cause many headaches. If you can
handle a certain number of points below a certain maximum, this
approach can work.
I would suggest playing with the dimensions of the matrix you apply
the contouring algorithm to as that will determine for the most part
about how many points you get back. then,I would add a safety that
clips the size of the matrix to a certain value.
If you absolutely must have the points in order, I'm afraid you're out
of luck as both the contourmap and the isolevel algorithms say nothing
about connectivity. This requires quite a bit of extra logic that is
a pain in th arse to implement, which is why no one has done it for a
jitter object so far (as far as I know). I did a rudimentary
connectivity-based object a few months ago, and it was really
difficult to say the least.
best,
wes
Right. Got it. The lines just appear to be joined up. But they aren't.
I spoke to a technological friend about this issue. Here's how he thought it could be done.
1) Make the silhouette completely black and white. (object is white)
2) Scan the matrix from the top for the first pixel containing white. Record its position.
3) Make this pixel the centre of a 3x3 region. Search the region clockwise for the next white pixel. If you find one, check that it's not one that you've already recorded.
4) If you can't find a white pixel, expand the region to 5x5, 7x7 until you do. If you've hit the edge of frame, move along the edge of frame until you find another white pixel.
5) Record the newly found pixel and make it the centre of a 3x3 region.
6) Repeat the process until you get back to the first pixel you started with.
Obviously this means you'll only get a solid shape around one object, but this would be sufficient for my needs.
So, great in theory. But obviously not too easy.
Does it sound far too difficult to implement this with real-time video?
Are there any existing objects or code that already do some of this?
Looks like something to do in js or code? Any ideas of what resources it would need to make something like this work?
If it was reasonably economical,I would be more than happy to commission someone with superior coding skills. I have realised that coding is not my thing.
I'm reviving this thread as there was a lot of good work in it that pertains exactly to what i'm trying to do.
Lyallmarcus, did you ever get this going as you wished?
I've been struggling the last few days trying to find a way to output the contour of in a sequential order (in order of connectedness).
as you mentioned in your last post, you have to implement an algorithm: find the first "on" pixel from the edge of the matrix, and then follow its connected neighbors until you get to the end of the line.
but i see no way of doing this with the objects i know of.
A variant on this is the marching squares algorithm (marching cubes for 3D shapes).