openpose for Max --- Skeleton Tracking based on Artificial Intelligence on 2D images (no Kinect)

hollyhook

9月 02 2017 | 4:08 午後

Max external for detecting persons in 2D images - real-time capable, using the awesome openpose library.

Windows x64 only, needs a nvidia graphic card. you can get the external jit.openpose.pose and a simple jsui for visualization here, for free.

jbailey

9月 02 2017 | 5:22 午後

incredible.

matmat

9月 03 2017 | 11:39 午前

awesome

phiol

9月 03 2017 | 4:27 午後

thanks a lot for sharing HOLLYHOOK

Louis TB

11月 01 2017 | 8:10 午後

Thanks for sharing ! :) Im really stoked to try it. I would have one question , what do i have to add specificly to the Path environment variable to get the extrenal to access the DLLs ?

I get error 126 when trying to load the external.

Thanks Again

ygreq

2月 04 2018 | 5:21 午後

How did you manage to use it? Is anyone able to help with the installment of the dependecies for running openPose? Thank you!!

I tried following what it says on the github, but it is too much for a noob like me

phiol

2月 04 2018 | 8:17 午後

never got it to work , the external is broken

enrico wiltsch

2月 05 2018 | 9:31 午前

Look amazing but never got it to work too...

matmat

2月 05 2018 | 9:46 午前

Same here,
I would love to test but, I get error 126 ...

ygreq

2月 05 2018 | 10:16 午前

My guess is that it is not broken. You just have to install all the dependencies. From what I figured from the installation page (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md), it says you need to install openCV and Caffe (I don't know why it says Ubuntu only. My guess is that we need to install these as well). Then install CMake GUI and Visual Studio. Andonly after that, you build OpenPose and test it. And if everything works, my guess is that the external will work as well.

But that is easier said then done. So I ask again. :) Can someone help us noobs with a for-noobs tutorial on how to install all that? Hollyhook? Please!! At least can you confirm what I wrote above?

Thank you so much!!
ygr

PS: Did you guys manage to install all the dependencies?

kcoul

2月 05 2018 | 7:28 午後

This sounds interesting. I'll give it a try and if I can get it to work, I'll make up a tutorial for how to get it working that will be not so developer-centric ;)

hollyhook

2月 05 2018 | 7:58 午後

The openpose developer had a openpose demo, which came with pre-build libraries for openpose and some 3rd party DLLs. The idea was, if you are able to run the demo (with cuda etc.), you can take all the dlls it comes with, put them on the PC somewhere so that windows can find them (in PATH) and the external runs. I can't find the openpose demo anymore, and they also have improved the build system. So i should re-build the external again, on the base of the new openpose... I'm quite busy right now, but will do it as soon as i find time. To be honest, I didn't expect that anyone uses it, because the feedback was kind of thin.. so didn't care about updating.

Btw, if anybody would like to try to build it, I'm happy to give the sources of the external away.

Btw2, Seen this? https://twitter.com/NataliaNeverova/status/959530358613176320 "code coming soon" \o/
could also be candidate for wrapping into a max external, most likely with the same limitations (PC only, Nvidia, DLL mess)

kcoul

2月 06 2018 | 1:14 午前

Thanks for that link, it looks much more interesting, surpassing Kinect instead of catching up to it.

Hopefully it works in realtime, will check the paper.

ygreq

2月 07 2018 | 8:11 午前

Thank you, Hollyhook, for you answer. My first observation regarding you mentioning that there wasn't such great interest in this object is the fact that people don't know about it existing. OpenPose, I mean. BTW, can you modify the title to something like: "OpenPose: Kinect Improved Alternative to Skeleton Tracking based on Artificial Intelligence". And I am adding a tag, Machine Learning. :) This way many maxers will find out about it for sure and interest will grow.

Now regarding what you said, maybe you can link a download to the build in case someone wants and knows how to fix it. But hopefully you have a bit of time to look into it and fix it. That would be really great. I can barely wait to try it!

I will start looking for that demo now. :) My guess is that maybe if I use the same code as you did, with the same demo, it might work, right? I might have to ask you other things about this later on. :)

And your link with the people masking with the DensePose is genius! Hope they manage to put the code out and someone manages to import it in Max as well. :)) Like I said, the only programming language I know is Max. Unfortunately! I already have great ideas for some projects involving this!!!

Thank you so much!

And again, if anyone manages to make this work, please help the whole community and write all the steps you took in detail. Thank you so much!

vichug

2月 07 2018 | 9:18 午前

waw that's pretty rad ! hopefully the densepose framework can be extended to non windows machines as well (it's a little early to speculate further on that however).

hollyhook

2月 08 2018 | 2:39 午後

Thank you @YGREQ. Looks like i was a little enthusiastic when writing the post. Your suggestion about changing the title is good. Once I get the external to run again, i will change it accordingly.

It is a bit of a hens-egg problem. I'm aware of the fact that this is is not easy to setup, and not targeting the "typical max user" (if that is actually existing). So it was for me more an experiment, to find out if it makes sense to release stuff like that at all, and i didn't care much about proper description. With the result that it was too nerdy apparently, and not just me got discouraged.

@VICHUG, it is actually not a windows/non-windows issue, it is that the algorithms for Machine Learning can run on many kernels in parallel, so you can get real-time results if you make use of a decent graphics card. "thank you, Gamers" seems to be a running joke in the ML community because graphic cards developed for games are great and affordable. The projects are typically developed with Linux, not windows.

Also I want to mention that i think there is so much potential with externals (densepose is only one example) that i don't understand the strategy of cycling74 here. The API for externals has layers of different ages, where you understand the latest only if you dig through down to the older ones, which is way too time consuming. And I would wish for a platform supporting the developers like unity asset store, if you want to release something as a max developer you're really left on your own, no infrastructure, no platform... this way expanding the capabilities of max is so much slower that it could be if there would be more support for the community. Max is loosing ground to vvvv - just my subjective impression is that in germany it already lost, young people don't chose max anymore.

ygreq

2月 11 2018 | 9:19 午前

I found this page on github that talks about the demo and how to use it: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/demo_overview.md

And this: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#installation---demo

If anyone manages to install, please let us know how exactly. For newbies, please! Thank you!

Mathieu Chamagne

3月 09 2018 | 2:15 午後

Just discovered your work on Patreon ; thanks for sharing !
I would really really love to give a try to this external, but even after installing all the libs, and adding the bin folder to the Path environment variable, I still have the same problem (error 126...)
(The demo DO work fine on my machine)
Please, help us running this amazing piece of code !
Thanks

ygreq

3月 11 2018 | 11:15 午前

Hey Mathieu! Where did you find the demo? Because I didn't.

And can you provide us noobs (that don't know python) with a few basic steps? Thank you!

hollyhook

5月 13 2018 | 3:27 午後

I re-build the max external with openpose 1.3.0. Anybody would like to test if it works? This is difficult for me, without having access to a second PC. It still requires the openpose windows portable demo as a first step of the installation procedure (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#windows-portable-demo)

phiol

5月 14 2018 | 2:47 午前

Thanks so much , will see if I can get it running and let you know . thx again

hollyhook

5月 14 2018 | 6:43 午前

Thank you. The external is for the time being here: http://aiton.de/jit.openpose%20for%20openpose%20v1.3.0.zip
Instructions are in the demo patch.

hope it works!

ygreq

5月 19 2018 | 7:15 午後

Thank you, Hollyhook!! I will try as well.

Trying to install all the prerequisites. I cannot find Visual Basic 2015. Do you know if 2017 Community is ok? I am asking because last time I tried installing, some of my apps stopped working. Thank you!

hollyhook

5月 19 2018 | 8:04 午後

Actually for the windows portable demo, you don't need to compile anything, so no need to install Visual. Admittedly the installation instructions are a bit confusing, so I should make more clear what is needed and what not.

The windows portable demo is shortly described here: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#windows-portable-demo. See the download link to the prebuild binaries. Base for the external is the openpose 1.3.0 running on GPU. So you need the binaries openpose-1.3.0-win64-gpu-binaries.zip

For the requirements (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#requirements) the section for the NVidia GPU is important.

Unfortunately I can't say if the separate installation of CUDA & CuDNN is absolutely necessary or not. The portable demo comes with some cuda dlls, so maybe that is already sufficient. Perhaps you could just try if it just works without installing cuda?

last but not least you need to download the COCO model. The openpose portable demo comes with a little script for downloading the right files.

Please keep me updated, so I can improve the installation instructions. Thank you!

ygreq

5月 19 2018 | 10:08 午後

So I installed CMake win64-x64 msi installer, Installed Visual Basic 2017, downloaded openpose-1.3.0-win64-gpu-binariews, went to openpose-1.3.0-win64-gpu-binaries\models\getModels.bat and got the models and everything else automatically. Opened the demo via openpose-1.3.0-win64-gpu-binaries\bin\OpenPoseDemo.exe. It gave me "OpenPoseDemo.exe has stopped responding".

I installed Cuda as well. It gives me the same error. Copied the CUDNN files over the Cuda folder like adviced. Same error.

hmm. I don't know what to do right now.

ygreq

5月 19 2018 | 10:16 午後

I am trying to figure out what the following means as I just run OpenPoseDemo.exe normally. Should I add something after the .exe? I remember there was this possibility to run exe with extra commands. Is this it?

Running on Video

# Ubuntu
./build/examples/openpose/openpose.bin --video examples/media/video.avi
# With face and hands
./build/examples/openpose/openpose.bin --video examples/media/video.avi --face --hand

:: Windows - Portable Demo
bin\OpenPoseDemo.exe --video examples\media\video.avi
:: With face and hands
bin\OpenPoseDemo.exe --video examples\media\video.avi --face --hand

:: Windows - Library
build\x64\Release\OpenPoseDemo.exe --video examples\media\video.avi
:: With face and hands
build\x64\Release\OpenPoseDemo.exe --video examples\media\video.avi --face --hand

hollyhook

5月 20 2018 | 8:47 午前

Ah, good you ask. The demo is supposed to be run from the command line, like this

In windows, press Windows key + R
In the little dialog that opens, type "cmd" to start the windows command window
In the command window, type "cd openpose-folder", where openpose-folder is the directory where you extracted the openpose.zip into. On my PC, it is C:\dev\openpose\openpose-1.3.0-win64-gpu-binaries
Now you can type the entire command like they instructed, e.g. bin\OpenPoseDemo.exe --video examples\media\video.avi to run it on a little demo video that comes with the zip file. Or, if you run omit the parameter, and just run it by typing bin\OpenPoseDemo.exe, it will use the web cam of the PC.
Adding -- face will enable detection of 70 key points in the faces, --hand is for detecting keypoints at joints of the fingers, both in addition to the pose (body) detection

ygreq

5月 20 2018 | 11:56 午前

Thank you, Hollyhook. I feel like a child, but I have to say I still have problems. I managed to play the demo with a video.

I copy/pasted the folder you gave in the Max 7/Packages folder. I copied all the other dlls from the openpose-1.3.0-win64-gpu-binaries\bin folder to the Max 7\Packages\jit.openpose folder except the dll that was already there. Like you said in the patcher comments.

The object jit.openpose.pose does not work. It gives me an error <<Error 126 loading external jit.openpose.pose>> It is not recognized.

Am I still missing something? And thank you again for getting me this far.

hollyhook

5月 20 2018 | 12:24 午後

I think you are almost there. error 126 means windows cannot find or load the openpose DLLs. You have to tell windows where they are. This can be done by means of an environment variable called PATH. There are guidelines how to do this, e.g. https://stackoverflow.com/questions/44272416/how-to-add-a-folder-to-path-environment-variable-in-windows-10-with-screensho

Add the directory name where DLLs are to the path. restart max, and ... ???

ygreq

5月 20 2018 | 12:56 午後

wait a second. OpenPose found the dlls. OpenPose works by itself. I mean the demo. And Max has the dlls copied inside the Max 7 folder.

And the question is which one needs to find the dlls? Max or something else?

hollyhook

5月 20 2018 | 12:59 午後

If they are in the same folder than the executable, windows will find the dlls. However, the Max executable is somewhere else (you could in theory move all the dlls into the max exe directory and it should work also)

ygreq

5月 20 2018 | 2:06 午後

You're a genius!

Now how did you manage to put slashes inside of the openpose object? Otherwise, I will back once I figure this out. Again. It's one of those things in Max that I always forget. :)

ygreq

5月 20 2018 | 2:24 午後

I managed to open via opendialog fold..

And it works!!!! 17 fps here on a GeForce 1060 with the Tai Chi in People's Park (Shenzhen).mp4 file

Uhuuuuu!!! Thank you! Thank you! Thank you!

What next?

ygreq

5月 20 2018 | 2:47 午後

From my understanding, this only sees the skeleton, right? Without the face tracking.. Am I right?

hollyhook

5月 20 2018 | 6:53 午後

Happy it works!
yeah, whats next.... First of all, updating the instructions, your questions give me the direction where improvement is needed, so thank you for asking. The face tracking is another external, it is already running here. I decided to split face detection and pose detection into two externals, in order to have best performance for each. Face is tracking 70 keypoints in a face, right now the external runs with 10fps (at least better than the demo). Will have to see if an how to move on, openpose has some more interesting things to offer: hands, 3D reconstruction with multiple cameras, and tracking of persons over frames...

Mathieu Chamagne

5月 21 2018 | 2:23 午後

totally amazing !
it works here @22 fps with a live video feed, on a GTX1070 :-)
thousands thanks !!!

looking forward all the "whats next" !!

ygreq

5月 22 2018 | 11:18 午後

Wait, I did not understand about the face detection? Is it integrated into the max patch?

phiol

5月 23 2018 | 12:17 午前

malade! thx hollyhook! you’re a beast

hollyhook

5月 23 2018 | 7:44 午後

Thanks guys, really appreciate your feedback.

@Ygreq, face tracking is not included.

Btw: https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html is amazing. With the new tensorflow interface in javascript, access to ML models will be much easier in future, so much will come.

MAX.XAM

5月 23 2018 | 9:58 午後

Thanks!

pandigital

5月 24 2018 | 11:30 午前

@hollyhook is there a tensorflow object available for max yet?

hollyhook

5月 25 2018 | 8:26 午前

There is no max object, but as max is good to connect with other programing languages, one still can use tensorflow. In my little side project deeploops (happy to drop the link here) I use tensorflow via the java API, and a mxj. Recently tensorflow.js was released, so this would probably the better way.

Openpose uses custom build caffe instead of tensorflow, and it has no java or js API, therefore the wrapper needs to be an external.

matmat

6月 03 2018 | 6:40 午後

Hey Fantastic !
It works here too :-)
Thank you very much HOLLYHOOK !

hollyhook

6月 03 2018 | 7:14 午後

thanks, glad to hear that!

I found a bug 🤦 If you swap the red and blue channels like in the picture, the detection quality improves quite a bit.

ygreq

6月 06 2018 | 8:58 午前

Great! Thank you, Hollyhook!

hollyhook

7月 29 2018 | 11:14 午前

update: external for face detection

it requires the same openpose dlls, means if you have jit.openpose.pose running, this patch works out-of-the-box.

ygreq

7月 30 2018 | 8:26 午前

Will try out soon!
Thank you so much, Hollyhook!

ygreq

11月 09 2018 | 7:56 午後

Hi Hollyhook!

It seems I cannot use the patch properly because the IDing keeps on switching. I mean user 1 and user 2 switch between themselves. So for example I would not be able to put a black circle on user 1's face as that circle would switch to user 2's face and back. A bit randomly.

BTW, it seems the system does the IDing as such. If one person enters the scene, it is tagged as user 1. If another person enters the scene, it is tagged as user 2 and so on. But if user 1 goes out of the frame, user 2 becomes user 1.

Any ideas on how to solve this, dear Hollyhook? Anyone else?
Thank you so much!

hollyhook

11月 09 2018 | 9:03 午後

Hi yreg, there is no tracking in openpose v1.3, so IDs can jump between persons anytime. In the latest openpose version there is some experimental single-person tracking, not sure if that would be a help if you wish to detect multiple users... I once created a simple tracking module in js, which works like 90% of the time. I can provide it, if you wish, but while writing the module, i understood that tracking in 2D is really not a simple task

For the face detection external, I implemented a filter which is supposed to output the largest face only, determined by comparing the length of the faces in y direction, like distance between eyebrow and chin.

ygreq

11月 23 2018 | 2:25 午後

Hi Hollyhook! Thank you for your reply. I wasn't able to reply on time because someone spammed this thread and the commenting was blocked.
https://cycling74.com/forums/i-am-not-able-to-comment-on-this-thread-does-this-happen-to-you-as-well

I talked to the admins to open it up again. Anyway..

That is unfortunate. Like I was saying, that limits the possibilities of OpenPose a lot. May I ask to see the tracking app you mentioned? Thank you!!

BTW, is it hard to implement the 1 person detection option?

Thank you very much!

hollyhook

11月 25 2018 | 5:06 午後

Attaching the simple js based person ID tracking. It simply calculates the distance between newly received skeletons to the old (saved) ones, and assigns the new budy to the nearest person ID accordingly - as I said, a 90% solution.

The single person tracking in openpose is not released yet, so there is no demo standalone yet, making installation more difficult than it already is. A quick search didn't bring up any release date for openpose 1.5, so right now I'd rather wait. Anybody checked if PoseNet is capable of person tracking?

stickmen_with_person_tracking.zip

application/x-zip-compressed 4.19 KB

WU JIAJIUN

12月 15 2018 | 6:17 午前

HI HOLLYHOOK,I have some problem with ID describe too. I use Message to see the array in jit.pose.pose object. However, no matter how many people I detect , I only see the 56 array length in it.( see the pic). In my thought it should be more data when I detect more than one person?

I have tried the post you share but I don't familiar with javascript. It seems have two JS in it but I don't know how to replace it to the patch you share in patreon.

And I read the stickmen.js , It says include "logging.js" and "body.js" , but I don't know how to include and I cannot find the "logging.js" either.

Could you tell me some solution? thank you!

hollyhook

12月 15 2018 | 7:36 午後

Hello WU JIAJIUN
thanks for your question. The left outlet of jit.openpose.pose sends 56 numbers for each detected person. If there are several person, it sends them very quickly one after another through the same outlet. The fist number is the ID of the person. If you wish to separate the data for each person, you could use the route or routepass object, like shown in this picture:

I hope this already clarifies the issue. If you wish to try out the patch witch person tracking js, let me know and I can upload a variant of the patch with the person ID tracking integrated.

ygreq

12月 16 2018 | 12:11 午後

Please do that, Hollyhook. I couldn't use what you already put here unfortunately. This: STICKMEN_WITH_PERSON_TRACKING.ZIP

WU JIAJIUN

12月 16 2018 | 12:52 午後

Thank you HOLLYHOOK! It solve my problem. Then I can make visual through using max/msp object!
Still, if you share the patch with tracking, I would love to try !

ygreq

4月 18 2019 | 7:13 午前

Look what I found. It seems this solution is much much faster: https://www.learnopencv.com/pose-detection-comparison-wrnchai-vs-openpose/

I wonder if it also solves the big problem we discussed earlier about people tracking.

ygreq

4月 25 2019 | 8:19 午前

I was wondering how cv.jit.face solves the problem with tracking. Any ideas?

ygreq

4月 25 2019 | 8:23 午前

Talking to myself here, but I think I have a clue on how to solve this tracking problem. I use openPose and I draw only the nose point. Then I use cv.jit to track those specific points. And considering cv.jit is more stable in this regard, it might just work.

Your STICKMEN_WITH_PERSON_TRACKING uses the same principles, hollyhook? I am not a js expert so I didn't understand it.

ygreq

4月 25 2019 | 10:58 午前

I made some modifications in preparation for cv.jit blob detection. I have no experience in using it. If someone is willing to help I would really appreciate it.

Use camera or the same video hollyhook used.

Max Patch

Copy patch and select New From Clipboard in Max.

ygreq

5月 02 2019 | 8:50 午前

I made the blob detection. The next step would be to use the id value of the blob sort algorithm instead of the one provided by Open Pose. I think I am on the right track here. Any improvements are welcome!

Max Patch

Copy patch and select New From Clipboard in Max.

Augustine Bannatyne - Leudar

6月 01 2019 | 9:19 午後

YGREQ - Just downloading etc now. Did you get it working in the end ? Also could it give me xyz coordinates of people in a space - Im not really bothered about their pose just yet or facial stuff - just wher ethey are standing in a room, left right, near far, up down (basically x,y,z)

ygreq

6月 01 2019 | 9:29 午後

It gives you the coordinates in a 2d space (x and y only) of each body part, but the id of each individual keeps on switching. What do you need it for?

Augustine Bannatyne - Leudar

6月 01 2019 | 9:37 午後

all sorts of things. I dont really mind the ID swapping , yet, but say when somebody walks away I notice the skeleton tracking gets smaller - could that not sort of be converted into a z (distance) coordinate) - or could you daisy chain webcams together ?

ygreq

6月 01 2019 | 9:43 午後

What happens if you have a short person and a tall person at the same time?
I dont get the daisy chain part

ygreq

6月 01 2019 | 9:55 午後

2 cam combo will not work. It will not know which of the people is standing in the back. Or if possible, there is a lot of triangulation to be done. And still considering you have different people of different sizes...

Augustine Bannatyne - Leudar

6月 01 2019 | 11:53 午後

Hmmm I get "this app cannot run on your pc" when I paste

bin\OpenPoseDemo.exe --video examples\media\video.avi

into cmd in the right directory. Any ideas ?

ygreq

6月 03 2019 | 7:27 午前

No idea. Just follow the steps or the comments above. I dont have my win pc with me and I dont consider myself to be any expert on this. Maybe someone else can help

hollyhook

6月 06 2019 | 11:33 午前

@ygreq I calculated for each person the distance of the joints between 2 pictures, and if it was below a certain threshold i took it as the same person. The idea with taking only the nose point is cool, maybe if you additionally take the ears to get it more robust against rotation ...

hollyhook

6月 06 2019 | 11:46 午前

@ Augustine you could try a very rough estimation of the distance by taking the spine length and triangulation. if you have only one person, and if you can assume that the person is in a straight pose, if should give some results. For multiple persons, you will quickly run into that problem that there is no tracking so you don't know who is who from one picture to the next. Openpose 1.4 can reconstruct the 3rd dimension if you use multiple cameras. (Never tried it myself)

Concerning the error message, it sounds like something is gravely wrong. Did you check the preconditions? win64, GPU, ...

albert

7月 27 2019 | 6:09 午後

Hello !
I try to run more than one openpose object on the same computer with different cameras.
For the moment I have 2 cameras with two openpose objects working perfectly on the same time.
But when I connect another camera on anther openpose object the patch crashing...
Somebody know if there is a limitation of the objet, or with more efficient computer can it work?
Thanks in advance!
albert

ygreq

7月 27 2019 | 10:54 午後

A better option would be to put together the images and then compute them with a single instance of open pose. I didn't try it, but it might work. And a lot better.

ygreq

7月 27 2019 | 10:55 午後

Btw, I managed to stabilize the user id using blob detection ;)

hollyhook

7月 28 2019 | 7:43 午前

@Albert, I never tried running several instances of openpose in parallel. I would expect it is running out of (graphics?) memory. What is your use case? Did you try to use the same object for all 3 cameras? It works on a image base, has no "memory" from previous images. If the image format is the same, it should create the same result.

Best, Constanze

albert

7月 28 2019 | 12:25 午後

Hi, thanks for the advices!
@YGREQ yes it works ! I mix the 4 webcams into single matrix and then I connect it to openpose object and I have 4 differents user ID.
@HOLLYHOOK my objectif is to know if somebody look a camera and which camera.
thanks! albert

ygreq

7月 28 2019 | 12:45 午後

Great to hear, Albert!

Augustine Bannatyne - Leudar

7月 28 2019 | 5:41 午後

@ HOLLYHOOK
Tx - maybe its not the most straightforward was of detecting peoples position in an area. WIth regards to install, I am really busy and getting my heads around touch designer at the moment. I found the install documentation for openpose really longwinded and time consuming and I've probably missed something important. A simple step 1, step 2, step 3 etc installation guide for PC would be really useful - if you know of one please link me !!