openpose for Max --- Skeleton Tracking based on Artificial Intelligence on 2D images (no Kinect)


    Sep 02 2017 | 4:08 pm
    Max external for detecting persons in 2D images - real-time capable, using the awesome openpose library.
    Windows x64 only, needs a nvidia graphic card. you can get the external jit.openpose.pose and a simple jsui for visualization here, for free.

    • Sep 02 2017 | 5:22 pm
      incredible.
    • Sep 03 2017 | 11:39 am
      awesome
    • Sep 03 2017 | 4:27 pm
      thanks a lot for sharing HOLLYHOOK
    • Nov 01 2017 | 8:10 pm
      Thanks for sharing ! :) Im really stoked to try it. I would have one question , what do i have to add specificly to the Path environment variable to get the extrenal to access the DLLs ?
      I get error 126 when trying to load the external.
      Thanks Again
    • Feb 04 2018 | 5:21 pm
      How did you manage to use it? Is anyone able to help with the installment of the dependecies for running openPose? Thank you!! I tried following what it says on the github, but it is too much for a noob like me
    • Feb 04 2018 | 8:17 pm
      never got it to work , the external is broken
    • Feb 05 2018 | 9:31 am
      Look amazing but never got it to work too...
    • Feb 05 2018 | 9:46 am
      Same here, I would love to test but, I get error 126 ...
    • Feb 05 2018 | 10:16 am
      My guess is that it is not broken. You just have to install all the dependencies. From what I figured from the installation page (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md), it says you need to install openCV and Caffe (I don't know why it says Ubuntu only. My guess is that we need to install these as well). Then install CMake GUI and Visual Studio. Andonly after that, you build OpenPose and test it. And if everything works, my guess is that the external will work as well.
      But that is easier said then done. So I ask again. :) Can someone help us noobs with a for-noobs tutorial on how to install all that? Hollyhook? Please!! At least can you confirm what I wrote above? Thank you so much!! ygr
      PS: Did you guys manage to install all the dependencies?
    • Feb 05 2018 | 7:28 pm
      This sounds interesting. I'll give it a try and if I can get it to work, I'll make up a tutorial for how to get it working that will be not so developer-centric ;)
    • Feb 05 2018 | 7:58 pm
      The openpose developer had a openpose demo, which came with pre-build libraries for openpose and some 3rd party DLLs. The idea was, if you are able to run the demo (with cuda etc.), you can take all the dlls it comes with, put them on the PC somewhere so that windows can find them (in PATH) and the external runs. I can't find the openpose demo anymore, and they also have improved the build system. So i should re-build the external again, on the base of the new openpose... I'm quite busy right now, but will do it as soon as i find time. To be honest, I didn't expect that anyone uses it, because the feedback was kind of thin.. so didn't care about updating.
      Btw, if anybody would like to try to build it, I'm happy to give the sources of the external away. Btw2, Seen this? https://twitter.com/NataliaNeverova/status/959530358613176320 "code coming soon" \o/ could also be candidate for wrapping into a max external, most likely with the same limitations (PC only, Nvidia, DLL mess)
    • Feb 06 2018 | 1:14 am
      Thanks for that link, it looks much more interesting, surpassing Kinect instead of catching up to it.
      Hopefully it works in realtime, will check the paper.
    • Feb 07 2018 | 8:11 am
      Thank you, Hollyhook, for you answer. My first observation regarding you mentioning that there wasn't such great interest in this object is the fact that people don't know about it existing. OpenPose, I mean. BTW, can you modify the title to something like: "OpenPose: Kinect Improved Alternative to Skeleton Tracking based on Artificial Intelligence". And I am adding a tag, Machine Learning. :) This way many maxers will find out about it for sure and interest will grow.
      Now regarding what you said, maybe you can link a download to the build in case someone wants and knows how to fix it. But hopefully you have a bit of time to look into it and fix it. That would be really great. I can barely wait to try it!
      I will start looking for that demo now. :) My guess is that maybe if I use the same code as you did, with the same demo, it might work, right? I might have to ask you other things about this later on. :)
      And your link with the people masking with the DensePose is genius! Hope they manage to put the code out and someone manages to import it in Max as well. :)) Like I said, the only programming language I know is Max. Unfortunately! I already have great ideas for some projects involving this!!! Thank you so much! And again, if anyone manages to make this work, please help the whole community and write all the steps you took in detail. Thank you so much!
    • Feb 07 2018 | 9:18 am
      waw that's pretty rad ! hopefully the densepose framework can be extended to non windows machines as well (it's a little early to speculate further on that however).
    • Feb 08 2018 | 2:39 pm
      Thank you @YGREQ. Looks like i was a little enthusiastic when writing the post. Your suggestion about changing the title is good. Once I get the external to run again, i will change it accordingly. It is a bit of a hens-egg problem. I'm aware of the fact that this is is not easy to setup, and not targeting the "typical max user" (if that is actually existing). So it was for me more an experiment, to find out if it makes sense to release stuff like that at all, and i didn't care much about proper description. With the result that it was too nerdy apparently, and not just me got discouraged.
      @VICHUG, it is actually not a windows/non-windows issue, it is that the algorithms for Machine Learning can run on many kernels in parallel, so you can get real-time results if you make use of a decent graphics card. "thank you, Gamers" seems to be a running joke in the ML community because graphic cards developed for games are great and affordable. The projects are typically developed with Linux, not windows.
      Also I want to mention that i think there is so much potential with externals (densepose is only one example) that i don't understand the strategy of cycling74 here. The API for externals has layers of different ages, where you understand the latest only if you dig through down to the older ones, which is way too time consuming. And I would wish for a platform supporting the developers like unity asset store, if you want to release something as a max developer you're really left on your own, no infrastructure, no platform... this way expanding the capabilities of max is so much slower that it could be if there would be more support for the community. Max is loosing ground to vvvv - just my subjective impression is that in germany it already lost, young people don't chose max anymore.
    • Feb 11 2018 | 9:19 am
      I found this page on github that talks about the demo and how to use it: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/demo_overview.md
      If anyone manages to install, please let us know how exactly. For newbies, please! Thank you!
    • Mar 09 2018 | 2:15 pm
      Just discovered your work on Patreon ; thanks for sharing ! I would really really love to give a try to this external, but even after installing all the libs, and adding the bin folder to the Path environment variable, I still have the same problem (error 126...) (The demo DO work fine on my machine) Please, help us running this amazing piece of code ! Thanks
    • Mar 11 2018 | 11:15 am
      Hey Mathieu! Where did you find the demo? Because I didn't. And can you provide us noobs (that don't know python) with a few basic steps? Thank you!
    • May 13 2018 | 3:27 pm
      I re-build the max external with openpose 1.3.0. Anybody would like to test if it works? This is difficult for me, without having access to a second PC. It still requires the openpose windows portable demo as a first step of the installation procedure (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#windows-portable-demo)
    • May 14 2018 | 2:47 am
      Thanks so much , will see if I can get it running and let you know . thx again
    • May 14 2018 | 6:43 am
      Thank you. The external is for the time being here: http://aiton.de/jit.openpose%20for%20openpose%20v1.3.0.zip Instructions are in the demo patch.
      hope it works!
    • May 19 2018 | 7:15 pm
      Thank you, Hollyhook!! I will try as well. Trying to install all the prerequisites. I cannot find Visual Basic 2015. Do you know if 2017 Community is ok? I am asking because last time I tried installing, some of my apps stopped working. Thank you!
    • May 19 2018 | 8:04 pm
      Actually for the windows portable demo, you don't need to compile anything, so no need to install Visual. Admittedly the installation instructions are a bit confusing, so I should make more clear what is needed and what not.
      The windows portable demo is shortly described here: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#windows-portable-demo. See the download link to the prebuild binaries. Base for the external is the openpose 1.3.0 running on GPU. So you need the binaries openpose-1.3.0-win64-gpu-binaries.zip For the requirements (https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md#requirements) the section for the NVidia GPU is important.
      Unfortunately I can't say if the separate installation of CUDA & CuDNN is absolutely necessary or not. The portable demo comes with some cuda dlls, so maybe that is already sufficient. Perhaps you could just try if it just works without installing cuda?
      last but not least you need to download the COCO model. The openpose portable demo comes with a little script for downloading the right files.
      Please keep me updated, so I can improve the installation instructions. Thank you!
    • May 19 2018 | 10:08 pm
      So I installed CMake win64-x64 msi installer, Installed Visual Basic 2017, downloaded openpose-1.3.0-win64-gpu-binariews, went to openpose-1.3.0-win64-gpu-binaries\models\getModels.bat and got the models and everything else automatically. Opened the demo via openpose-1.3.0-win64-gpu-binaries\bin\OpenPoseDemo.exe. It gave me "OpenPoseDemo.exe has stopped responding".
      I installed Cuda as well. It gives me the same error. Copied the CUDNN files over the Cuda folder like adviced. Same error.
      hmm. I don't know what to do right now.
    • May 19 2018 | 10:16 pm
      I am trying to figure out what the following means as I just run OpenPoseDemo.exe normally. Should I add something after the .exe? I remember there was this possibility to run exe with extra commands. Is this it?

      Running on Video

      # Ubuntu
      ./build/examples/openpose/openpose.bin --video examples/media/video.avi
      # With face and hands
      ./build/examples/openpose/openpose.bin --video examples/media/video.avi --face --hand
      
      :: Windows - Portable Demo
      bin\OpenPoseDemo.exe --video examples\media\video.avi
      :: With face and hands
      bin\OpenPoseDemo.exe --video examples\media\video.avi --face --hand
      
      :: Windows - Library
      build\x64\Release\OpenPoseDemo.exe --video examples\media\video.avi
      :: With face and hands
      build\x64\Release\OpenPoseDemo.exe --video examples\media\video.avi --face --hand
    • May 20 2018 | 8:47 am
      Ah, good you ask. The demo is supposed to be run from the command line, like this
      1. In windows, press Windows key + R
      2. In the little dialog that opens, type "cmd" to start the windows command window
      3. In the command window, type "cd openpose-folder", where openpose-folder is the directory where you extracted the openpose.zip into. On my PC, it is C:\dev\openpose\openpose-1.3.0-win64-gpu-binaries
      4. Now you can type the entire command like they instructed, e.g. bin\OpenPoseDemo.exe --video examples\media\video.avi to run it on a little demo video that comes with the zip file. Or, if you run omit the parameter, and just run it by typing bin\OpenPoseDemo.exe, it will use the web cam of the PC.
      5. Adding -- face will enable detection of 70 key points in the faces, --hand is for detecting keypoints at joints of the fingers, both in addition to the pose (body) detection
    • May 20 2018 | 11:56 am
      Thank you, Hollyhook. I feel like a child, but I have to say I still have problems. I managed to play the demo with a video.
      I copy/pasted the folder you gave in the Max 7/Packages folder. I copied all the other dlls from the openpose-1.3.0-win64-gpu-binaries\bin folder to the Max 7\Packages\jit.openpose folder except the dll that was already there. Like you said in the patcher comments.
      The object jit.openpose.pose does not work. It gives me an error <<Error 126 loading external jit.openpose.pose>> It is not recognized.
      Am I still missing something? And thank you again for getting me this far.
    • May 20 2018 | 12:24 pm
      I think you are almost there. error 126 means windows cannot find or load the openpose DLLs. You have to tell windows where they are. This can be done by means of an environment variable called PATH. There are guidelines how to do this, e.g. https://stackoverflow.com/questions/44272416/how-to-add-a-folder-to-path-environment-variable-in-windows-10-with-screensho Add the directory name where DLLs are to the path. restart max, and ... ???
    • May 20 2018 | 12:56 pm
      wait a second. OpenPose found the dlls. OpenPose works by itself. I mean the demo. And Max has the dlls copied inside the Max 7 folder.
      And the question is which one needs to find the dlls? Max or something else?
    • May 20 2018 | 12:59 pm
      If they are in the same folder than the executable, windows will find the dlls. However, the Max executable is somewhere else (you could in theory move all the dlls into the max exe directory and it should work also)
    • May 20 2018 | 2:06 pm
      You're a genius!
      Now how did you manage to put slashes inside of the openpose object? Otherwise, I will back once I figure this out. Again. It's one of those things in Max that I always forget. :)
    • May 20 2018 | 2:24 pm
      I managed to open via opendialog fold..
      And it works!!!! 17 fps here on a GeForce 1060 with the Tai Chi in People's Park (Shenzhen).mp4 file
      Uhuuuuu!!! Thank you! Thank you! Thank you!
      What next?
    • May 20 2018 | 2:47 pm
      From my understanding, this only sees the skeleton, right? Without the face tracking.. Am I right?
    • May 20 2018 | 6:53 pm
      Happy it works! yeah, whats next.... First of all, updating the instructions, your questions give me the direction where improvement is needed, so thank you for asking. The face tracking is another external, it is already running here. I decided to split face detection and pose detection into two externals, in order to have best performance for each. Face is tracking 70 keypoints in a face, right now the external runs with 10fps (at least better than the demo). Will have to see if an how to move on, openpose has some more interesting things to offer: hands, 3D reconstruction with multiple cameras, and tracking of persons over frames...
    • May 21 2018 | 2:23 pm
      totally amazing ! it works here @22 fps with a live video feed, on a GTX1070 :-) thousands thanks !!!
      looking forward all the "whats next" !!
    • May 22 2018 | 11:18 pm
      Wait, I did not understand about the face detection? Is it integrated into the max patch?
    • May 23 2018 | 12:17 am
      malade! thx hollyhook! you’re a beast
    • May 23 2018 | 7:44 pm
      Thanks guys, really appreciate your feedback. @Ygreq, face tracking is not included. Btw: https://storage.googleapis.com/tfjs-models/demos/posenet/camera.html is amazing. With the new tensorflow interface in javascript, access to ML models will be much easier in future, so much will come.
    • May 23 2018 | 9:58 pm
      Thanks!
    • May 24 2018 | 11:30 am
      @hollyhook is there a tensorflow object available for max yet?
    • May 25 2018 | 8:26 am
      There is no max object, but as max is good to connect with other programing languages, one still can use tensorflow. In my little side project deeploops (happy to drop the link here) I use tensorflow via the java API, and a mxj. Recently tensorflow.js was released, so this would probably the better way.
      Openpose uses custom build caffe instead of tensorflow, and it has no java or js API, therefore the wrapper needs to be an external.
    • Jun 03 2018 | 6:40 pm
      Hey Fantastic ! It works here too :-) Thank you very much HOLLYHOOK !
    • Jun 03 2018 | 7:14 pm
      thanks, glad to hear that!
      I found a bug 🤦 If you swap the red and blue channels like in the picture, the detection quality improves quite a bit.
    • Jun 06 2018 | 8:58 am
      Great! Thank you, Hollyhook!
    • Jul 29 2018 | 11:14 am
      it requires the same openpose dlls, means if you have jit.openpose.pose running, this patch works out-of-the-box.
    • Jul 30 2018 | 8:26 am
      :O
      Will try out soon! Thank you so much, Hollyhook!
    • Nov 09 2018 | 7:56 pm
      Hi Hollyhook! It seems I cannot use the patch properly because the IDing keeps on switching. I mean user 1 and user 2 switch between themselves. So for example I would not be able to put a black circle on user 1's face as that circle would switch to user 2's face and back. A bit randomly.
      BTW, it seems the system does the IDing as such. If one person enters the scene, it is tagged as user 1. If another person enters the scene, it is tagged as user 2 and so on. But if user 1 goes out of the frame, user 2 becomes user 1.
      Any ideas on how to solve this, dear Hollyhook? Anyone else? Thank you so much!
    • Nov 09 2018 | 9:03 pm
      Hi yreg, there is no tracking in openpose v1.3, so IDs can jump between persons anytime. In the latest openpose version there is some experimental single-person tracking, not sure if that would be a help if you wish to detect multiple users... I once created a simple tracking module in js, which works like 90% of the time. I can provide it, if you wish, but while writing the module, i understood that tracking in 2D is really not a simple task For the face detection external, I implemented a filter which is supposed to output the largest face only, determined by comparing the length of the faces in y direction, like distance between eyebrow and chin.
    • Nov 23 2018 | 2:25 pm
      Hi Hollyhook! Thank you for your reply. I wasn't able to reply on time because someone spammed this thread and the commenting was blocked. https://cycling74.com/forums/i-am-not-able-to-comment-on-this-thread-does-this-happen-to-you-as-well
      I talked to the admins to open it up again. Anyway..
      That is unfortunate. Like I was saying, that limits the possibilities of OpenPose a lot. May I ask to see the tracking app you mentioned? Thank you!! BTW, is it hard to implement the 1 person detection option?
      Thank you very much!
    • Nov 25 2018 | 5:06 pm
      Attaching the simple js based person ID tracking. It simply calculates the distance between newly received skeletons to the old (saved) ones, and assigns the new budy to the nearest person ID accordingly - as I said, a 90% solution.
      The single person tracking in openpose is not released yet, so there is no demo standalone yet, making installation more difficult than it already is. A quick search didn't bring up any release date for openpose 1.5, so right now I'd rather wait. Anybody checked if PoseNet is capable of person tracking?
    • Dec 15 2018 | 6:17 am
      HI HOLLYHOOK,I have some problem with ID describe too. I use Message to see the array in jit.pose.pose object. However, no matter how many people I detect , I only see the 56 array length in it.( see the pic). In my thought it should be more data when I detect more than one person? I have tried the post you share but I don't familiar with javascript. It seems have two JS in it but I don't know how to replace it to the patch you share in patreon.
      And I read the stickmen.js , It says include "logging.js" and "body.js" , but I don't know how to include and I cannot find the "logging.js" either.
      Could you tell me some solution? thank you!
    • Dec 15 2018 | 7:36 pm
      Hello WU JIAJIUN thanks for your question. The left outlet of jit.openpose.pose sends 56 numbers for each detected person. If there are several person, it sends them very quickly one after another through the same outlet. The fist number is the ID of the person. If you wish to separate the data for each person, you could use the route or routepass object, like shown in this picture:
      I hope this already clarifies the issue. If you wish to try out the patch witch person tracking js, let me know and I can upload a variant of the patch with the person ID tracking integrated.
    • Dec 16 2018 | 12:11 pm
      Please do that, Hollyhook. I couldn't use what you already put here unfortunately. This: STICKMEN_WITH_PERSON_TRACKING.ZIP
    • Dec 16 2018 | 12:52 pm
      Thank you HOLLYHOOK! It solve my problem. Then I can make visual through using max/msp object! Still, if you share the patch with tracking, I would love to try !
    • Apr 18 2019 | 7:13 am
      Look what I found. It seems this solution is much much faster: https://www.learnopencv.com/pose-detection-comparison-wrnchai-vs-openpose/
      I wonder if it also solves the big problem we discussed earlier about people tracking.
    • Apr 25 2019 | 8:19 am
      I was wondering how cv.jit.face solves the problem with tracking. Any ideas?
    • Apr 25 2019 | 8:23 am
      Talking to myself here, but I think I have a clue on how to solve this tracking problem. I use openPose and I draw only the nose point. Then I use cv.jit to track those specific points. And considering cv.jit is more stable in this regard, it might just work. Your STICKMEN_WITH_PERSON_TRACKING uses the same principles, hollyhook? I am not a js expert so I didn't understand it.
    • Apr 25 2019 | 10:58 am
      I made some modifications in preparation for cv.jit blob detection. I have no experience in using it. If someone is willing to help I would really appreciate it. Use camera or the same video hollyhook used.
    • May 02 2019 | 8:50 am
      I made the blob detection. The next step would be to use the id value of the blob sort algorithm instead of the one provided by Open Pose. I think I am on the right track here. Any improvements are welcome!
    • Jun 01 2019 | 9:19 pm
      YGREQ - Just downloading etc now. Did you get it working in the end ? Also could it give me xyz coordinates of people in a space - Im not really bothered about their pose just yet or facial stuff - just wher ethey are standing in a room, left right, near far, up down (basically x,y,z)
    • Jun 01 2019 | 9:29 pm
      It gives you the coordinates in a 2d space (x and y only) of each body part, but the id of each individual keeps on switching. What do you need it for?
    • Jun 01 2019 | 9:37 pm
      all sorts of things. I dont really mind the ID swapping , yet, but say when somebody walks away I notice the skeleton tracking gets smaller - could that not sort of be converted into a z (distance) coordinate) - or could you daisy chain webcams together ?
    • Jun 01 2019 | 9:43 pm
      What happens if you have a short person and a tall person at the same time? I dont get the daisy chain part
    • Jun 01 2019 | 9:55 pm
      2 cam combo will not work. It will not know which of the people is standing in the back. Or if possible, there is a lot of triangulation to be done. And still considering you have different people of different sizes...
    • Jun 01 2019 | 11:53 pm
      Hmmm I get "this app cannot run on your pc" when I paste
      bin\OpenPoseDemo.exe --video examples\media\video.avi
      
      into cmd in the right directory. Any ideas ?
    • Jun 03 2019 | 7:27 am
      No idea. Just follow the steps or the comments above. I dont have my win pc with me and I dont consider myself to be any expert on this. Maybe someone else can help
    • Jun 06 2019 | 11:33 am
      @ygreq I calculated for each person the distance of the joints between 2 pictures, and if it was below a certain threshold i took it as the same person. The idea with taking only the nose point is cool, maybe if you additionally take the ears to get it more robust against rotation ...
    • Jun 06 2019 | 11:46 am
      @ Augustine you could try a very rough estimation of the distance by taking the spine length and triangulation. if you have only one person, and if you can assume that the person is in a straight pose, if should give some results. For multiple persons, you will quickly run into that problem that there is no tracking so you don't know who is who from one picture to the next. Openpose 1.4 can reconstruct the 3rd dimension if you use multiple cameras. (Never tried it myself)
      Concerning the error message, it sounds like something is gravely wrong. Did you check the preconditions? win64, GPU, ...