n4m - FaceMesh + HandPose (Google MediaPipe)


    May 20 2020 | 1:52 pm
    Hi Folks,
    I've made two new contributions to the Node For MaxMSP Community Library wrapping Google's MediaPipe (in a similar fashion to PoseNet using Electron) for these two:
    Keen to see how they work on different operating systems, pull requests and bug logging via the issue trackers in the repo welcome :)
    Stars also welcome!

    • May 24 2020 | 4:33 am
      super. between 5 to 15 frames on OSX 10.15.3 Max 8.1.3 here ya go: ⭐✰⋆🌟✪🔯✨
    • May 24 2020 | 1:23 pm
      Wasn't expecting this kind of a negative/unhelpful response - but I'm actually getting ~20fps on Mojave on my mbp for both scripts. I thought this might be useful to share with the Max community. Hopefully some people here will find it useful.
      If you run Facemesh on the cpu instead of webassembly you'll likely get 5fps, so stick with the WASM option.
    • May 25 2020 | 9:05 am
      Thanks a lot, really nice!
      Handpose is also around 20 fps. My framerate with facemesh is not as high as yours it's around 5 till 10 fps maximum (wasm)
      Max 8.1.3 Mojave 10.14.6 MBP 2,9GHz i9 - intel UHD graphics 630 1536MB
    • May 26 2020 | 1:28 am
      Amazing - ~40fps with webgl MBP 13" +egpu.
    • May 27 2020 | 4:48 am
      Awesome, thanks for the reports folks!
      My stats are almost identical to Aartcore's, except my GPU
      Max 8.1.3 Mojave 10.14.6 MBP 2,9GHz i9 - Radeon Pro Vega 20 4 GB
    • Jun 21 2020 | 10:28 pm
      Hi this is really great, I can't believe it took me this long to try it out. Thanks for sharing. Perfect for just tracking my hand instead of trying to do that with posenet. I am seeing similar numbers, with my mbp i7, radeon pro 560. hand pose is around 20 fps / facemesh was getting 15fps
    • Jun 21 2020 | 11:22 pm
      quick question, has anyone tried lowering the resolution for better performance? I just changed the dims in the camera.js file to 320x240 from 600x500 and i think im seeing better performance.
    • Jun 22 2020 | 2:45 pm
      Hi. This looks amazing. However, after downloading the dependencies, when Electron opens I just get a blank screen, and it doesn't launch/open my webcam. Is there something I might be doing wrong?
    • Jun 22 2020 | 4:12 pm
      Hi folks - I've now wrapped both of these to two Max For Live Devices which are really fun to use. I'll be adding these to the repos early next week for everyone. I've also added a whole bunch of improvements to the scripting (offscreen rendering so you don't need the window to be visible to get the data, full model controls for handpose, replicated FaceOSC's gestures for Facemesh, so you can use "mouth width" and "eye height" etc) and more which I'll show you when I commit the updates. :)
      I've quick question, has anyone tried lowering the resolution for better performance? I just changed the dims in the camera.js file to 320x240 from 600x500 and i think im seeing better performance.
      yep! that's a great idea. I've actually added params for window size into the next update that you can control from max when you spawn the window - I might default them to 320x240 for better performance for most people
      Hi. This looks amazing. However, after downloading the dependencies, when Electron opens I just get a blank screen, and it doesn't launch/open my webcam. Is there something I might be doing wrong?
      Hi, thanks for your interest in the project! Could be a few things, firstly - check for a .log file inside the folder to see if everything downloaded correctly from npm. If not, it'll be inside the log. Second, could be security on your OS (catalina can sometimes be a pain with this), haven't tried on windows but heard success from windows users.
      You can try toggling your video device from the GUI controls also to see if that works. In the menu bar for the electron window there is an option to "Enable Debug Console" - if you open that up and see an error message post it here or in the issues tab for the repo on github :)
      You'll also need to keep your maxpatch inside the same folder with all the other js files and node_modules. saving a patch into a different folder often breaks things with node.
    • Jun 22 2020 | 5:28 pm
      @LYSDEXIC Thanks for your reply. Hmm, it's strange. I'm certain it has all downloaded - everything opens and looks right in Visual Studio. Interestingly - I can't see a GUI to toggle the camera with. I also do get an error, but it doesn't look like a fatal one: Electron Security Warning (Insecure Content-Security-Policy). Electron is also managing to connect to Max. Security-wise I'm on 10.12, and generally don't have problems granting access to things. Do you think there could be anything else up? Thanks.
    • Jun 22 2020 | 9:05 pm
      Wow. thanks LYSDEXIC. Can't wait to see the updates
    • Oct 01 2020 | 7:32 pm
      Hi Lysdexic, I am having issues trying to get my webcam running in handpose and with jit.grab. On my mbp w the built in cam this works, and with my external cam. But I am now running windows10 and only works one at a time. Do you know of a solution?
    • Oct 20 2020 | 7:23 am
      Great work, thanks for sharing! I would like to draw the mesh in Max instead of the electron window. Is it possible?
    • Oct 20 2020 | 10:48 am
      Great!
    • Nov 27 2020 | 11:41 am
      Really amazing workflow @LYSDEXIC ! ⚝⚝⚝⚝⚝ Thanks so much! And to @DANTE LENTZ for launching me here :) Handpose framerate on Win 10 > from 20 to 28fps Notes: on fast hand movements the tracker in Electron looks less efficient then the example run by mediapipe... https://viz.mediapipe.dev/demo/hand_tracking Is there something wrong I did using your dev? Tnx a lot!
    • Feb 26 2021 | 11:08 am
      Hello! I'm trying to open the patch (thanks for sharing) but every time it crashes after few seconds... So i didn't see a lot. I wanted to ask if there was a way to adjust this. Anyway thank you a lot for sharing this!
    • Feb 26 2021 | 2:10 pm
      Hi folks - thanks for the bug reports - can you make issues on github (make sure to list your system specs, param settings and OS) - just "it crashes" isn't quite as useful for finding out what that could be :)
      I also have an update to both of these I need to push that is 80% complete when I can get two seconds free.. I'll do what I can to get it in soon - I use these patches for the course I teach at University and the semester starts again in a few days!
    • Feb 26 2021 | 3:12 pm
      I would like to draw the mesh in Max instead of the electron window. Is it possible?
      Yep! I'm drawing them in 3D in these two Max For Live devices you can see below.
      But I just checked and the patch in the repo already has this in it for regular Max - so it's already ready to go :)
      These M4L devices will come out for free in April and will be lots of fun, I actually engineered gesture recognition in them for the face and hands and worked in Palm normal to tell which way up your hands are and stuff too. Plus you can control your modular via CV :)
    • Mar 11 2021 | 8:35 pm
      Gold!
    • Mar 12 2021 | 8:46 am
      BTW, for the face mesh do you know if it's possible to stop the jittering somehow? And I see it does not detect ample motion like a wink. I am guessing there is no solving this, right?
    • May 21 2021 | 11:27 am
      Any news regarding the 2 things I wrote above? Are there any solutions? Thank you so much!
    • May 28 2021 | 6:30 am
      BTW, for the face mesh do you know if it's possible to stop the jittering somehow? And I see it does not detect ample motion like a wink. I am guessing there is no solving this, right?
      @YGREQ - I've found lighting conditions dramatically affect it - also if you wear a hat, glasses or have facial hair it will not perform optimally (like FaceOSC) I can accurately detect a wink when I place a good light source evenly frontlighting my face. I just use a small 30cm fluro tube I bought for $5 and it works much better for detecting eyes (front lighting will cut down on shadows in eye sockets) and also helps eliminating jitter (when the model can't find landmarks).
      The jitter could also be from processing lag if you're dropping frames - but if you're getting a decent FPS it's most likely the lighting conditions. Give it a shot! I was surprised at the difference - but YMMV. In two days I'm *finally* pushing this update that will help with some other stuff! Was going to wait to answer your question till then but that seems a little silly - LMK how you go
    • May 30 2021 | 6:21 pm
      Oh! Thnk you so much for ayour answer. I will test asap with proper lighting conditions!<3
    • Aug 16 2021 | 3:32 pm
      Thanks so much for this, it's great! Was wondering if you were thinking of doing the same for the holistic whole body tracking.....would be eternally grateful if you did or could point to how to do so.
    • Nov 04 2021 | 10:00 am
      @Monica, you can find an openPose integration here which is for body tracking: https://cycling74.com/forums/openpose-for-max-sh%C3%A4ring-some-hot-machine-learning-stuff
    • Nov 04 2021 | 10:28 am
      I am looking to map the MeshMap to a mask (an actual mask in a png file with alpha). Any clues on how I can achieve this? I am thinking of just sending the osc data to map the mask in openGL. But maybe there is a simpler solution out there. Also if you already achieved this maybe you don't mind sharing the patch? Thank you so much!
    • Nov 04 2021 | 10:35 am
      A problem I notice: 1.Node for Max - Mediapipe:: Facemesh window looses its image and resets itself from time to time coming back up after a few seconds. 2.Once it comes back up, the OSC data stops streaming even though the face is detected again. I have to Stop the patch and restart it. Is this a general problem or am I doing something wrong? BTW, the link to https://github.com/tensorflow/tfjs-models/tree/master/handpose is taking nowhere. Is the original project dead? :/
    • Dec 31 2021 | 12:09 am
      Having a small problem with HandPose and was wondering if anyone else experienced it as well. I can only get handpose to recognize my webcam if I start the program with it unplugged and then access it through OBS. This is not the case with FaceMesh or Posenet, which i havent tested extensively, but seem to work fine. The error message i get is : "this browser does not support video capture, or this device does not have a camera " In the handpose camera.html file (when i Toggle Developer Tools) I am getting an error : "DOMException: Could not start video source" as well as a lot of "failed to load SourceMap" things. Anyway just throwing this out there.
    • Feb 09 2022 | 10:39 am
      This is amazing work, but I'm having trouble with this patch. It won't turn on my webcam, and the jit.pwindow from the jit.world where it reads your hand mesh is just white/blank and no hand tracking data are going through the other parts of the patch. When I add a jit.grab to the jit.world it opens the webcam, but the image is just superimposed over the blank white pwindow and still no hand recognition.
      When I open the patch, I hit the button to Install dependencies in NPM, it completes, then hit the toggle, but no camera or hand tracking. No Max errors listed either and Electron seems to open, though I haven't used these connective programs before so I'm not sure how to trouble shoot.
      Any suggestions? Thanks again for this work! I'm excited to experiment with it for a project I've wanted to do for a long time. -j
    • Feb 09 2022 | 1:27 pm
      Hi Jacob, Try what ive laid out above.
      Start Handpose, and then try to access your webcam from OBS.
      what operating system?
    • Feb 13 2022 | 9:14 am
      Thanks Dante! Mac OS 12.0.1 I opened OBS which turns on my camera, and then opened the Handpose patch, but no change in the issue: a blank white image is displayed in both the pwindow coming out of the jit.world context and a blank white external display in the Electron program.
      Any other suggestions?
      ...actually i found a weird bug that might be a problem. It's a little hard to explain but in Max when I open the n4m-Handpose patch, there is an error saying "can't find file sketch.js" When I double click that error it opens the subpatch "p drawHandToWorld" and shows that there is an js call object "js sketch.js," (pics attached) and when I double click it, it opens an empty window (no code). That's the first pic below. Plus there isn't a "sketch.js" program in the n4m-Handpose directory, so i figure i maybe accidentally changed that js object when it was unlocked and saved it, etc. However, when I go back to the main patch window and then click on the "p fingerPositions"-->"p drawHandToWorld," the same js object that had the error is now named "js n4m-handpose_gl-sketch.js" and has a program written in it (second pic). So it seems like there are two versions of that subpatch in this patch somewhere, and when I do a Find for the "fingerPositions" text, it says there are 2 matches, but I can only find one from the main patch window (third pic)...something fishy?
    • Feb 25 2022 | 3:47 am
      Hey Jacob, Report the issues on github like Lysdexic requests. Maybe it just needs a little tweaking. Don't forget to list your specs.