Kinect 1 and Kinect 2 now supported in Max with dp.kinect and dp.kinect2 objects
Hi all. I've finally released v1.0 of my dp.kinect object. Its been release quality for over a year. Yet, I polished it even more and its available for you at http://hidale.com/shop/dp-kinect/
I also have a limited beta of my dp.kinect2 object. This allows you to use the new Kinect v2 sensor in Max. It is compatible with dp.kinect. If you have any existing patches, the only change you might need to make is to put a "2" in the object name. More info on how to get into the limited beta at http://hidale.com/shop/dp-kinect2/
Hello Diablodale
Are dp.kinect 1 are supported for Osx Yosemite and Max7 ?
Microsoft does not support Macintosh/Osx, therefore dp.kinect and dp.kinect2 do not.
Hello Diablodale
I read on your wiki about OpenNi … so that would still be the way to go for the kinect 9model 1414 or new model?) on Mac and OSX Yosemit ?
Sorry if these questions aren't new to you and maybe answered in the forum (I'm trying to read everything about it right now). I'm trying to determine if I should get a Windows system … but I am mostly a Mac user. If you could clarify for me … that would be great … Thanks so much!
Primesense, creator of the first Kinect and OpenNI, has been bought by Apple and the public OpenNI libs were taken offline. While OpenNI-based stuff might still work now it's a shot in the dark for the future. In my experience KinectSDK is superior to OpenNI and the dp.kinect object is the most advanced Kinect external for Max. It made me switch from Mac to Windows for my projects that use Kinect.
DTR writes well. OpenNI, in general, is a dead-end technology and I don't recommend building new solutions which are based on it.
The Kinect v2 sensor is very good hardware and the drivers/SDK for it in generally good condition. If you were starting a new project (not old legacy), I recommend you use the Kinect v2 sensor and solutions like dp.kinect2 to access/control it.
At this time, it is Windows. So using bootcamp on your Mac hardware is also an option.
https://social.msdn.microsoft.com/Forums/en-US/bb379e8b-4258-40d6-92e4-56dd95d7b0bb/confirmed-list-of-usb-30-pcie-cardslaptopsconfigurations-which-work-for-kinect-v2-during?forum=kinectv2sdk
https://social.msdn.microsoft.com/Forums/en-US/20dbadae-dcee-406a-b66f-a182d76cea3b/troubleshooting-and-common-issues-guide?forum=kinectv2sdk
http://blogs.msdn.com/b/kinectforwindows/archive/2014/07/28/developing-with-kinect-v2-on-a-mac.aspx
Thanks to both of you !
I'll try to find a solution to stay on Mac and share with the community what we will find (or write). Sending the infos via OSC could be a solution as well but I want to use the image capture as well as the data from the kinect.
mac for me in this last 3 ys is a delusion
This looks like an option on mac but I haven't tried it yet.
https://github.com/ofTheo/ofxKinectV2
I don't recommend this ofx hack. It is extremely limited and buggy. It is not a Max object/external.
It only outputs unaligned rgb, depth, and ir; and the depth is not output at full 30fps.
In addition no body/joint tracking, no hand tracking, no audio, no face tracking, no accelerometer, no engagement tracking, no emotion detection, etc.
If one is so compelled to only use Mac hardware, then use bootcamp. I have read of success by others on the public microsoft Kinect forums.
Okay.. I hope I can get a clear answer on this.. I am currently using the Kinect 1414 with old openni.. and it is working... However I am curious about the V2 as obviously would work much better and be more responsive.. I have a MBP that has a a windows partition.. If I installed windows 8 on it would I need to use max on the windows side for my patches or could I just use the kinect on the windows side and port the sensor info to the mac side ( with osc or syphon?) to use with Max.. If I need to do it all on the Windows side.. could one use a standalone or do you need to mirror the mac application..I guess I would like a clarification about the workflow to facilitate the K V2 if one has been doing all patches in Mac.. that Apple is throwing such a kink into the works is totally annoying
Thanks for answer..
Ast.
Sorry to slightly hijack this thread.. I'm trying to get hold of a Kinect 2 for Windows, and I can't get one in Europe at all. Have tried Microsoft stores in various countries (I'm based in the UK but have tried France, Germany etc.) but there seems to be no stock at all. I've even tried ordering from the US or Canada, but they won't supply to Europe. Anyone got any ideas perchance?
@ASTRAPHL, dp.kinect2 is a Windows external and Microsoft only supports the Kinect2 on Windows.
It is your choice how you use locally or transmit the data to another computer or VM. I personally recommend Windows for all solutions and do not use Mac.
@Joseph, I have also heard this rumor in some countries. What I heard to do is buy the separate Kinect for XBox One + the Kinect for Windows adapter.
@Joseph yes it's true I couldn't get kinect2 shipped directly abroad either. if you have a friend in the usa maybe they can ship it to you. UK is out of stock!
i buy on from a boy who don t use Kinect 2 in his Xbox one i pay 40 euro + adapter 39 euro
I asked Microsoft here in The Netherlands. The Kinect v2 for Windows isn't produced anymore. It's now the regular Xbox Kinect v2 + Adapter for Windows that you need to get. I did and it works fine with dp.kinect2.
I am using a Kinect sensor model 1414 on Mac OSX Yosemite and OSX Lion. It works beautiful. Anybody interested I can send you the contact person I have who debugged that for me.
hey Marie-Helene,
couldn't you do setup a fork on GitHub of the openNI repos or something like this to share that debugged solution, if you are inclined to sharing it ? i'm very interested and i guess a lot of people are...
I understand but I am not the person who debugged it. So I can only give you the contact.
dp.kinect and dp.kinect2 do not work on any version of Osx. They are Windows only externals.
Yes in deed Diablodale
Your dp.kinect and dp.kinect2 only work on Windows.
And I am sure I will use that one day as it seems very well done.
But for those "stock" on Mac here's a solution
25 euro vs 200 $ isn t the same thing
after all dk.kinect work w/ max
i want said thank to DIABLODALE
who share 2 project on max/msp w/ us
here is not easy to see,
tools are not art
tools are tools
as a suggestion not a criticism:
is impossible find the way to write you
i read this in your site but there is not way to
share show talk art project with you.
read this like as a suggestion not a criticism against you.
"My mission is to strengthen the relationships between one another through art; to create opportunities for artists to express the art that is in them; to bring creativity to corners of our urban areas that thirst for that connection to the art."|
thank
Hi @Luxi. I encourage you to read the support page on my website.
http://hidale.com/support/
It provides you help, documentation, and a way to open a new issue if you can not resolve it yourself. It additionally provides a way to contact me. My intention is to centralize all this information so that others, like youself, can get the support you need.
--Dale
Raising this thread from the dead :-)
i've been using the kinect version 1473 on mac with jit.openni for almost 2 years now but wanted to try the
dp.kinect in bootcamp in windows 7.
I downladed and started the trial but sadly can't get the damn thing to work. ;(
dp.kinect looks very very solid and would love to try and purchase a license.
after unplugging the kinect to try a different usb por ton my laptop
I got a "The Kinect attached is an unsupported model.". I cannot find any mention of this on Dale's site .
Can anyone confirm this : model 1473 kinect v1 does not work with dp.kinect.
Thank you very much for any help on this.
phiol
Microsoft's licensing only supports use of the Kinect for Windows for your solutions and they enforce it with their technology.
I recommend referring to the setup gude at the following link and the licensing section directly linked. Both walk you through the hardware requirements, restrictions, and possibilities.
https://github.com/diablodale/dp.kinect/wiki#terms-of-use-licensing-conditions-and-warranty
Separately....I'm surprised 1473 worked with jit.openni. You are lucky. :-)
If your computer meets the high requirements, consider getting a used XBox One Kinect + the new adapter sold that converts it to USB3. You might be able to get it all under $100.
Thanks a lot diablodale,
I'll try and see if I can find one
thanks
phiol
Thanks. I go test it. Do you thing it's a good tool just for detect bodies presence on a dark environment? I don't need skeleton detection. thanks
The Kinect v2 will detect bodies and skeleton joints in the dark. I recommend it.
The Kinect for Windows v1... it doesn't work as well in the dark when you need to detecting body positions and skeleton joints. You will need to test to see if it works in your specific patch and room.
If you don't need position...then the Kinect for Windows v1 might work for you. You could try using the raw infrared stream. With it, you would look for any big or bright objects. Those objects would probably be people.
If you had still objects (chairs, tables, columns, etc.), your patch could ignore those pixel ranges.
You could also use some externals (jit.change, cv.jit.moments, cv.jit.opticalflow, etc.) which detect changes/deltas in video (like infrared) streams. These changes would probably also be people.
Hello,did someone test dpkinect2 on max/msp 5.07?thx
carl
I casually tried it on Max 5.1.9 sometime in the past but promise no support for it. Try/test it yourself on your specific computer. You have two weeks full functionality to try it. https://hidale.com/shop/dp-kinect2/
FYI: You can upgrade to Max 5.1.9 for free. And the upgrade to modern Max is nicely priced and the feature set difference is substantial.
Thx a lot, very kind
I bought with my friend Max 7 and we installed it on the pc where we make research in lab; we would like to use kinect also with laptop outside lab, so maybe we need for our needs to use two kinects more, ,and so two computers, so we should buy two licences more for Max/msp, but now we don't have so much money:-)
Now we'll see how to move
thx again
cheers
carl
Both Max and dp.kinect support situations where you have a development and/or performance PC. Each has their own licensing terms and abilities.
You can read about dp.kinect terms at http://hidale.com/terms/
For Max, I found this in their FAQs https://cycling74.com/support/faq_authorization/#.Vok6BBUrJhE
Hello Dale,
Thank you for lovely plug-in and responding to all of the question on here, even if they are only bickering about how they want to find way around using your plugin...lol
I am a kinect 1 user about to buy a used v2. I plan to use your plug-in with max v6.1 on Windows 8 via bootcamp loaded on my Macbook Pro. Rarely do I see people reference model numbers of the v2 kinect but I do see they are out there. Do all version work with your plugin?
Thanks!
All production models of the Kinect v2 should work with dp.kinect2. Prototype v2, retail v2, Xbox One v2...all of them should work. :-)
The most common hardware issues I've seen are computer related (cpu/bus speed, usb3 port compatibility, etc.). Best guidance I can provide there is to follow Microsoft's hardware requirements and to run their Kinectv2 config verifier https://dev.windows.com/en-us/kinect/hardware-setup
If you otherwise encounter a problem, please do report it at https://github.com/diablodale/dp.kinect2/issues so I can start investigating.
Happy patching!
Hey Dale. Thanks for your timely response. I am about to purchase my kinect in a few minutes. Yes I have read the hardware specs. It asks for a 3.1ghz processor which I find kinda funny, because not a single Apple laptop comes with anything much over a 2.5hgz, so Im not sure if I should believe that or not...I am hopping it will work with lesser of a model just may not obtain full frame rates, which should be ok. And PRAY that my usb chipset is acceptable. Cheers!
hi,
dp.kinect2 , unique 1. Max 7.1 32 bit
Everytime I enable the colormap output (aligned or not), the fps drops to 15. and so does skeleton and cloud.
Even if i disable speech, depth, cloud, etc.
It does append only with colormap on
if colormap is disabled, everything else run at 28-30fps
My Laptop is quitepowerfull.
Did you encountered a similar issue?
Can you run it colormap (unique 1) at 30 fps?
Does it comes from The kinect that limits the framerate cos of ressources?
or the usb3?
Do you plan a 64 bit version?
Thanks.
I develop some modules with your object:
user selection and priority. skeleton packing and visualisation. screen touch calculation.
I'm thinking about sharing them. What do you think will be the best repository to share it?
Hello. I have had the 64-bit version of dp.,kinect2 since 2014. :-) It is included in the ZIP download. In my experience, it performs slightly better than the 32-bit version.
Microsoft has chosen to limit color output to 15fps in situations where there is "not enough light".
https://github.com/diablodale/dp.kinect2/wiki#known-issues-limitations-report-problems
The Microsoft driver controls this and it is impossible to change. :-/ When you enable multiple types of data output, like color and depth together, the Microsoft drivers keep the frames in sync with each other. If you are in a "not enough light" situation, then the color drops to 15 fps....and because the frames are sync'd by the Microsoft driver then *all* the data drops to 15 fps.
To test this, put your Kinect2 in a place that has much more light. When you get enough light for the Kinect2 color camera, the fps will suddenly change to be ~30fps.
Thank you for choosing dp.kinect2 for your solutions. Where to share your ideas? I think here in the Max forums is one place. You can paste compressed patches. Another place is to put them on github. However because of the patch file format, the detailed version control is not as useful. Still...github is a good place to share open source files/ideas and you can version your patches, put them in directories, etc.
Hi diablodale,
I'm a student and I was working with the trial version, when it expired, and I didn't finish my work, there is any way to have a temporaly code for dp.kinect?
Hope so, waiting your answer.
Hello Adrian. I replied to your email a few hours ago. I'll repeat my answer here.
I provide 14 days for everyone to try and test dp.kinect and dp.kinect2. It seems you had a positive experience and want to use dp.kinect as part of your project. This means you have transitioned out of the try/test phase. You are now doing real work on a project.
I treat everyone equally and fairly. Therefore, a paid license is needed for use in your projects. You can buy a license at the same place you downloaded http://hidale.com/shop/
I do appreciate your question. Thank you for considering dp.kinect for your project solution.
Hi,
I will be interested in getting an aligned color map in High definition, or at double size ( 1024 x 828).
At the moment, your external output it at 512x424 with align = 1 , even if colormapres = 5.
Perhaps when colormapres = 5, you could do a lookup transform with the HD color map.
or perhaps give us access to the internal DepthFrameToCameraSpaceTable so i can do the look up myself on a FHD image.
I could then generate texture_coordinates for jit.gl.mesh
Thanks
Hello. I have good news. I have (silently released) a beta v1.1 of dp.kinect2 which will do alignment of the depthmap -> color using @align=2. You can download the beta at the usual http://hidale.com/shop/dp-kinect2/
Using @align=2 will create a depthmap and a colormap which are the full HD 1920x1080 resolution. Other matrices which depend on the depthmap (IR, player, pointcloud) will also have the HD resolution. By using the optional 5th outlet for a pointcloud https://github.com/diablodale/dp.kinect2/wiki/Matrix-Based-Data#opengl-point-cloud-optional-outlet-5 you can output an HD matrix that can be fed into shaders, jit.gl.sketch, jit.gl.mesh, etc. for displacement maps, quick pointcloud visualization, and more.
Try the beta and tell me if it meets your needs.
Great, i test it right now.
How do you interpolate the depth map(512x424) to FHD?
How do you fill the gaps beetween a background and user, the jump in depth?
Is it really necessary to upscale all the maps when only color is needed?
I mean that heavy cpu use could occur.
perhaps align=2 only for colormap, and align=3 for upscaling all others maps
I pack cloud(floatTo 3x4x8bitPacking)+(norm+user)+rgba maps on a 1536x424 texture that i send through syphon
im afraid that packing 3xFHD is going to be GPU heavy
By the way, do you have a shader for depth > cloud . with intrisics and FOV.
so i dont have to move around a 3 plane cloud texture
Just to know
I get back to you when tested
Patrice
The Kinect SDK provides the necessary APIs for me to create a pixel-pixel mapping between the various sized data feeds from the Kinect. I use what the SDK provides and do not do any proprietary depthrealworld coordinate interpolation of my own.
@align=0 and @align=2 there is no change or upscaling of the colormap data. You get the full unchanged resolution data from the rgb camera. Color data is not shifted, nothing mapped, nothing up/downscaled. This is intentional.
@align=1 will cause the colormap to be aligned/mapped down to the depthmap's 512x424 resolution. This is intentional.
Written in a different way....
@align=0 is Kinect data in native resolutions, nothing aligned/mapped
@align=1 is Kinect data in 512x424 resolutions, everything aligned to the depthmap (which is native 512x424)
@align=2 is Kinect data in 1920x1080 resolutions, everything aligned to the colormap (which is native 1920x1080)
I do not do unneeded work in the code. If you don't need a data feed, then don't enable it and I won't map it internally. I only do mapping/alignment work when you enable a feed. I have a 3 year old quad-core intel laptop. My optimizations in v1.1 keep dp.kinect2's CPU usage to single digit %.
If you want to create a matrix of voxels (x,y,z,r,g,b) then you must align with =1 or =2. Giving you aligned matrices of either 512x424 or 1920x1080. The beta v1.1 also supportes more filtering of data. @depthonlyplayers now supports filtering any combination of the data output. If you look closely, it is a bitfield.
I did consider outputting the "formal" jitter/opengl matrix. However, I chose to not do this because Max required a 12-plane matrix just to output the alpha value. There was too much wasted memory. In this beta, I focused on very low-level instruction set CPU memory cache optimization. I was successful and have measurable improvements in both CPU usage, memory usage, and memory/pci-bus wait times. In a future release, I do plan to have a dp.kinect2 option to output textures directly. This texture option could slightly improve patch performance by removing one step if your patch doesn't need matrices.
Keep in mind that GPUs are designed and optimized to handle 3 and 4 plane textures. One of their top use cases is drawing color textured objects. Therefore, they need to be able to pass around rgb(a) textures. To the GPU there is no difference between RGB texture and XYZ texture...its just 3 (or 4) plane data.
I removed only the 1.0.1 externals, put the 1.1b externals and restart Max 7.1
all the others kinect files stay the same
in 32 and 64 both give:
Error 126 loading external dp.kinect2
tried to renter the Registration Name
no luck
...
Other customers have seen this error; it is a Max error reporting that it can't load the external. My code hasn't even run yet. For example, here is an issue just like yours. https://github.com/diablodale/dp.kinect2/issues/16
If that doesn't resolve your issue, please open a new issue at the same location. I provide a sticky post there detailing the information that I will need to troubleshoot. It is likely because the Kinect runtime need to be re(installed) or there is a Windows runtime DLL missing.
Error 126 _ runtime reinstall not working
issue submited
reinstalling various part of kinect system does not help
but everything works fine in kinect studio ...
going back to dp.kinect v1.0 :/
In regard to optimization:
It will be great to output even the cloud map and user map as texture.
I pass them directly on the gpu and process them with slabs too.
I create normals, erode them or transform them.
I obtains 3 float32 rgba textures of 512x424:
xyzi : xyz + user
uvwi : norm + user
rgba : colormap
I then send them to a jit.gl.mesh with a geometry shader
this geom shader, kills the null points or artifacts and generate diverse pixlet aspects or tubes.
and phong illumination from the norms.
For Spout, it takes only rgba textures with values in between 0. 1. that are converted internally to char(8b).
So i had to scale and encode the xyzi texture (float32) into 4x8bits texture and decode them after the spout receiver back to float32
Perhaps i could send you the patch. it's quite efficient to pass the cloud+user+color from 1 max app to the other.
I created a texture output feature tracking issue at https://github.com/diablodale/dp.kinect2/issues/18
Any related comments or ideas there are appreciated.
various comments on this feature posted on github
Hi,
20/03/2016 _ beta expired
Could you upload the new beta?
thanks
Hi, Spa. I've posted another beta; this one includes fixes for the two bugs you found. Thank you for reporting them.
Beta download at the bottom of https://hidale.com/shop/dp-kinect2/
Hi to everyone that is looking to use Kinect within max, there is a simple free and open source solution using - KinectV2OSC
https://github.com/microcosm/KinectV2-OSC here is a link where you could find it, its stable and very reliable.
If you are working on a simple project that does not require anything than skeletal data this is the best.
It sends data over UDP, there is a free UDP receiver external made by Sadam - https://cycling74.com/tools/the-sadam-library/
And uses OSC which can be found at - http://cnmat.berkeley.edu/patch/4029
I would be happy to supply you with free help patch or any info in case you cannot put it all together, just message me.
Also I am currently adding more features to the KinectV2OSC to support the rest of the data streams.
Using free and open source tools will save you the hassle of buying multiple licenses for different computers and distributing your work the way you like.
Hi all, has anyone done timings for dp.kinect to see how fast it can report positions in ideal conditions? I'm looking for the fastest possible way to respond to music conductors hand motions and was initially advised that Kinect would not be quick enough so have been looking to set up a dumb high frame rate video capture system - see this thread:
But I'd love to use Kinect if possible. It's a taxing case - I need to have visuals react 'simultaneously' to the conductor's down stroke, so every millisecond counts.
Thanks in advance.
I'm also interested in info others can share. Here are some ideas for conversation.
The Kinect v1 and v2 sensor's frame rate provides raw data to the CPU at 1/30 second, i.e. every 33ms. On top of that is processing time needed to convert that raw data into a Max compatible format. When I developed dp.kinect and dp.kinect2, I did performance profiling; hunting down large consumers of CPU. dp.kinect2 has the most recent benefits of that where I used careful parallelization and SIMD instructions. This processing time will range based on your own computer config and the Kinect features enabled. Overwhelmingly, your processing time and lag is determined by the 1/30 second frame rate.
For conversation, lets simplify and assume the raw->Max format processing time of dp.kinect(2) is 0.0 ms. That means that you can get new visual data from the sensor 33ms after it occurs. The data is still rather raw. For example, you will have a 512x424 pixel bitmap in 16-bit monochrome that represents the IR image (aka irmap) seen by the sensor. Now, you need to process that irmap. Are you going to do body/joint analysis and recognition? Are you going to look for a cluster of at least 50 pixels that has a value > 512 to "see" an LED on the tip of the conductor's wand? Are you going to use a library to create a neural network to recognize gestures? There is measurable CPU work that is needed to do this analysis/tracking step. The CPU time needed is not easily guessable; instead needs prototyping.
Still, you already know you are at least 33ms from the actual event.
What about our mind/bodies? Given it's a electrochemical system, there is also lag there. It takes measurable time for the retina to respond to the photons of light, do some local processing, send that signal up to the brain, do low level processing, and higher level cognitive processing. Then, the brain sends a signal to the hands, arm, diaphragm, etc to play the note "on the 1". There are studies suggesting the raw eye->brain is 13-40ms and the end-to-end (eye->brain->hand) is 250-400ms. However...if 250ms was the lag to respond to the downbeat, that would likely be problematic. Well...if we all operate in the same lag, its not a problem...but let's set that aside for this discussion.
So how does a professional music player respond quicker than 250ms? With training and prediction. There are studies which suggest that the processing in the eye and in the larger nervous system does lag correction. It uses old data, trends of it, and new data to simulate/create/act on things which might not have actually been seen yet. When it eventually *is* seen, our bodies update the lag correction and continue this loop until we die.
Computers can do a similar thing. There are algorithms that can use old data to predict new likely data. Simple algorithms like holt's double exponential smoothing are built into dp.kinect and dp.kinect2 and you can tune the algorithm using 5 parameters on the @smoothing attribute. There are also libraries from Microsoft (I have yet to integrate them into dp.kinect/2) and other parties which can create neural nets for gesture recognition. This latter gesture approach is interesting to me for your situation because you could train the net with much real-world conductor data. I would hope that the result is a gesture recognizer that can recognize all the different ways conductors hit the downbeat (relaxed, plucking, bouncy, forceful stomp, kamikaze plunge, etc.) and can accommodate the moving and swaying of the conductor's body, which without special handling, greatly changes the real-world spacial location of the "1".
These gesture recognizers usually use neural nets which have a component of prediction. The training will educate the recognizer that when a "4" occurs, and then there is a downward movement of the wand, that this is likely to be a "1". It will keep looking at the incoming spacial and temporal data matching it to the training and thereby tracking a "1" beginning to form. Based on the past data it will know the likely velocities and accelerations associated with a "1". Given that historical model and the current velocities/accelerations + the tempo of the preceding 1234, it can predict temporarily and in space where/when the "1" will occur. Now with a prediction (which usually has a factor of likely occurrence) in time and space units, other algorithms can now act on that data (e.g. drive a global clock).
Having another sensor which has a faster frame rate may or may not improve your end result. Yes, you can feed more data with less lag into an algorithm in a given unit time. This might allow for faster-than-needed recognition or perhaps-more-importantly correction of a incorrect prediction. Does this improve the end result? Maybe or maybe not. Why? Because it is likely your conductor's tempo is less than 150 beats per minute -> 2.5 beats per second -> 400ms. 400ms - 33ms Kinect lag = 367ms for a gesture recognizer to process data, make a prediction, and get that prediction to the next algorithm (e.g. clock).
367ms is a long time for computers. I have faith that a good gesture recognizer can generate output faster than that. The recognizer can operate in parallel with the incoming data. It does not have to wait until a sensor heartbeat to output the gesture recognized. This can facilitate the clock's "1" event to be async from the sensor heartbeat...and I think that's a good thing.
Thank you so much for that. I had hoped to look into tempo-extraction as a follow-on project after the reactive visualization, but your insights - especially on the prediction and perceptual side of things have made this much more interesting to me. I had originally thought the tempo could be determined by detecting the bottom of downward strokes of the hand/baton (perhaps from video but this would be more error prone than Kinect), but as you suggest, say a more generalized training of a neural net is probably the better bet.
I have ordered the Kinect USB adapter, so look forward to doing a little testing and capturing some conductor’s motions. I’ll report back.
If anyone has recommendations on how to configure the Kinect SDK to minimize latency please let me know.
Hi all, preliminary testing on a mid-range laptop shows a perceived motion lag of ~70ms with the skeletal model displayed by a basic dp.kinect2 patch. I have tested some video capture options too and now have high confidence that Kinect will give the best, i.e. quickest-reacting results for my application. If anyone would like to read more on the testing, please see the other thread: https://cycling74.com/forums/low-latency-video-input-seemingly-impossible-WHfMzJO
My next steps are to invest in some more beefy Windows hardware and develop more realistic Max visualization tests.
(Dale: at some point I definitely want to investigate tempo derivation and beat prediction. If you ever have any interest in collaborating, please let me know.)
I found the following handy summary of 3D Cameras: http://rosindustrial.org/news/2016/1/13/3d-camera-survey
They indicate Kinect latency is 20ms minimum. Dale, you mention the frame rate is 30fps (or 15) above, but could that be just for the visible light camera? I haven't found an official source for this 20ms time, but did find some interesting Kinect articles:
On the use of prediction to offset latency (which they measured at ~100ms)
http://research.microsoft.com/en-us/um/people/benko/publications/2015/p93-knibbe.pdf
On joint smoothing and latency:
https://msdn.microsoft.com/en-us/library/jj131429.aspx?f=255&MSPPError=-2147217396
Fun topic inquiries. I'll have to get yet more techie...
I think it is possible for joints to have a lag time more than 33ms. Technically, all frames can lag as I wrote in the previous post. I don't have any hard test data on the joint lag. Instead, I can conjecture. The Microsoft drivers/APIs for the Kinect do the joint recognition using a pre-trained learning/decision tree that has parts running in parallel on the GPU and CPU both. As far as I know, the joints are recognized with no temporal history. They scan a single depth frame and from that single frame do the entire joint recognition. This is one reason why "synchronized" frame timestamps are slightly off between each other; the depth frame is needed for joint analysis. Perhaps with a fast GPU/CPU they are both within the same 33ms time-window and are therefore bundled together as a sync'd package of frames. However, if one frame crosses the bundle "cutoff time" even by an infinitesimal amount, that late frame will be in the next 33ms bundle *or* discarded if a newer more-in-sync frame is available.
The OS drivers/apis that Microsoft published for the Kinectv2 system will provide frames of visual and joint data no faster than 30fps (33ms). When I wrote/tested dp.kinect2, I coded timers and collected samples. It was always ~33ms. Using the event-style of APIs, Microsoft controls the timing. If one attempts to get data faster than 33ms via polling-APIs, the APIs will fail with catchable error results...Microsoft still controls the timing. Yes, there is a potential for 1/15fps if you use the color camera in dim lighting. The workaround is to not use the color camera (don't enable @colormap) and then you will not encounter the otherwise uncontrollable slower frame rate.
dp.kinect2 is coded to provide synchronized frames of data. At any bang to dp.kinect2, the visual and joints data will be synchronized together. Meaning: where you visually see your elbow in an RGB image will have matching XYZ for the elbow joint data. Overwhelmingly, my customers wanted synchronized data. Technically, it is possible to get unsynchronized data streams. With the v1, you use the @sync attribute. With the v2, you can create two separate dp.kinect2 objects in your patch for more flexibility.
When I was doing those timing tests above, I remember looking at the timestamps for the independent frame-types of data. The timestamps were on a relative clock that all frames jointly used. The frames had generally good sync and I don't remember any major concerns. I don't have that data anymore yet my memory tells me that they were always +/-1 frame (+/- 33ms). I would have raised alarms to Microsoft and remember the escalation if the sync was not good. Earlier in the private alpha for Kinectv2 prototype (I was in that group) there were sync issues but those were fixed before they released it to the public.
The holt's smoothing available with the @smoothing attribute does use temporal data. Part of its algorithm uses prediction to try and compensate for lag. These are the tunable parameters which are part of the whitepaper link discussion. If you want the raw (no holt's smoothing) data, you can set all zeros on that same attribute.
The juggling article you link is interesting. I would enjoy reading more along that topic path. Everyone reading this post, keep in mind that the article's researchers had projector lag. Gamers are familiar with this lag. There is a measurable lag between a signal being output by a computer on HDMI and a projector emitting the colored light. This lag can range from 16ms to 100+ms. The researcher's results are strongly limited by their commodity projector's lag to display an image on the ball. Yes, their projector has 60 fps. But...the frame being emitted at any point in time is the frame the computer sent to it 16->100ms in the past.
Thanks, Dale.
As you point out, intrinsic display latency is a factor that has to be considered so I did some tests on the Asus PA279Q I have been using:
it consistently lags the Windows laptop LCD by 20ms over HDMI. That leaves another 15ms lag found in my external monitor timings which I guess we attribute to the overhead of driving a larger screen area?
For my MacBook Pro, the Asus display lags the laptop by 25ms over HDMI. Over DisplayPort it's fractionally less at 22ms. Interestingly the DisplayPort timings were noticeably more variable than with HDMI.
This is probably a dumb question. It says that there is a 14 day free trial, but I could only find the paid version. Could someone direct me?
Hi. The trial is intrinsic; part of the download itself. The setup guides for both sensors v1 and v2 have step-by-step Wiki so you can start your trial.
v1 sensor and dp.kinect: https://github.com/diablodale/dp.kinect/wiki
v2 sensor and dp.kinect2: https://github.com/diablodale/dp.kinect2/wiki
Hi Dale (or anyone else who wants a go) , am having a play with the trial (v2) and im struggling to get the @face3dmodel working. Im just not getting any vertices, indices or triangle data out. Have tried modifying the v1 help file to no avail. Any help would be most welcome.
The overwhelmingly common issue with getting 3d face model on the v2 Kinect is the requirement for slow rotations of the head while it is scanning and building the model.
You can read of this necessity and the messages (e.g. modelstatus) on which to monitor at https://github.com/diablodale/dp.kinect2/wiki/Message-based-Data#face-tracking
I'll caution also that the Microsoft algorithms often fail to scan and model faces with large beards. The only workaround is to cut the beard. :-|
Unearthing this one.
Did anyone test dp.kinect2 with Max 8 ?
Hi. I've the dev for both externals. I am not aware of any issues with dp.kinect or dp.kinect2 with Max 8. I did a unit test pass of all the features; everything worked.
In addition, I have been in contact with, I think, three customers that specifically were using Max 8 and reported a problem. After investigation, none of them had any issues related to dp.kinect or dp.kinect2.
Instead, the issues were Max. For example, Cycling74 yet again moved the search path/folders in which Max 8 looks for externals/packages. After the customer installed their external (they used dp.kinect2) into the new Max folder, it worked well. The install docs in the wiki are already updated to remind customers of Max's folder change.
If anyone does have an issue, please check https://hidale.com/support/ for install docs, past resolved issues and their solutions, or how to open a new issue if nothing there helps.
Happy patching! :-)
Hi (diablo)Dale, thanks a lot for you answer, this is very precious.