dp.kinect: max external based on Kinect SDK and compat with jit.openni
Hi all. I've released the first beta for my new dp.kinect max external.
If you know jit.openi ...well... this is the Microsoft SDK version of it.
Full details are at http://hidale.com/dp-kinect/ along with download.
If you find bugs, please open an issue at https://github.com/diablodale/dp.kinect/issues
hi diablodale,
i could test it, but perhaps, u could specify if:
Does it works with the xbox kinect or only with the 'pc' kinect?
Do i have to uninstall (kinect drivers and openni and nite) and install Microsoft SDK?
or the 2 objects ( and their drivers) jit.openni and dp.kinect can coexist on the same win7 system ?
Advantages of going the Microsoft SDK way, respect to openni, present and future...?
better precision? different skeleton tracking? gesture tracking?
by the way, thanks a lot for your work.
I installed windows on my mac only to test jit.openni
still dreaming of a mac build though... but i never coded externals for max, so i have difficulties to imagine building such a complex object with external libraries.
On your site nothing is mentioned about for which platform the external is made. I just guess its Windows only. The zip file only contained a README.md file which seems to be a plain text file...;-)
I have a Mac and would be interested in a Kinectconnection...
hi,
thanks so much,
it works, that's great !
Can also use to record sequences "KinectStudio" ! ,
is it me or it's faster and more accurate than jit.openni ?
mathieu
@Spa, it should work on both versions of the Kinect as allowed in the MIcrosoft license. There are limits and Microsoft specifies them.
You will need to install the MIcrosoft Kinect drivers which are available with the SDK or with the Kinect runtime. Download and install them as you need. I have not tested having the Kinect SDK and SensorKinect (OpenNI) installed at the same time. I am sure that only one set of low-level Kinect hardware drivers can be active at a time. I recommend you choose which Kinect stack you want to use and have only that installed.
I recommend you query google to find the advantages of the Kinect SDK or OpenNI. Its somewhat a religious discussion except that the Kinect SDK has much richer support for the Kinect. The OpenNI stack is bound by the driver support exposed in SensorKinect. At this time, the dp.kinect and dp.kinect objects are almost the same in functionality. The underlying APIs might have more features on both sides (MSFT does audio and face tracking...OpenNI does gestures) but the max externals I've written don't expose that data yet.
There is no dp.kinect planned for Osx. Microsoft will first have to release an SDK and then a Mac developer somewhere will have to port dp.kinect. I doubt that would happen given the lack of porting of jit.openni.
@stefantiedje you are correct. This external is Windows only. Microsoft does not support the Macintosh with their SDK.
If you know of a Mac developer, I am interested is someone porting jit.openni to the Mac. So far, no one has stepped up to do so.
Ooooo. GitHub is somewhat confusing. Sorry. On the download page look below what you clicked. You will see a download package dp.kinect v0.3.9 BETA.zip
@matmat Any speed increase in the dp.kinect object is probably due to the underlying Kinect SDK. I do have some minor code improvements due to my approach in dp.kinect vs jit.openni, but it is probably nothing that would make it easily seen.
If anyone finds bugs or wants an enhancement, please do log them at https://github.com/diablodale/dp.kinect/issues
Love....that external ! Works fast and respond well. The only thing is there's no image in the right pwindow is it normal ? Or there's a special configuration that I have to do ? Nice work BTW
Ben
@benoit, the one at the right is connected to the playermap outlet. This is equivalent output to the usermap on jit.openni. Check the documentation to see how you can use it. In short, it gives you a matrix with cells that have values of the players (users) identified. It can allow you to ignore chairs, floors, walls, etc and instead only "see" the people.
new beta is ready, v0.4.4
* supports multiple kinects
* supports all depthmaps and colormaps
* and yet more functionality too
more details and download locations at http://hidale.com/dp-kinect/
v0.4.5
* now will open the first kinect it finds if you send an open message. This should handle the simple case for most people and is similar to the behavior of jit.grab
Same deets same place http://hidale.com/dp-kinect/
Hey, when you say: * supports multiple kinects, is that for just RGB/depth map or also skeleton tracking? If that's true I'm switching back to windows!
dp.kinect supports anything the MIcrosoft Kinect runtime supports for multiple kinects regarding rgb, depth, and skeleton tracking.
Unfortunately, the Microsoft Kinect runtime (at this time) only supports tracking on one Kinect per PC. As soon as their runtime is updated to support 2+, dp.kinect will immediately support it.
FYI, Microsoft is artificially limiting it to one engine due to their self-imposed CPU limit. I hope that soon, they will remove this artificial limit. For example, on my PC, I can enable everything on dp.kinect with two kinects and I'm only using 15-20% of my CPU.
There is a new runtime and SDK coming out in a few weeks. We can both...hope...that they raise the ceiling.
Tanx for clarifying! It's strange that MS has this limitation too. OpenNI has it as well but there it just malfunctions which makes it look like a bug rather than an imposed limitation. Well at least it's clear in MS's case. AFAIK PrimeSense has never even commented on the issue.
Got an overclocked quad-core multi-threaded i7 CPU with cycles to spare so here's hoping they remove this ridiculous limit...
v0.4.9 is up at http://hidale.com/dp-kinect/
* handles simple case of a PC having only one connect, it automatically connect to that Kinect if you send open with no idkinect set
* unique attribute added, only outputs matrices if a truely new frame is received for the depth or color stream
* joint orientation is now available; note...the format is different than in jit.openni
* sync attribute added, forces that only frames which are within 1/60 sec between each other are available for output. this can work in addition to the unique attribute
thanks Cwiek
cool!
this matrix to jit.phys.multiple and let's go to the physic world !
Hi all. the v0.4.9 beta above has the new @sync feature. I believe it does...technically...work correctly. You will get only sync'd frames. However, I'm not pleased with the number of frames which are discarded. It is due to subtle drift by the three major components (bang from Max, Kinect camera hardware, Kinect SDK subsystem). Very small drift in any of the three can put them out of sequence and result in the 1/60 sec rule not being satisfied.
I have an idea on how to increase the number of sync'd frames to be made available. I will explore this idea in a later beta. Until then, you can choose to have all the frames without 1/60 sec sync (its default behavior) or sync frames to be within 1/60sec (@sync 1). The latter will likely reduce the number of frames made available to you.
v0.5.0 is now available in the normal places like http://hidale.com/dp-kinect/
I made a big improvement in the @sync functionality. There should be many more frames available that are sync'd. You may also notice that the default value for @waittime is no longer zero. If you enable @sync, I recommend you keep the new default value or set @waittime to a value greater than the framerate of the depth sensor.
Please note, the skeleton data does not yet utilize the new sync approach. It will in an upcoming update.
Also @align doesn't do anything yet.
v0.5.3 is now available in the normal places like http://hidale.com/dp-kinect/
@align will now align depth and color frames
@sync will sync color, depth, and skeletons
Dale-
So glad to see you've released a Microsoft driver version! But is my patch going to stop working on October 30th?????
very good question !!!! Or is there gonna be a large cost to pay for personnal usage ? Because diablodale your kinect objects are very good and useful, your place belong in the books written about the kinect...
During the beta period, I have an expiration to encourage feedback, report bugs, and for people to retrieve the new beta versions which fix any bugs or test new features. When these new betas are released, the expiration date for that beta version of dp.kinect will be for another 30 days.
At http://hidale.com/dp-kinect/ I have recently updated the licensing terms for the beta. It also hints to the "flavor" or direction for the eventual licensing terms.
I hope that the external provides the Max community a useful and reliable means to use the Kinect. The feedback from you all during the beta period will help us all.
I had some issues coverting from your old OpenNI version. I think I was getting the position of users' hands in pixels instead of meters. Can I still do that with this version?
[jit.openni @output_skeleton_orientation 1 @skeleton_value_type 1]
@Ed, it is true. I have not yet exposed skeleton joints in screen coordinates; the equivalent of @skeleton_value_type 1
I do have two questions.
1) What is the usefulness of having a skeleton orientation (which is a real world 3x3 xyz rotation matrix) yet have screen coordinates which are only x,y in which the rotation matrix should not be applied?
2) I have not used the orientation matrix on dp.kinect. I output what I get from the SDK but hasn't checked that it is correct. Is it correct? Please note, in dp.kinect, it is a 4x4 matrix (the last rows/column are zero except the bottom right is 1) while in jit.openni it is just the 3x3. Should be simple to convert from the 3x34x4 if you want that. I kept it 4x4 since 'that's what Max6 uses in the anim nodes.
Well, for my current app, I'm not using orientation at all, just the screen position coordinates of the head and hands. The positions are used to interact with video on the screen with the normal Kinect video modified and mixed in. The user video is partially used for user position feedback. If I understand your question correctly, I could see similar applications where the orientation/direction of the head or hands, etc. could be used as another modifier or parameter for certain types of position dependant interaction with video. I wouldn't, at this time at least, give it a very high priority though.
@Ed, ok I understand. FYI, if you don't need the orientation, I recommend you not have @output_skeleton_orientation=1 set. It'll save a few CPU cycles. Of course, its completely your choice. :-)
There is likely to be an update to jit.open very soon; a few minor tweaks were made to make it MacOS compatible. The orientations are strictly for the 3D x,y,z real world space. I might disable the orientation data if @skeleton_value_type=1. Why? Because there is no relation to the projective coordinates without some rather complex mathematics.
My priority at this time is to jump on some features finally exposed in the new v1.6 SDK. It came out a few days ago and I've been digging into the changes like raw IR camera support, depth values greater than 4m, etc.
Documentation is now available at the wiki https://github.com/diablodale/dp.kinect/wiki
New version of dp.kinect is available at https://github.com/diablodale/dp.kinect/downloads
Enjoy! And I look forward to your feedback on this release.
A jit.openni for Mac would be really nice! jit.grab.freenect is missing the essential Skeleton features that I badly need.
At the moment I'm working with OSCeleton and my own made Max app to align everything with the depth map. Far from ideal since I can't use them at the same time.
ps: If you need a beta tester, I would be happy to volunteer!
read over at https://cycling74.com/forums/here-is-my-working-jit-openni-max-jitter-openni-external-for-windows for info on a mac verrsion of jit.openni. Its good news. :-)
There is a new location for download of dp.kinect. Get it at http://hidale.com/dp-kinect/#download
The wiki, readme files, and cycling74 toolbox have all been updated for this new location.
dp.kinect v0.6.4 is now available at the normal download locations. the major change is the addition of Infrared (IR) output. With this addition, the dp.kinect external now has a complete superset of features of jit.openni yet maintains very good compatibility so you can chose the external best for your patch.
and on to v0.6.5. This beta release has only one visible change. I fixed a backwards compatibility bug which now allows the external to work with the v1.5 Kinect runtime.
There are some invisible changes as I prepare dp.kinect for the addition of a major feature...hopefully in March.
Get it at http://hidale.com/dp-kinect/#download
Hi dale, will there be 64bit version of the external, since max 6.1 came out?
Yes, there will be a 64-bit version. :-) Until it is ready, the easy workaround is to run Max x86 and Max 64 both.
Write a simple patch in Max x86 that has the dp.kinect object and then sends its data to localhost using a udpsend (for messages only) or jit.net.send (if you want matrices).
Then, do all the patching you want in Max64 and have a matching udpreceive or jit.net.recv. With everything going over localhost, there is enough bandwidth to meet your needs.
The two Max versions will run side-by-side with no known issues.
I intend to release a new dp.kinect within 2 weeks with several substantial features. Anyone have some time to test a private build? If so, contact me at my email address
Switching over from the jit.openni thread...
Just ported my whole system from OSX to Win7 (basically: downloading a whole bunch of Win versions of the externals I'm using). Adapted your help patch to output 2 skeletons and route them into my merging patch. Quick first test seems to be running smoothly. The 2 skeletons are working and I'm getting very good rendering frame rates. CPU load doesn't exceed 50%.
Really hate having to go back to Windows but if it means no longer having to rely on my elderly macbookpro for the 2nd Kinect then it just needs to be done... Have been wanting to compare Win vs OSX performance for a long time too.
I'm up for testing your new build. Though I'm working in my small home studio for the time being. Don't have the whole installation setup for full on testing right now.
Multiple questions for the kinect master...
Would the Xtion Pro and ProLive work with dp.kinect (other driver??)(multiple)
Would Xtion work with jit.openni (osx and win)
Have you got experience with them?
Using dp.kinect can i have at the same time 2 different depth cameras: a xbox360 kinect and a kinect for windows or an XtionPro. Can i swap them without changing drivers.
Of what i understood, you recommand dp.kinect over jit.openni cos of long term dev with microsoft sdk. The problem is that you licensing is only 30 days, so no permanent install. After nearly 1 year of betas, wouldn't be the time for a licensing allowing user to make long term installation?
I've got the take decision regarding this issue.
By the way, can we hope you'll have the same interest in developing a max external for kinect2 (xbox one)?
Thanks for the hard work
dp.kinect is exclusively for use with the Microsoft Kinect.
Yes, I do known some people have success using the Xtion with jit.openni. There are multiple versions of the Asus sensors. I do not know which do or don't work. It depends on the hardware driver they provide. I do not own an Asus sensor.
dp.kinect supports multiple Kinects on the same PC. Each Kinect much be on its own USB 2.0 controller. It is not possible to have two Kinects on the same USB 2 controller and they fully function. There is not enough bandwidth.
It is not possible to install the official SDK Kinect hardware drivers -and- the open source Kinect OpenNI v1.x hardware drivers at the same time. The drivers conflict with each other because they are trying to control the same hardware. It should be possible to install the Xtion drivers at any time and there be no conflict.
I already have private clients which have licensed dp.kinect for their own private use. Each client has their own needs and I provide them a version of dp.kinect which meets those needs. If you are interested in licensing dp.kinect, you are welcome to contact me at dale@hidale.com
My intention is to continue to develop dp.kinect to support new features of the Microsoft SDK. Two that I am working on now are face recognition and gesture support. As the SDK is updated to support the Kinect2, I would like to support the enhancements or new features which that hardware allows.
Great work Diablodale! Amazing.
I am working on puppetry with the kinect and was surprised that there wasn't orientation tracking of the head in OpenNi but fortunately there is in dp.kinect!
I have one question though (I hope I haven't overlooked it in the wiki):
when I select position + orientation I get 3 floats for position but then followed by 5 floats. Looking at it I guess the first 4 floats are the quaternion. Is that correct? What is the last float then? Quality? Some extra orientation?
curious to know,
Maarten
[edited wrong assumption]
Depending on your specific attribute choices, you are probably seeing:
/skel/userid/jointname x y z confidence qx qy qz qw
offtopic post moved to Jitter forum :
Yo Mr Dale,
your external has expired... any chance to get a new one in the next few hours? i mean i prepared to use it on a presentation today (27th sept) and now it's not working so i'm in trouble :D
Can you help me out or should i try to find another solution?
thanx in advance!
I've uploaded the new v0.8.6 to the normal location http://hidale.com/dp-kinect/
IMPORTANT You need to register the external using the included .dpregkey file. No information is sent/received from the Internet during this registration. How? Its in the wiki https://github.com/diablodale/dp.kinect/wiki#setup
New features in v0.8.6 are:
- color camera control
- new registration mechanism
appreciate Dave, still in time! :) thanx for the great work anyways!
@pry, if you have not yet solved your issue, please open a new topic as your inquiry here is offtopic.
If you think there is a bug in dp.kinect please open an issue at https://github.com/diablodale/dp.kinect/issues
hi.
first I want to say thank you for this object, it's very nice to have directly kinect in max (and on pc too!). I worked with openni first, and synapse, but it seems very unstable for my computer... so dp is kind of a graal! :)
I read all [dp.kinect] wiki, it's working, as I have a moving picture of myself, but now, I have problems to make skeleton calibration. how can I have [calib_success] thing? is it a message? how can I do it? I checked "skeleton" attribute. "skeleton format" is checked too. I'm a bit confused... I don't work with osc-route. next, it says:
"The format of the max route friendly message @skeletonformat=1 with no orientation data:
skel userid jointname x y z confidence"
does it means I have to make for tracking someone's head a [route skel "userid" head x y z confidence] object??? what does it means?
arrrgh!
very confusing... :(
Please refer to the wiki documentation on this subject at https://github.com/diablodale/dp.kinect/wiki/Message-based-Data
Then open the demo patcher that is included with the ZIP file. On that top level patch you will dp.kinect. the 5th outlet will output messages that you are seeking. Everything out the 5th outlet is a message and all those messages are documented at the URL above.
The demo patch shows how to look for "user". you could use the same method to look for "calib_success" if you want.
Are you familiar with Max messages? This is a very important topic to learn to be able to use Max. The tutorials that come with Max (help menu, max tutorials) are excellent and can teach you have to view Max messages, manipulate them, use the route object on them, etc. Perhaps #18 (Data Collections) can assist you.
I suspect you will be using a (route) object to route messages and to get to the data (coordinates or other values) that you want.
I'm not sure I understand what you wrote...but I'm happy that whatever you wanted to work is now working. :-)
Hello Diablodale
Can Dp.kinect be opened and used in max for live?
If you have a full registered copy of Max installed, ...
and are using dp.kinect within Max for Live, ...
I would expect it to work. Though I have not done any testing on it.
I recommend you download the trial and see if it meets your needs at https://hidale.com/shop/dp-kinect/
The wiki has lots of documentation to help you https://github.com/diablodale/dp.kinect/wiki including how to open an issue if you have problems.
Thanks for answering so fast :)
My trial is sadly expired. If I cannot make it work in max for live. Do you know if it is possible to control Abelton Lives tempo control from max/msp?
I am not familiar with controlling Ableton. I suggest you ask that question as a separate thread in these Max forums in the Max for Live area.
I will. Thank you.