Microsoft made available yesterday the Kinect for Windows SDK which will allow.Net developers to write Kinecting apps in C, C#, or VB, bringing new uses to the already runaway hit Kinect. The Kinect for Windows SDK beta includes drivers, rich APIs, installation documents, and resource materials.
Article Index
Introduction To Kinect
Depth Map
Microsoft SDK
Beyond Kinect Games
The Microsoft SDK
After an initially cool reaction to the open source drivers, Microsoft realized that it had a big hit on its hands and decided to produce first an SDK, which could be used with the existing Kinect hardware, and a little later a special Kinect tailored to being used with a PC for more experimental purposes.
The Kinect has been a huge hit for Microsoft and has given the 360 a new lease on life in its sixth year on the market. Microsoft has released an SDK for developers to create Windows apps in.
The Microsoft Kinect SDK has a number of drawbacks compared to the open source drivers and middleware from PrimeSense. The first is that the SDK only works with Windows 7 and Windows 8. You can't use it to develop applications under Linux or the Mac. Also it isn't open source, so you can look under the hood to find out how it works and modify it to do what you want.
Microsoft Kinect Sdk
There may be some negatives but there are also a number of positives. The first is that, unlike the open source drivers, the SKD drivers are complete and plug-and-play. They are easy to use. In addition they give you access to the raw data streams and the audio capabilities. As well as providing color and depth data, the Kinect has an array of microphones that can be used to perform speech input and other advanced audio processing. So far the audio capabilities of the Kinect have mostly been ignored by experimenters - perhaps because of the lack of support for it in the open source drivers.
Another really good reason for using the Microsoft SDK is that it provides a much better Skeleton recognition and tracking algorithm that you will find in alternatives and in PrimeSense's NITE software. So if you intend to use player tracking then the Microsoft SDK does it best.
Finally you can choose to use C#, Visual Basic .NET or C++ to write your programs in using either the full Visual Studio IDE or the free Express editions. In this book all of the examples are in C# - an edition using C++ is in preparation. The reason for choosing C# is that while it isn't as fast as C++ it is easier and has access to a lot of easy to use controls and class libraries.
Tracking People and Objects
So how does Kinect track people?
There are two answers to this question. The old way and the new way designed by Microsoft Research.
The new Microsoft way of doing things has significant advantages, but at the moment only Microsoft has access to it. However, it has built the same algorithm into the SDK, so you can use it, but you can't modify it.
Until the SDK became available the most used body tracking software was NITE from PrimeSense. This isn't open source but you can use it via a licence key that is provided at the OpenNI website. This works in the way that all body tracking software worked until Microsoft found a better way, so let's look briefly at its principles of operation as this is the sort of method you will have to implement to track more general objects.
The Nite software takes the raw depth map and finds a skeleton position that shows the body position of the subject. Maccleanse 7 0 1. It does this by performing a segmentation of the depth map into objects and then tracks the objects as they move from frame-to-frame.
This is achieved by construct an avatar, i.e a model of the body that is being detected, and attempting to find a match in the data provided by the depth camera. Tracking is a matter of updating the match by moving the avatar as the data changes.
This was the basis of the first Kinect software that Microsoft tried out and it didn't work well enough for a commercial product. After about a minute or so it tended to lose the track and then not be able to recover it. It also had the problem that it only worked for people when were the same size and shape as the system's developer - because that was the size and shape of the avatar used for matching.
Body recognition using the SDK
Microsoft Research has recently published a scientific paper and a video showing how the Kinect body tracking algorithm works - it's almost as amazing as some of the uses the Kinect has been put to!
There are a number of different components that make Kinect a breakthrough. Its hardware is well designed and does the job at an affordable price. However once you have stopped being amazed by the fast depth measuring hardware your attention then has to fall on the way that it does body tracking. In this case the hero is a fairly classical pattern recognition technique but implemented with style.
So what did Microsoft Research do about the problem to make the Kinect work so much better?
They went back to first principles and decided to build a body recognition system that didn't depend on tracking but located body parts based on a local analysis of each pixel. Traditional pattern recognition works by training a decision making structure from lots of examples of the target. In order for this to work you generally present the classifier with lots of measurements of 'features' which you hope contain the information needed to recognise the object. In many cases it is the task of designing the features to be measured that is the difficult task.
The features that were used might surprise you in that they are simple and it is far from obvious that they do contain the information necessary to identify body parts.
The features are all based on a simple formula:
where (u,v) are a pair of displacement vectors and d(c) is the depth i.e. distance from the Kinect of the pixel at x. This is a very simple feature it is simply the difference in depth to two pixels offset from the target pixel by u and v.
The only complication is that the offset is scaled by the distance of the target pixel i.e. divided by d(x). This makes the offset depth independent and scales them with the apparent size of the body.
It is clear that these features measure something to do with the 3D shape of the area around the pixel - that they are sufficient to tell the difference between say an arm or a leg is another matter.
<ASIN:0596518390>
<ASIN:0321643399>
<ASIN:3540436782>
Notice: MediaWiki has been updated. Report any rough edges to [email protected]
Welcome to the OpenKinect project
Language:
English • español • suomi • français • italiano • português do Brasil • 中文(简体)
About
OpenKinect is an open community of people interested in making use of the amazing Xbox Kinect hardware with our PCs and other devices. We are working on free, open source libraries that will enable the Kinect to be used with Windows, Linux, and Mac.
The OpenKinect community consists of over 2000 members contributing their time and code to the Project. Our members have joined this Project with the mission of creating the best possible suite of applications for the Kinect. OpenKinect is a true 'open source' community!
Our primary focus is currently the libfreenect software. Code contributed to OpenKinect where possible is made available under an Apache20 or optional GPL2 license.
Source code is available here: https://github.com/OpenKinect/libfreenect
Get started right away by installing the software to your platform.
Communications
If you want to participate or just watch the progress of the OpenKinect effort, subscribe to the OpenKinect mailing list. In the application form, please tell us something about yourself and you'll be approved automatically. You could also subscribe to the low-traffic announcement-only mailing list.
You can follow us on Twitter@openkinect. Please use the #tag #openkinect when tweeting your work.
You can meet people in your area working on OpenKinect through Meetup Groups:
You can also chat with people developing on OpenKinect software on IRC: #OpenKinect on irc.freenode.net or using this web based chat.
Channel logs (daily rotation) can be found here.
Project information
Project Roadmap - The current roadmap for the project (libfreenect, analysis library, and applications)
People - Who is doing what: project leader, maintainers, contributors etc.
Project History - The bounty, key dates and milestones
Project Policies - The official name of the project, license, contribution policy, developers coordination and decision making
Installation - How to download, build and install on Linux, OS X and Windows
Contributing Code - Official repositories, use of a fork and source header, signing off, submissions and evaluation etc.
Code Integration - How to deal with how we use git: repository policy, git usage, workflow, starting development, integration process etc.
Contributing - There are many ways to contribute: testing, administrative tasks, support related, documentation, collaboration etc.
FAQ - Frequently asked questions
Documentation - Documenation
Project Ideas - Ideas and concepts to explore using OpenKinect
Gallery and websites - Videos and links to things people are doing with OpenKinect
Official Logos - Official OpenKinect logos for use in your projects
API Documentation
High Level - High-level API documentation
Low Level - Low-level API documentation
Wrappers
C Synchronous - Provides functions to get data instead of callbacks
Common Lisp - Getting started with libfreenect on Common Lisp
GFreenect (GLib) - Use Freenect from GLib. Also provides GObject Introspection which means automatic bindings for many other languages (Python, Javascript, Vala)
Utilities
Record - Dumps Kinect data to PPM, PGM, and a bin dump for RGB, Depth, and Accel respectively.
Fakenect - libfreenect simulator/mock interface that lets you use the kinect demos without having a kinect (plays back data from Record)
Knowledge base
Microsoft Azure Kinect Sdk
Protocol Documentation - Kinect USB procotol, structures and hardware control commands for the cameras, motor, LED and audio
Reference design - US Patent Application 'Depth mapping using projected patterns'
NUI Camera DSP - Camera DSP, architecture, instruction set, firmware, and capabilities
lsusb output - Device identifier output
USB Devices - Overview of the hardware devices
USB Protocol Information - Other information about the Kinect USB protocol
Init Analysis - Messing with various init sequences
Imaging Information - Information about the imaging data returned by the Kinect
Research Material - Research material for software layer implementation
Hardware_info - Hardware information
Calibration - Gathering information for including calibration facilities
Links
OpenNI
http://openni.org - Open Natural Interaction, an industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction (NI) devices, applications and middleware
http://github.com/openni - Open source framework for natural interaction devices
http://github.com/PrimeSense/Sensor - Open source driver for the PrimeSensor Development Kit
Tech
http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066/ - Hardware teardown. Chip info is here. (via adafruit)
http://kinecthacks.net/kinect-pinout/ - Pinout info of the Kinect Sensor
http://www.sensorland.com/HowPage090.html - How sensors work and the bayer filter
http://www.numenta.com/htm-overview/education/HTM_CorticalLearningAlgorithms.pdf - Suggestions to implement pseudocode near the end
http://www.dwheeler.com/essays/floss-license-slide.html - Which licenses are compatible with which
http://www.eetimes.com/design/signal-processing-dsp/4211071/Inside-Xbox-360-s-Kinect-controller - Another Hardware Teardown. Note this article incorrectly states that the PS1080 talks to the Marvell chip.
http://nvie.com/posts/a-successful-git-branching-model/ - Model for branching within Git
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/SubmittingPatches - Linux contribution procedure