Kinect does Minority Report interface, Air Guitar prototype

By on December 10, 2010, 2:03 PM
Many Kinect motion controller hacks have teased a Minority Report interface, but all of them have only shown that it could be possible with a bit more work. Until now: not only is it possible, but it's been done. The Robot Locomotion Group and Learning Intelligent Systems teams at MIT have developed a system that uses the Kinect drivers for Linux to detect all ten of your fingers, along with your palms, so that you can use them to interact with a display.

"This is a graphical interface inspired by the movie 'Minority Report,' reads the video's description. "It uses the Kinect sensor from Microsoft, and the recently released libfreenect driver for interfacing with the Kinect in linux. The graphical interface and the hand detection software were written at MIT to interface with the open source robotics package 'ROS', developed by Willow Garage ( The hand detection software showcases the abilities of the Point Cloud Library (PCL), a part of ROS that MIT has been helping to optimize. The hand detection software is able to distinguish hands and fingers in a cloud of more than 60,000 points at 30 frames per second, allowing natural, real time interaction."

Next up we have a prototype of an Air Guitar in action, courtesy of Chris Oshea. The prototype was written in C++ and uses the openFrameworks and openCV for image processing, the ofxKinect addon and the libfreenect driver on Mac, as well as help from the openframeworks and openkinect communities.

"First it thresholds the scene to find a person, then uses a histogram to get the most likely depth of a person in the scene," Oshea explains. "Then any pixels closer than the person to the camera are possible hands. It also uses contour extremity finding on the person blob to look for hands in situations where your hand is at the same depth as your body. It only works if you are facing the camera front on. Then it uses one hand as the neck of the guitar, drawing a virtual line from the neck through the person centroid to create the guitar line. The other hand is tracked to see if it passes through this line, strumming the guitar. The neck hand position controls the chord."

Of course, there are still limitations, but Oshea made a point to give a big thank you to Microsoft for bringing the technology to the mass market. That's exactly why the Kinect is so ground-breaking: not only is it cheap, but it's also widely available.

User Comments: 5

Got something to say? Post a comment
Guest said:

This technology is great, cant wait till operating systems are built to incorporate this kind of thing, hope for a redesigned navigation/file-browsing system to marry up to this soon

Guest said:

Good on Microsoft for seeing the potential and buying another companies ideas and innovations and getting it out to the public. It would be nice to see Microsoft come up with some good ideas on their own a little more often though.

Kinnect is cool technology but I think it is premature for a great many uses. The lag time is often over a 1/4 of a second for one thing. Once they get that down to 1/100 th of a second then I think it will really have the potential to move beyond the gimmicky realm.

tonylukac said:

You think Kinect is going to work with an OS. They don't quite have the voice control bugs ironed out. I tried it on Vista and it was disgusting.

Johny47 said:

This is so cool and sci fi(or not) very soon that minority report style evidence board it looked like might not be that far away for people/places like NSA or FBI etc =P

Cota Cota said:

Like i said

PC's - Dos Xbox - Nada :P

Load all comments...

Add New Comment

TechSpot Members
Login or sign up for free,
it takes about 30 seconds.
You may also...
Get complete access to the TechSpot community. Join thousands of technology enthusiasts that contribute and share knowledge in our forum. Get a private inbox, upload your own photo gallery and more.