Kinect does Minority Report interface, Air Guitar prototype

Many Kinect motion controller hacks have teased a Minority Report interface, but all of them have only shown that it could be possible with a bit more work. Until now: not only is it possible, but it's been done. The Robot Locomotion Group and Learning Intelligent Systems teams at MIT have developed a system that uses the Kinect drivers for Linux to detect all ten of your fingers, along with your palms, so that you can use them to interact with a display.

"This is a graphical interface inspired by the movie 'Minority Report,' reads the video's description. "It uses the Kinect sensor from Microsoft, and the recently released libfreenect driver for interfacing with the Kinect in linux. The graphical interface and the hand detection software were written at MIT to interface with the open source robotics package 'ROS', developed by Willow Garage (willowgarage.com). The hand detection software showcases the abilities of the Point Cloud Library (PCL), a part of ROS that MIT has been helping to optimize. The hand detection software is able to distinguish hands and fingers in a cloud of more than 60,000 points at 30 frames per second, allowing natural, real time interaction."

Next up we have a prototype of an Air Guitar in action, courtesy of Chris Oshea. The prototype was written in C++ and uses the openFrameworks and openCV for image processing, the ofxKinect addon and the libfreenect driver on Mac, as well as help from the openframeworks and openkinect communities.

"First it thresholds the scene to find a person, then uses a histogram to get the most likely depth of a person in the scene," Oshea explains. "Then any pixels closer than the person to the camera are possible hands. It also uses contour extremity finding on the person blob to look for hands in situations where your hand is at the same depth as your body. It only works if you are facing the camera front on. Then it uses one hand as the neck of the guitar, drawing a virtual line from the neck through the person centroid to create the guitar line. The other hand is tracked to see if it passes through this line, strumming the guitar. The neck hand position controls the chord."

Of course, there are still limitations, but Oshea made a point to give a big thank you to Microsoft for bringing the technology to the mass market. That's exactly why the Kinect is so ground-breaking: not only is it cheap, but it's also widely available.