Computer Reads Body Language
July 14, 2017 | Carnegie Mellon UniversityEstimated reading time: 4 minutes
Researchers at Carnegie Mellon University's Robotics Institute have enabled a computer to understand body poses and movements of multiple people from video in real time — including, for the first time, the pose of each individual's hands and fingers.
This new method was developed with the help of the Panoptic Studio — a two-story dome embedded with 500 video cameras — and the insights gained from experiments in that facility now make it possible to detect the pose of a group of people using a single camera and a laptop computer.
Yaser Sheikh, associate professor of robotics, said these methods for tracking 2-D human form and motion open up new ways for people and machines to interact with each other and for people to use machines to better understand the world around them. The ability to recognize hand poses, for instance, will make it possible for people to interact with computers in new and more natural ways, such as communicating with computers simply by pointing at things.
Detecting the nuances of nonverbal communication between individuals will allow robots to serve in social spaces, allowing robots to perceive what people around them are doing, what moods they are in and whether they can be interrupted. A self-driving car could get an early warning that a pedestrian is about to step into the street by monitoring body language. Enabling machines to understand human behavior also could enable new approaches to behavioral diagnosis and rehabilitation, for conditions such as autism, dyslexia and depression.
"We communicate almost as much with the movement of our bodies as we do with our voice," Sheikh said. "But computers are more or less blind to it."
In sports analytics, real-time pose detection will make it possible for computers to track not only the position of each player on the field of play, as is now the case, but to know what players are doing with their arms, legs and heads at each point in time. The methods can be used for live events or applied to existing videos.
To encourage more research and applications, the researchers have released their computer code for both multi-person and hand pose estimation. It is being widely used by research groups, and more than 20 commercial groups, including automotive companies, have expressed interest in licensing the technology, Sheikh said.
Sheikh and his colleagues will present reports on their multi-person and hand pose detection methods at CVPR 2017, the Computer Vision and Pattern Recognition Conference July 21-26 in Honolulu.
Tracking multiple people in real time, particularly in social situations where they may be in contact with each other, presents a number of challenges. Simply using programs that track the pose of an individual does not work well when applied to each individual in a group, particularly when that group gets large. Sheikh and his colleagues took a "bottom-up" approach, which first localizes all the body parts in a scene — arms, legs, faces, etc. — and then associates those parts with particular individuals.
The challenges for hand detection are greater. As people use their hands to hold objects and make gestures, a camera is unlikely to see all parts of the hand at the same time. Unlike the face and body, large datasets do not exist of hand images that have been annotated with labels of parts and positions.
But for every image that shows only part of the hand, there often exists another image from a different angle with a full or complementary view of the hand, said Hanbyul Joo, a Ph.D. student in robotics. That's where the researchers were able to make use of CMU's multi-camera Panoptic Studio.
"A single shot gives you 500 views of a person's hand, plus it automatically annotates the hand position," Joo said. "Hands are too small to be annotated by most of our cameras, however, for this study we used just 31 high-definition cameras, but still were able to build a massive data set."
Joo and fellow Ph.D. student Tomas Simon used their hands to generate thousands of views.
"The Panoptic Studio supercharges our research," Sheikh said. It now is being used to improve body, face and hand detectors by jointly training them. Also, as work progresses to move from the 2-D models of humans to 3-D models, the facility's ability to automatically generate annotated images will be crucial, he said.
When the Panoptic Studio was built a decade ago with support from the National Science Foundation, it was not clear what impact it would have, Sheikh said.
"Now, we're able to break through a number of technical barriers primarily as a result of that NSF grant 10 years ago," he said. In addition to sharing the code, we're also sharing all the data captured in the Panoptic Studio."
In addition to Sheikh, the multi-person pose estimation research included Simon and master's degree students Zhe Cao and Shih-En Wei. The hand detection study included Sheikh, Joo, Simon and Iain Matthews, an adjunct faculty member in the Robotics Institute. Gines Hidalgo Martinez, a master's degree student, collaborates on this work, managing the source code.
The CMU AI initiative in the School of Computer Science is advancing artificial intelligence research and education by leveraging the school's strengths in computer vision, machine learning, robotics, natural language processing and human-computer interaction.
Suggested Items
D Coupon Testing and Data Insights With GreenSource Fabrication
04/17/2024 | Marcy LaRont, PCB007 MagazineMarcy LaRont spoke with Steve Karas of GreenSource Fabrication at the SMTA UHDI conference in March. He presented a case study that GreenSource undertook with a customer on critical via reliability with advanced materials and used the experience to highlight the importance and effectiveness of D coupon testing. He also discussed GreenSource’s approach to data aggregation and a new system they developed to use collected data effectively.
IPC Publishes Comprehensive Strategy to Address Electronics Industry’s Global Workforce Challenge
04/10/2024 | IPCOne of the most difficult and urgent challenges facing the electronics industry is a chronic shortage of adequately skilled workers. Today, IPC unveiled an expansion of its strategy to address the workforce challenges of the U.S. electronics manufacturing industry and called on its more than 3,000 member companies to join in the effort.
UHDI Fundamentals: ASC Sponsors Ultra High Density Interconnect Symposium
03/21/2024 | Anaya Vardya, American Standard CircuitsAmerican Standard Circuits (ASC) is a major sponsor of an upcoming symposium on one of the most talked about new technologies that has come along in quite a while: ultra high density interconnect (UHDI).
onsemi Aligns Business Groups to Expand Product Portfolio and Accelerate Growth
03/13/2024 | onsemionsemi announced the formation of the Analog and Mixed-Signal Group (AMG) which will be led by newly appointed group president, Sudhir Gopalswamy.
Indium Corporation Experts to Present at SEMI THERM
03/07/2024 | Indium CorporationIndium Corporation Global Account Manager and Senior Thermal Technologist Tim Jensen and Product Development Specialist for Thermal Interface Materials Miloš Lazić will present on thermal interface material (TIM) technology at SEMI-THERM, taking place March 25–28 in San Jose, California, U.S.