Identifying Artificial Intelligence 'Blind Spots'
January 25, 2019 | MITEstimated reading time: 6 minutes
Once the feedback data from the human is compiled, the system essentially has a list of situations and, for each situation, multiple labels saying its actions were acceptable or unacceptable. A single situation can receive many different signals, because the system perceives many situations as identical. For example, an autonomous car may have cruised alongside a large car many times without slowing down and pulling over. But, in only one instance, an ambulance, which appears exactly the same to the system, cruises by. The autonomous car doesn’t pull over and receives a feedback signal that the system took an unacceptable action.
“At that point, the system has been given multiple contradictory signals from a human: some with a large car beside it, and it was doing fine, and one where there was an ambulance in the same exact location, but that wasn’t fine. The system makes a little note that it did something wrong, but it doesn’t know why,” Ramakrishnan says. “Because the agent is getting all these contradictory signals, the next step is compiling the information to ask, ‘How likely am I to make a mistake in this situation where I received these mixed signals?’”
Intelligent Aggregation
The end goal is to have these ambiguous situations labeled as blind spots. But that goes beyond simply tallying the acceptable and unacceptable actions for each situation. If the system performed correct actions nine times out of 10 in the ambulance situation, for instance, a simple majority vote would label that situation as safe.
“But because unacceptable actions are far rarer than acceptable actions, the system will eventually learn to predict all situations as safe, which can be extremely dangerous,” Ramakrishnan says.
To that end, the researchers used the Dawid-Skene algorithm, a machine-learning method used commonly for crowdsourcing to handle label noise. The algorithm takes as input a list of situations, each having a set of noisy “acceptable” and “unacceptable” labels. Then it aggregates all the data and uses some probability calculations to identify patterns in the labels of predicted blind spots and patterns for predicted safe situations. Using that information, it outputs a single aggregated “safe” or “blind spot” label for each situation along with a its confidence level in that label. Notably, the algorithm can learn in a situation where it may have, for instance, performed acceptably 90 percent of the time, the situation is still ambiguous enough to merit a “blind spot.”
In the end, the algorithm produces a type of “heat map,” where each situation from the system’s original training is assigned low-to-high probability of being a blind spot for the system.
“When the system is deployed into the real world, it can use this learned model to act more cautiously and intelligently. If the learned model predicts a state to be a blind spot with high probability, the system can query a human for the acceptable action, allowing for safer execution,” Ramakrishnan says.
Page 2 of 2Suggested Items
Latest Test and Inspection Solutions from GOEPEL electronic at SMTconnect 2024
04/29/2024 | GOEPEL electronicGOEPEL electronic will be demonstrating automated test and inspection equipment at SMTconnect, taking place in Nuremberg from June 11 to 13, 2024.
TSMC Celebrates 30th North America Technology Symposium
04/29/2024 | TSMCTSMC unveiled its newest semiconductor process, advanced packaging, and 3D IC technologies for powering the next generation of AI innovations with silicon leadership at the Company’s 2024 North America Technology Symposium.
QinetiQ Achieves UK’s First Jet-to-Jet Teaming Between Aircraft and Autonomous Drone
04/29/2024 | QinetiQQinetiQ has successfully trialled the UK’s first Crewed-Uncrewed-Teaming demonstration between a crewed aircraft and an autonomous jet drone.
Lockheed Martin Australia, The Department Of Defence Sign Strategic Partnership Head Contract
04/26/2024 | Lockheed MartinLockheed Martin Australia signed a landmark AUD$500 million contract with the Department of Defence to build Australia’s future Joint Air Battle Management System under project - AIR6500 Phase 1 (AIR6500-1).
Cadence, TSMC Collaborate on Wide-Ranging Innovations to Transform System and Semiconductor Design
04/25/2024 | Cadence Design SystemsCadence Design Systems, Inc. and TSMC have extended their longstanding collaboration by announcing a broad range of innovative technology advancements to accelerate design, including developments ranging from 3D-IC and advanced process nodes to design IP and photonics.