The ability to shop independently, especially in grocery stores, is important for maintaining a high quality of life. This can be particularly challenging for people with visual impairments (PVI). Stores carry thousands of products, with approximately 30,000 new products introduced each year in the US market alone, presenting a challenge even for modern computer vision solutions. Through this work, we present a proof-of-concept socially assistive robotic system we call ShelfHelp, and propose novel technical solutions for enhancing instrumented canes traditionally meant for navigation tasks with additional capability within the domain of shopping. ShelfHelp includes a novel visual product locator algorithm designed for use in grocery stores and a novel planner that autonomously issues verbal manipulation guidance commands to guide the user during product retrieval. Through a human subjects study, we show the system’s success in locating and providing effective manipulation guidance to retrieve desired products with novice users. We compare two autonomous verbal guidance modes achieving comparable performance to a human assistance baseline and present encouraging findings that validate our system’s efficiency and effectiveness and through positive subjective metrics including competence, intelligence, and ease of use.
@inproceedings{agrawal2022shelfhelp,title={ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane},author={Agrawal, Shivendra and Nayak, Suresh and Naik, Ashutosh and Hayes, Bradley},booktitle={22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)},topic={shelfhelp},url={},year={2023},}
2022
IROS
ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane
The ability to shop independently, especially in grocery stores, is important for maintaining a high quality of life. This can be particularly challenging for people with visual impairments (PVI). Stores carry thousands of products, with approximately 30,000 new products introduced each year in the US market alone, presenting a challenge even for modern computer vision solutions. In this work we present our work-in-progress investigating technical solutions for enhancing instrumented canes traditionally meant for navigation tasks with capability within the domain of shopping. Our system includes a novel visual product search algorithm designed for use in the wild and a novel planner that autonomously issues verbal commands to guide the user in a reaching task to acquire them.
@inproceedings{agrawal2022shelf,title={ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane},author={Agrawal, Shivendra and Hayes, Bradley},booktitle={IROS 2022 SCIAR Workshop},topic={shelfhelp},year={2022},}
IROS
A Novel Perceptive Robotic Cane with Haptic Navigation for Enabling Vision-Independent Participation in the Social Dynamics of Seat Choice
Goal-based navigation in public places is critical for independent mobility and for breaking barriers that exist for blind or visually impaired (BVI) people in a sight-centric society. Through this work we present a proof-of-concept system that autonomously leverages goal-based navigation assistance and perception to identify socially preferred seats and safely guide its user towards them in unknown indoor environments. The robotic system includes a camera, an IMU, vibrational motors, and a white cane, powered via a backpack-mounted laptop. The system combines techniques from computer vision, robotics, and motion planning with insights from psychology to perform 1) SLAM and object localization, 2) goal disambiguation and scoring, and 3) path planning and guidance. We introduce a novel 2-motor haptic feedback system on the cane’s grip for navigation assistance. Through a pilot user study, we show that the system is successful in classifying and providing haptic navigation guidance to socially preferred seats, while optimizing for users convenience, privacy, and intimacy in addition to increasing their confidence in independent navigation. The implications are encouraging as this technology, with careful design guided by the BVI community, can be adopted and further developed to be used with medical devices enabling the BVI population to better independently engage in socially dynamic situations like seat choice.
@inproceedings{agrawal2022novel,title={A Novel Perceptive Robotic Cane with Haptic Navigation for Enabling Vision-Independent Participation in the Social Dynamics of Seat Choice},author={Agrawal, Shivendra and West, Mary Etta and Hayes, Bradley},booktitle={Proceedings of the IEEERSJ International Conference on Intelligent Robots and Systems},topic={social_guidance},year={2022},}
2019
HRI Best Paper Runner-up
Explanation-based reward coaching to improve human performance via reinforcement learning
For robots to effectively collaborate with humans, it is critical to establish a shared mental model amongst teammates. In the case of incongruous models, catastrophic failures may occur unless mitigating steps are taken. To identify and remedy these potential issues, we propose a novel mechanism for enabling an autonomous system to detect model disparity between itself and a human collaborator, infer the source of the disagreement within the model, evaluate potential consequences of this error, and finally, provide human-interpretable feedback to encourage model correction. This process effectively enables a robot to provide a human with a policy update based on perceived model disparity, reducing the likelihood of costly or dangerous failures during joint task execution. This paper makes two contributions at the intersection of explainable AI (xAI) and human-robot collaboration: 1) The Reward Augmentation and Repair through Explanation (RARE) framework for estimating task understanding and 2) A human subjects study illustrating the effectiveness of reward augmentation-based policy repair in a complex collaborative task.
@inproceedings{tabrez2019explanation,title={Explanation-based reward coaching to improve human performance via reinforcement learning},author={Tabrez, Aaquib and Agrawal, Shivendra and Hayes, Bradley},booktitle={2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)},pages={249--257},year={2019},organization={IEEE},topic={explainable_ai},award={Best Paper Runner-up}}
Posters
“Is this seat taken?” Towards a Novel Robotic Cane for Participation in the Social Dynamics of Seat Choice for Blind IndividualsProject pageWinner, Annual Research Expo 2022
This poster presents our work that describes a robotic cane system that can find socially preferred seats and navigate its users to the seats using a novel haptics method.