Shivendra Agrawal (lead)
Suresh Nayak (Graduate researcher)
Ashutosh Naik (Graduate researcher)
Bradley Hayes (Advisor)
We are working towards making an end-to-end system that can assist with independent grocery shopping as shopping with sighted guide is prohibitive and often causes a loss of privacy. Grocery shopping primarily consists of three main subtasks: navigation, product retrieval, and product examination. Our current work focuses on product retrieval. ShelfHelp can locate items on the shelf and verbally provide fine-grain manipulation guidance to help people retrieve the desired item from an aisle.
Video with sound (recommended version)
ShelfHelp uses a novel 2-stage CV pipeline to locate products on the shelf. The pipeline doesn’t need any re-training. In the first stage, we use a YoloV5 network that we train on the SKU-110K dataset, giving us the most likely bounding boxes to contain any product. In the second stage, we take these most likely regions and compare them against the image of the desired product. We freeze the weights of an autoencoder that we train on the MS-COCO dataset and take just the encoder portion to extract features from the desired product image and the proposed regions (the feature vectors are compared using cosine similarity). For each new image added to the database, we pass it through the frozen encoder and save the generated feature vector on disk. This step again doesn’t require any re-training of the autoencoder.
- AAMASShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic CaneIn 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS) 2023
- IROSShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic CaneIn IROS 2022 SCIAR Workshop 2022