Am I Allergic to This? Assisting Sight Impaired People in the Kitchen

What is it about?

Blind and partially sighted people need assistance with food packaging that often contains safety-critical information. Therefore, Textual Visual Question Answering (VQA) could prove critical in increasing the independence of people affected by sight loss. For example, comprehending text within an image is necessary to determine: what type of soup is in a can, how long to cook a microwave meal, when a box of eggs will expire, and whether a meal contains an ingredient they are allergic to. This handful of examples relate to a kitchen setting - a particularly challenging area for visually impaired people. We extended the existing Aye-saac voice assistant prototype with this task and setting in mind. We developed textual VQA components to accurately understand what a user is asking, extract relevant text from images in an intelligent manner, and to provide natural language answers that build upon the context of previous questions. As our system is created to be assistive, we designed it with a particular focus on privacy, transparency, and controllability. These are vital objectives that existing systems do not cover. We found that our system outperformed other VQA systems when asked real food packaging questions from visually impaired people in the VizWiz VQA dataset.

Featured Image

Photo by Brett Jordan on Unsplash

Photo by Brett Jordan on Unsplash

Why is it important?

Malnutrition is a seemingly unrelated health issue that is actually associated with sight loss. Grocery shopping is difficult, eating is difficult, and meal prep can even be dangerous - with a risk of injury and people affected by sight loss reporting that they feel unsafe. With no/reduced spatial awareness or depth perception, sharp knives and extremely hot objects are clearly of concern. We have tried to highlight the importance of textual VQA with this in mind, and we really hope that it becomes a fundamental part of future VQA systems and evaluation. In addition, we believe that privacy, transparency, and controllability are necessary goals when designing assistive technologies - especially when answering safety-critical questions.