Seeing AI app describes the world to users with vision disabilities

Americas, May 15 2017

Many millennials spent many an afternoon watching the educational science program, Bill Nye The Science Guy. In light of policy changes in scientific research in the USA, Netflix released an exclusive series called Bill Nye Saves the World, bringing a science-based program targeting the same millenials that grew up watching him. This program explores topics from modern medicine, climate change, artificial intelligence and other controversial topics related to science.

A Man jumping through the air, doing a trick on a skatebaord

In the third episode, “Machines Take Over the World,” Bill Nye explores how technology improves the lives of people, bringing in Saqib Shaikh, a software engineer for Microsoft that has been blind since he was seven. In hopes of improving the lives of people with vision disabilities, Shaikh developed SeeingAI, an app that describes scenes or imagery to the user in real time using talking computer technology.

At this time the app can only describe or make educated guesses based on what it has already ‘seen’ through visual algorithms. However, the basis for its visuals is already quite extensive, able to tell the user someone’s age, gender or even describe emotional cues. It is also able to memorize a person’s face as was seen when identifying Bill Nye on the episode the app was featured in of Bill Nye Saves the World.

One challenge that the developers have been tackling has been teaching the AI to be able to describe a scene in the same way that a person with sight would in plain speech. Therefore, an important factor has been intuitive learning in a way that is typical in human language. Besides being able to identify objects, scenes or people, SeeingAI can also take a photo of text, such as a book, news article, labels, or menus, and translate it to spoken word for the user. The app has currently been developed for use with smartphones, as well as Pivothead glasses.

Perhaps one of the greatest benefits that Shaikh boasts of SeeingAI is the independence that it allows the user to have. Rather than having to rely only on one’s own senses or needing a companion to describe what’s going on around them, the app enables users the freedom to explore their environments alone.

While SeeingAI has been developed, it has not yet been given a set release date for the public, but this will surely be a welcome addition to people with vision disabilities. Watch the video Below:


Share this post: