New AI app NaviSense helps visually impaired people find objects and navigate. How it works
A new AI-based app called NaviSense helps visually impaired users find everyday objects faster and identify objects in real-time without preloaded databases.
A new AI-based app called NaviSense helps visually impaired users find everyday objects faster and identify objects in real-time without preloaded databases.
Interesting Engineering writes about this, noting that the application connects to external LLMs and VLMs.
The app was developed by researchers at Pennsylvania State University on the basis of smartphones and uses artificial intelligence not only to identify objects, but also to make it easier for visually impaired users to find them in space using audio and tactile cues.
The team presented the technology at the ACM SIGACCESS ASSETS 25 conference in Denver, where it received one of the awards.
NaviSense aims to solve long-standing problems with similar software, as many modern tools for visually impaired people rely on libraries of objects that must be pre-loaded.
Professor Vijaykrishnan Narayanan, who was involved in the development of the app, said that this limits the flexibility of such technological solutions.
«Previously, object models had to be preloaded into the service’s memory before they could be recognized. This is very inefficient and gives users much less flexibility when using these tools,» said Narayanan.
He said the app team turned to artificial intelligence to overcome this bottleneck.
The app connects to an external server running large-scale language models, allowing NaviSense to interpret voice prompts, scan the environment, and identify targets without relying on static databases.
«Using VLM and LLM, NaviSense can recognize objects in its environment in real time based on voice commands, without the need to pre-load object models. This is a major milestone for this technology,» Narayanan assured.
NaviSense listens to the user’s verbal query, searches for objects in space, and filters out irrelevant ones. When the system needs clarification, it asks additional questions.
In addition, the system tracks the user’s hand movements, controlling the movement of the phone, and then provides directional guidance to help them reach the object.
The team tested NaviSense with 12 participants in a controlled environment.
A team of scientists is currently improving the model’s energy consumption and efficiency.
«This technology is quite close to commercial release, and we are working to make it even more accessible,» Narayanan said.
We previously wrote that Ukrainian startup CheckEye, which is developing an artificial intelligence-based product for vision testing, attracted $700,000 in investment over three years.
Meanwhile, a team of researchers from the US and the UK has assembled an artificial neuron that behaves almost like a living brain cell, copying its impulses and adapting to new signals. This could become the basis for robots that better sense and understand the world around them.
Neuralink’s Blindsight chip was once again tested on monkeys: it helped them see what was not in their field of vision
Neuralink’s Blindsight chip was once again tested on monkeys: it helped them see what was not in their field of vision
Kharkiv doctors won at IT Arena 2024 with their eXtra Vision AR technology. Here’s how the tool works, allowing doctors to see through the patient
Kharkiv doctors won at IT Arena 2024 with their eXtra Vision AR technology. Here’s how the tool works, allowing doctors to see through the patient
Read the country’s main IT news in our Telegram
Read the country’s main IT news in our Telegram
