Assistive device for blind and visually impaired people
SoundMap is a smart belt, equipped with a camera, able to provide real-time information on the surrounding environment of a person through Audio Augmented Reality (Audio AR).
Similarly to the way we easily identify the position of whistling birds, the device scans in real-time the surrounding environment and produces directional sounds (through audio AR) in the earphones connected to the device. Blind and visually impaired people are, therefore, able to map and understand their environment.
The device is rather small and can be worn as a belt or around the chest. It works on an external battery, but for the tests, we used a simple extension cable.
Why we built this?
285 million of people in the world are blind or visually impaired, among which 39 million people are blind. In Switzerland, 20% of the elder suffer from visual impairment.
The world is full of environment understanding softwares, but this softwares are mostly made for autonomous vehicles and robotics. We want to bring these technologies to blind and visually impaired people.
This is a first demonstration of Rudolf identifying people in a room. Rudolf has no prior information on the number of persons to identify and in which region they might be.
In this second example, Rudolf is looking for a chair and sitting on it in just a few seconds.
How does it work?
1. Obstacle Detection
The first step of our pipeline is to generically identify obstacles. To do so, we use a 3D-camera that understands depths and prevents collision by generating sounds that warn the user of an obstacle more on the right or more on the left.
2. Object Recognition
Our object recognition technologies relies on deep-learning to identify more than 20 classes of objects like chairs, tables, benches, bikes, cars... We process the images in real-time, up to 25 frames per second, using optimized hardware and a Raspberry Pi. We then estimate the position in space of all the objects.
3. Sound Map creation
Based on the detected objects, their angle and their distance, we generate a specific type of sound for each object. Each object is then given a sound, and the sound is played in the corresponding direction, with a volume that corresponds to the distance of the object. We use this same technology for obstacle detection, but only with a single sound specific to obstacles.
4. Change of scene detection
Having continuous feedback in the ears might be annoying. To overcome this issue, we only scan the environment when the scene surrounding a user has changed significantly.
Who are we?
The team behind SoundMap:
Maël is a 23 years old Ph.D. student at Idiap and has a background in Statistics, Machine Learning, and Speech Processing. He develops the project and contributes to the core pipeline programming.Bruno is a graduate computer scientist from RWTH Aachen University, with a focus on Computer Vision. He builds innovative and scalable computer vision pipelines.Rudolf is a 26 years old speech processing engineer, who is currently doing a research internship at Idiap. He works on the creation of the sound map from an image.Juan Pablo is a 26 years old mechatronics engineer and first-year Ph.D. student at Idiap. He has expertise in hardware and manages the 3D-printed case.Ophélie is a biomedical engineer postdoc from EPFL, with over 6 years of experience in experimental research. She designs and runs experimental protocols with patients from partner hospitals.Teerawat is a Ph.D. student at the University of Geneva in neuroscience and psychology, with a background in Physics. He joined us during the Science2Market program at EPFL and manages most of the business-orientated topics.Hugues is a graduate student from EPFL in biology and computer vision. He worked on pose estimation previously and helps us build robust software.
10.2020. Our V2 is almost ready! We have built an updated version and will be ready to test it really soon with partners.
09.2020. SoundMap was selected among dozens of projects to follow the Science2Market program at EPFL Innovation Park. This program will provide us training and expertise on ways to bring our technology to the market as soon as possible.
08.2020. SoundMap won the International Create Challenge, an AI-based competition. We won both the 1st prize and the special healthcare award. We are truly thrilled by the results, and there is a lot more to come for this project ! The full live video pitch and ceremony can be found here.
Working with us
We are interested in working with associations for blind and visually impaired people, but also with hospitals and research centers, to collaborate on research projects, and improve the product. Also, if you or someone from you family is visually impaired, and would like to test the product, just send us an email.