3/2/17 ABBI Project Completed

This week saw the conclusion of our EU-funded ABBI project, a three-year collaboration with the Istituto Italiano di Tecnologia, Lund University, University of Hamburg, and the Chiossone Institute. You can read more about the ABBI project here.

The primary aim of the project was to develop and evaluate an audio bracelet to assist in the rehabilitation of young visually impaired children, to help them develop spatial understanding and movement skills. Our recent research on the project investigated how the capabilities of the ABBI audio bracelet could be used in other ways, to assist visually impaired children. This culminated in our CHI 2017 paper about audible beacons and how sound from such devices could be used to support independent play and movement. The following video gives a summary of this work.

ABBI

Introduction

The ABBI (Audio Bracelet for Blind Interaction) project aims at improving spatial cognition, mobility and social skills in visually impaired children. It will develop motion-sensitive “bracelets” that use sound to convey spatial information (e.g., movement, position), to help develop spatial awareness and understanding. Our role on the project at Glasgow is to develop software for the project, to research sound design, and to investigate novel ways of using the ABBI bracelets to help people with visual impairment.

Research Summaries

Publications

  • Audible Beacons and Wearables in Schools: Helping Young Visually Impaired Children Play and Move Independently. Euan Freeman, Graham Wilson, Stephen Brewster, Gabriel Baud-Bovy, Charlotte Magnusson, and Hector Caltenco. Proceedings of CHI 2017.
  • Towards a Multimodal Adaptive Lighting System for Visually Impaired Children. Euan Freeman, Graham Wilson, and Stephen Brewster. Proceedings of ICMI 2016 Demos.
  • Multimodal Affective Feedback: Combining Thermal, Vibrotactile, Audio and Visual Signals. Graham Wilson, Euan Freeman, and Stephen Brewster. Proceedings of ICMI 2016 Demos.
  • Automatically Adapting Home Lighting to Assist Visually Impaired Children. Euan Freeman, Graham Wilson, and Stephen Brewster. Proceedings of NordiCHI 2016 Posters.
  • Using Dynamic Audio Feedback to Support Peripersonal Reaching in Young Visually Impaired People. Graham Wilson and Stephen Brewster. Proceedings of ASSETS 2016.
  • Automatically Adapting Home Lighting to Assist Visually Impaired Children. Euan Freeman, Graham Wilson, and Stephen Brewster. Proceedings of CHI 2016 Extended Abstracts.
  • Using Dynamic Audio Feedback to Support Peripersonal Reaching in Visually Impaired People. Graham Wilson and Stephen Brewster. Proceedings of ASSETS 2015 Demos.
  • Effects of Sound Type on Recreating the Trajectory of a Moving Source. Graham Wilson, Stephen Brewster, Hector Caltenco, Charlotte Magnusson, Sara Finocchietti, Gabriel Baud-Bovy, and Monica Gori. Proceedings of CHI 2015 Extended Abstracts.

Website & Social

Project Partners

The ABBI project comprised five partners from a variety of backgrounds. Our role was to investigate feedback and interaction design, whilst developing software for the project.

glasgowlogo   lundlogo_wide   iit_logo   uegnr7kt   uhh_logo

Acknowledgements

This project was funded by the European Commission’s FP7 programme (#611452).

flag_yellow_low

ABBI: Adaptive Lighting for Visually Impaired Children

Introduction

The ABBI (Audible Bracelet for Blind Interaction) project has been developing an audio bracelet for visually impaired children. Audio bracelets are wearable sound sources that produce sound in response to movement. The main purpose of these audio bracelets is rehabilitation for visually impaired children, as they can be used in activities that improve spatial cognition.

Adapting home lighting

The ABBI bracelet has other capabilities beyond making sounds in response to movement. One of these is the ability to be a Bluetooth beacon, a device that can be used for estimating proximity to people or places. At University of Glasgow, we investigated if we could use the beacon capabilities of the ABBI bracelet to adapt the lighting at home, based on a child’s location and activity. For example, if a child entered their bedroom and we could detect that, then we could automatically turn the lights on and increase them to maximum brightness, to help them see furniture and obstacles. We could also change the lighting during play to stimulate a child’s vision; for example, a coloured lamp could create colourful patterns in response to an ABBI bracelet moving around the room.

A video at the bottom of this page demonstrates a prototype of an adaptive lighting system. We describe the development and initial evaluation of this prototype in two papers, also listed below.

Publications

  • Freeman, E., Wilson, G. and Brewster, S.: Automatically Adapting Home Lighting to Assist Visually Impaired Children. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction – NordiCHI ’16, ACM Press, 2016.
  • Freeman, E., Wilson, G., and Brewster, S.: Towards a Multimodal Adaptive Lighting System for Visually Impaired Children. In Proceedings of ICMI 2016 Demonstrations, ACM Press, pp. 398-399, 2016.

Video demonstration

ABBI: Audible Beacons for Visual Impairment

Introduction

The ABBI (Audible Bracelet for Blind Interaction) project has been developing an audio bracelet for visually impaired children. Audio bracelets are wearable sound sources that produce sound in response to movement. The main purpose of these audio bracelets is rehabilitation for visually impaired children, as they can be used in activities that improve spatial cognition. Some of our research at Glasgow investigated other ways of using sound to help visually impaired children, specifically while at nursery or school.

Audio Feedback in Schools

We spoke to visual impairment education experts to discover the problems that young visually impaired children have while at nursery and school. These discussions uncovered a number of problems, mostly relating to play and social activities. We also received many suggestions of how sound from audio bracelets and other devices could be used to address these problems; for example, using sound to encourage children to try new activities. Please see our CHI 2017 paper (details at end) for more about what we learned from this study.

We developed three scenarios that represent the issues we learned about, with examples of how sound from audio bracelets could be used to help. An online survey for visual impairment experts investigated design issues relating to these scenarios and their use of audio feedback. One of our main findings from this survey is that sound should not just come from audio bracelets. Instead, sound should also come from places in the room and from other peoples’ locations in the room. We also learned that speech and familiar sounds (e.g., from objects) should be used to inform children about nearby places and activities.

Audible Beacons

Our findings from these studies led to the development of Audible Beacons, devices that can produce sound and be used for estimating proximity to people or places. They are essentially Bluetooth beacons that can be remotely controlled to produce audio feedback. A small form-factor is necessary so that the beacons can be worn by children (like an audio bracelet) or placed in the room (like a beacon).

Audible Beacons combine audio output with Bluetooth beacon capabilities. They can be worn like an audio bracelet or placed in the room like a beacon.

Audible Beacons combine audio output with Bluetooth beacon capabilities. They can be worn like an audio bracelet or placed in the room like a beacon.

The audio bracelet produced by the ABBI project (shown below) has full Audible Beacon capabilities: it has Bluetooth for remote control, it has beacon functionality, and it can synthesise audio on demand.

The ABBI audio bracelet. A 3D printed enclosure can be worn on a wrist strap or placed in a room.

The ABBI audio bracelet. A 3D printed enclosure can be worn on a wrist strap or placed in a room.

Publications

  • Audible Beacons and Wearables in Schools: Helping Young Visually Impaired Children Play and Move Independently. Euan Freeman, Graham Wilson, Stephen Brewster, Gabriel Baud-Bovy, Charlotte Magnusson, and Hector Caltenco. Proceedings of CHI 2017.
  • Automatically Adapting Home Lighting to Assist Visually Impaired Children. Euan Freeman, Graham Wilson, and Stephen Brewster. Proceedings of CHI 2016 Extended Abstracts.

Videos

The following video accompanies our CHI 2017 paper.

4/10/16 ICMI and NordiCHI

We’re going to be at ICMI in Tokyo and NordiCHI in Gothenburg later in the year, presenting research from the ABBI Project. Look our for our posters and demos and come and say hello!

At NordiCHI we’ll be presenting a poster:

  • Automatically Adapting Home Lighting to Assist Visually Impaired Children
    • Euan Freeman, Graham Wilson, and Stephen Brewster

And at ICMI we are presenting two demos:

  • Towards a Multimodal Adaptive Lighting System for Visually Impaired Children
    • Euan Freeman, Graham Wilson, and Stephen Brewster
  • Multimodal Affective Feedback: Combining Thermal, Vibrotactile, Audio and Visual Signals
    • Graham Wilson, Euan Freeman, and Stephen Brewster

10/11/15 SAVIE Meeting

On Friday (6/11/15) we were in Edinburgh for the SAVIE (Scottish Association for Visually Impaired Education) meeting. Steve, Graham and Euan spoke about the ABBI project, sharing some research findings and ideas for future work. We had a lot of interesting discussion with the other attendees about how wearable tech could be used to assist blind and visually impaired children. Our presentation slides from the meeting are available here. If you were at the meeting and would like to get in contact, please email stephen.brewster@glasgow.ac.uk

8/9/15 RNIB Techshare Europe

We will be attending the RNIB Techshare conference in Glasgow this week, to demo and talk about the ABBI project. Steve is giving a presentation about the ABBI project, while Graham and Euan will be giving interactive demos of the bracelet during the Tech Try sessions. The demos will showcase the bracelet and its capabilities and will demonstrate how the technology could be used to help blind and visually impaired people. We look forward to seeing you there!

ABBI: Audio Bracelet for Blind Interaction

Introduction

The ABBI (Audio Bracelet for Blind Interaction) Project aims at improving spatial cognition, mobility and social skills in blind children, through motion-sensitive “bracelets” on the wrists/ankles and the use of the auditory modality to convey spatial information (movement, position). It is an EU-funded collaboration with IIT, Lund University, University of Hamburg and Instituto David Chiossone Onlus. The core idea of the ABBI system is to improve spatial cognition abilities of visually-impaired individuals through the use of other sensory modalities (e.g. touch and hearing). To that end, the project will develop technologies and procedures to rehabilitate brain processes and functions involved in spatial cognition of children and adults with visual disabilities through natural audio-motor and, possibly, tactile-motor association. In particular, the idea is that the auditory modality could play the role that visual signals normally play in the development of sensorimotor skills, spatial cognition, navigation and social interaction skills. This approach is based on a renewed understanding of the role of vision and on the role that another sensory signal (e.g. audio) associated with the motor signal might have in helping the blind child to build a sense of space.

ABBI

The research being done at Glasgow University focuses on two aspects: 1) the design of suitable audio and haptic feedback and 2) exploring the potential of other wearable devices for providing information to the visually impaired. The Multimodal Interaction Group has an established history in designing non-visual feedback to support a variety of interfaces, including those for the visually impaired. For ABBI, our expertise will be applied to designing audio feedback designs that are clear, informative and enjoyable for users of various ages (from several months up to teenage years) and in environments with varying audio complexity.

With the proliferation of wearable devices, such as watches, glasses, arm bands etc. it is becoming easier to provide new sensors and feedback methods for the visually impaired in small and simple form factors. With greater access to information could come greater freedom and security. At Glasgow, we will look at expanding the information available to visually impaired users, including what information is most useful, how to convey it and how the user can control the amount and type of information that is presented to them.

Following the Route of a Moving Sound Source Through a Room

The ABBI (Audio Bracelet for Blind Interaction) device is designed for visually impaired and blind children to wear on the wrist and produce sound based on the movement of the arm through space. The primary function is to inform a child (or adult) about his/her own movements to aid spatial cognition rehabilitation. However, the device could also be worn by friends and family and be used to inform the visually impaired person of others’ movement in the environment. In this paper, we describe an initial experiment that measured how well blindfolded sighted individuals could track a moving sound source in 2D horizontal space and then walk the same route to the same end position. Six sounds, including natural sounds, abstract sounds, Earcons and speech, were compared to identify which type of sound produced more accurate route recreation. Our preliminary test suggests that all of the initial sounds facilitate recreation of 2D horizontal movement trajectories similarly well, although birdsong was problematic and speech and waves were more promising. This may mean that personalisation of ABBI sounds is possible while retaining their positive effects for rehabilitative support.

Use case diagram 1 Use case diagram 2Slide02

 

Supporting Accurate Reaching with Dynamic Audio Feedback

Blind children engage with their immediate environment much less than sighted children, particularly through self-initiated movement or exploration. Research has suggested that providing dynamic feedback about the environment and the child’s actions within/against it may help to encourage reaching activity and support spatial cognitive learning. This paper investigated whether the accuracy of peripersonal reaching can be improved by the use of dynamic sound from both the objects to reach for and the reaching hand itself (via a worn speaker). As a first step, we ran a study that tested the efficacy of static and dynamic audio feedback designs with blind and visually impaired young adults, to identify optimal feedback designs. The results showed that dynamic audio feedback helps to build connections and spatial links between the objects and the reaching hand and participants were able to reach for objects more accurately compared to constant (unchanging) feedback.

Girl sensory box Participant crop

 

 

Publications

  • Wilson, G. & Brewster, S. (2015) “Using Dynamic Audio Feedback to Support Peripersonal Reaching in Visually Impaired People”, to appear in Proceedings of ASSETS 2015, Oct 26-28, Lisbon, Portugal.
  • Magnusson, C., Caltenco, H., Finocchietti, S., Cappagli, G., Wilson, G. & Gori, M. (2015) “What Do You Like? Early Design Explorations of Sound and Haptic Preferences”, to appear in Proceedings of MobileHCI 2015 Posters, Aug 24-27, Copenhagen, Denmark.
  • Wilson, G., Brewster, S., Caltenco, H., Magnusson, C., Finocchietti, S., Baud-Bovy, G. & Gori, M. (2015) “Effects of Sound Type on Recreating the Trajectory of a Moving Source”, Proceedings of CHI 2015 Extended Abstracts, Apr 18-23, Seoul, South Korea.