Remote Haptics and Mid-Air Gestures

Introduction

Interaction with mobile devices, including phones and wearables, is often limited by their small screens. A limited amount of space is available for displaying content and giving users feedback about their interactions, which can make introducing new interaction modalities difficult. Novel interactions, such as speech and in-air gestures, require good feedback but mobile devices are unable to give this feedback. We can overcome this problem by using alternative types of feedback, instead. For example, tactile feedback can be given about interactions with mobile phones, keeping the limited screen space available for showing other content.

In-air gesture interaction is a novel way of interacting with mobile phones and recent products have been released with these capabilities. However, limited feedback is given about these interactions, which makes them difficult to use. We investigated ways of giving tactile feedback about in-air gestures, which means feedback can be given without affecting on-screen content. This project was a three-year PhD studentship partly funded by Nokia Research Centre in Finland.

Summary

We investigated tactile feedback for in-air gesture interfaces. Giving such feedback is challenging for one obvious reason: users do not physically contact in-air gesture systems, meaning novel solutions are required to present tactile sensations. In this project we compared ultrasound haptics (see UltraHaptics) to tactile feedback from wearable devices, like smart-watches and activity trackers. We also investigated feedback design, to see how the complexity of feedback affected interaction. For more information see here or our ICMI 2014 paper on tactile feedback for above-device interfaces [1].

Publications

[1] E. Freeman, S. Brewster, and V. Lantz: Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions . In Proceedings of icmi ’14 (2014).
[Bibtex]
@inproceedings{Freeman2014,
author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
booktitle = {Proceedings of ICMI '14},
publisher = {ACM Press},
title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
url = {http://research.euanfreeman.co.uk/papers/ICMI\_2014.pdf},
year = {2014}
}

Ultrasonic Haptic Feedback

Introduction

Ultrasonic haptic feedback involves the creation of focused air pressure waves from an array of ultrasound transducers. These are reflected off the skin to create tactile sensations without being in direct contact with an actuator. It is potentially useful for gestural interfaces, such as those that utilise body position, hand movements or finger gestures for input, as these interfaces suffer from a lack of tactile feedback. Ultrasonic haptic feedback is based on the principle of acoustic radiation pressure, where a phased array of ultrasonic transducers creates a beam focused at a point in 3D space. The narrow focus of the beam is determined by the wavelength of the ultrasound (e.g., 8.6mm at 40kHz) and the ultrasound is modulated with a lower frequency, such as 200Hz, so as to be perceivable by mechanoreceptors in the skin. When a focal point is reflected off the skin, the force produced creates a localised tactile sensation akin to “air”, “breeze” or “wind”. Focal points can be produced at high spatial resolution and moved rapidly in the space above the array. The applications of this novel form of haptic feedback in HCI have been steadily increasing. It has the advantage of providing tactile feedback in mid-air, without the user holding a device or having one attached. This research has been conducted in collaboration with Bristol University and UltraHaptics.

Array from topArray close upArrayUser

 

Localisation and Perception of Motion

Research on ultrasonic haptics has tested the detection or differentiation of one or multiple points of feedback , the two-point visual-tactile threshold and presented interaction prototypes with limited user studies. Research is needed on what spatial or temporal parameters influence localisation and perception of motion. Two lab experiments were carried out: the first tested localisation of static feedback on the hand to determine spatial resolution for ultrasonic haptics. The second tested the perception of motion across two axes on the hand, to identify which characteristics of feedback (distance, duration, number of stimulated positions and movement direction) elicit convincing sensations of motion.

Experimental Setup with hand small ArrayAxes

Publications

  • E. Freeman, S. Brewster, and V. Lantz: Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions . In Proceedings of icmi ’14 (2014). [DOI] Website
  • G. Wilson, T. Carter, S. Subramanian, and S. Brewster: Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent Motion. In Proceedings of acm chi 2014, pp. 1133-1142 (2014). [DOI] Website
  • G. Wilson, E. Freeman, T. Carter, S. A. Seah, S. Brewster, and S. Subramanian: Ultrasonic Haptic Feedback for Gestural Interfaces Using a Moveable Hand-Mounted Array . In Proceedings of world haptics 2013 (2013).

Around-Device Interaction

Gesture interaction in the space around small devices is one way of overcoming limitations of touchscreen input. Gestures performed near a device allow users to interact more casually, for example: quickly dismissing incoming phone calls when busy; or checking for new text messages without having to pick the phone up. This style of interaction also allows input when touch is not suitable, for example: when hands are wet and messy from preparing food but you need to quickly check a recipe; or when wearing gloves outside in the cold. Finally, around-device interaction can also avoid problems caused by finger occlusion, where the interacting fingers cover parts of the display when providing touchscreen input. Wearable devices, like smart-watches, would especially benefit from less occlusion as screen sizes are much smaller than on phones.

Gesture Feedback

Good feedback is important to help users gesture effectively, although giving good feedback from small devices can be difficult. Visual feedback is not always ideal because it takes away already limited screen space. On-screen cues may also be difficult to notice from a short distance away. Instead, we look at how other output types can be used to give useful feedback. Audio and tactile feedback in particular are promising output modalities for gesture feedback.

Audio and tactile feedback are not without their own difficulties, however, and our research seeks to address these. One of the challenges of trying to deliver tactile feedback in a gesture interface is that users are not always in contact with the device they are interacting with; instead, they may be gesturing from several feet away or from across the room. Non-contact forms of tactile feedback (such as ultrasound haptics [1]) and distal methods of tactile feedback (such as wearable devices) should also be considered. We looked at these types of tactile feedback in our ICMI 2014 paper [2].

Gesture Design

We’ve also looked at gesture design for around-device interfaces [3]. In our paper we give recommendations for designing usable and acceptable mid-air gestures for mobile phones. We also present a set of evaluated gestures. In that work we also looked at the social acceptability of gestures, with similar findings to our earlier work on gesture acceptability [4].

References

[1] G. Wilson, T. Carter, S. Subramanian, and S. Brewster: Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent Motion. In Proceedings of acm chi 2014, pp. 1133-1142 (2014).
[Bibtex]
@article{Wilson2014,
author = {Wilson, Graham and Carter, Tom and Subramanian, Sriram and Brewster, Stephen},
journal = {Proceedings of ACM CHI 2014},
keywords = {Ultrasound,haptic feedback,localisation,perception},
pages = {1133--1142},
title = {{Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent Motion}},
url = {http://dl.acm.org/citation.cfm?id=2557033},
year = {2014}
}
[2] E. Freeman, S. Brewster, and V. Lantz: Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions . In Proceedings of icmi ’14 (2014).
[Bibtex]
@inproceedings{Freeman2014,
author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
booktitle = {Proceedings of ICMI '14},
publisher = {ACM Press},
title = {{Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions}},
url = {http://research.euanfreeman.co.uk/papers/ICMI\_2014.pdf},
year = {2014}
}
[3] [doi] E. Freeman, S. Brewster, and V. Lantz: Towards Usable and Acceptable Above-Device Interactions . In Proceedings of mobile hci ’14 posters (2014).
[Bibtex]
@inproceedings{Freeman2014a,
author = {Freeman, Euan and Brewster, Stephen and Lantz, Vuokko},
booktitle = {Proceedings of Mobile HCI '14 Posters},
doi = {10.1145/2628363.2634215},
publisher = {ACM Press},
title = {{Towards Usable and Acceptable Above-Device Interactions}},
url = {http://research.euanfreeman.co.uk/papers/MobileHCI\_2014\_Poster.pdf},
year = {2014}
}
[4] J. Rico and S. Brewster: Usable gestures for mobile interfaces: evaluating social acceptability . In Proceedings of the 28th international conference on \ldots (2010).
[Bibtex]
@inproceedings{Rico2010,
author = {Rico, Julie and Brewster, Stephen},
booktitle = {Proceedings of the 28th international conference on \ldots},
file = {:C$\backslash$:/Users/user/AppData/Local/Mendeley Ltd./Mendeley Desktop/Downloaded/Rico, Brewster - 2010 - Usable gestures for mobile interfaces evaluating social acceptability.pdf:pdf},
isbn = {9781605589299},
title = {{Usable gestures for mobile interfaces: evaluating social acceptability}},
url = {http://dl.acm.org/citation.cfm?id=1753458},
year = {2010}
}