Mid-air Interaction in Cars

Mid-air gestures in cars are being used by an increasing number of drivers on the road. Usability concerns mean good feedback is important, but a balance needs to be found between supporting interaction and reducing distraction in an already demanding environment. Visual feedback is most commonly used, but takes visual attention away from driving. We investigated novel non-visual alternatives to support the driver during mid-air gesture interaction: Cutaneous Push, Peripheral Lights, and Ultrasound feedback. These modalities lack the expressive capabilities of high resolution screens, but are intended to allow drivers to focus on the driving task.

Cutaneous Push

A new form of haptic feedback — Cutaneous Push — was defined. Six solenoids were embedded along the rim of the steering wheel, creating three bumps under each palm. We investigated the efficacy of novel static and dynamic Cutaneous Push patterns, and their impact on driving performance. In simulated driving studies, the cutaneous patterns were tested. The results showed pattern identification rates of up to 81.3% for static patterns and 73.5% for dynamic patterns and 100% recognition of directional cues. Cutaneous Push notifications did not impact driving behaviour nor workload and showed very high user acceptance. Cutaneous Push patterns have the potential to make driving safer by providing non-visual and instantaneous messages, for example to indicate an approaching cyclist or obstacle.

Publications

  • Di Campli San Vito P., Brewster S., Pollick F., Brown E., Skypchuk L. and Mouzakitis A.: Haptic Navigation Cues on the Steering Wheel. In Proceedings of the 37th Annual ACM Conference on Human Factors in Computing Systems ât CHI'19, ACM Press, 2019.  Link.
  • Shakeri, G., Brewster, S., Williamson, J. and Ng, A.: Evaluating Haptic Feedback on a Steering Wheel in a Simulated Driving Scenario. In Proceedings of CHI '16 Extended Abstracts, ACM Press, 2016.
  • Shakeri, G., Ng, A., Williamson, J. & Brewster, S.: Evaluation of Haptic Patterns on a Steering Wheel. In Proceedings of AutomotiveUI '16, ACM Press, 2016.

Uni- and Bimodal Feedback for Mid-air Gestures

We looked at novel uni- and bimodal feedback combinations of Visual, Auditory, Cutaneous Push, and Peripheral Lights for mid-air gestures and found that non-visual feedback modalities, especially when combined bimodally, offered just as much support for interaction without negatively affecting driving performance, visual attention and cognitive demand. These results provide compelling support for using non-visual feedback from in-car systems, supporting input whilst letting drivers focus on driving.

Publications

  • Shakeri, G., Williamson, J. H. and Brewster, S.: Bimodal Feedback for In-Car Mid-Air Gesture Interaction. In Proceedings of 19th ACM International Conference on Multimodal Interaction (ICMI’17), ACM Press, 2017.
  • Shakeri, G., Williamson, J. H. and Brewster, S.: Novel Multimodal Feedback Techniques for In-Car Mid-AirGesture Interaction. In Proceedings of the 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutoUI '17, ACM Press, 2017.
  • Freeman, E., Vo, D.-B., Wilson, G., Shakeri, G. and Brewster, S.: Towards Mid-Air Haptic Widgets. In CHI 2016 Workshop on Mid-Air Haptics and Displays: Systems for Un-instrumented Mid-Air Interactions, 2016.  Link.

Ultrasound for Mid-air Gestures

We investigated the above bimodal combinations as well as uni- and bimodal Ultrasound feedback during the Lane Change Task to assess the impact of gesturing and feedback modality on car control during more challenging driving. The results of these studies suggest that Visual and Ultrasound feedback are not appropriate for in-car usage, unless combined multimodally. If Ultrasound is used uni-modally it is more useful in a binary scenario. Findings suggest that multimodal feedback significantly reduces eyes-off-the-road time compared to Visual feedback without compromising driving performance or perceived user workload, thus it can potentially reduce crash risks. Novel design recommendations for providing feedback during mid-air gesture interaction in cars are provided, informed by the experiment findings.

Publications

  •  Shakeri, G.: Multimodal feedback for mid-air gestures when driving. In University of Glasgow, 2020.  Link.
  • Shakeri, G., Freeman, E., Frier, W., Iodice, M., Long, B., Georgiou, O. and Andersson, C.: Three-in-one: Levitation, Parametric Audio, and Mid-Air Haptic Feedback. In Proceedings of the 37th Annual ACM Conference on Human Factors in Computing Systems at CHI'19 Interactivity, ACM Press, 2019.  Link.
  • Shakeri, G., Williamson, J. H. and Brewster, S.: May the Force Be with You: Ultrasound Haptic Feedback for Mid-Air Gesture Interaction in Cars. In Proceedings of the 10th International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications - AutoUI '18, ACM Press, 2018.