Accompanying the rise of mobile and pervasive computing, applications now need to adapt to their surrounding environments and provide users with information in the environment in an easy and natural manner. In this paper we describe a user interface that integrates multimodal input on a handheld device with external gestures performed with real world artifacts. The described approach extends reference resolution based on speech, handwriting and gesture to that of real world objects that users may hold in their hands. We discuss the varied interaction channels available to users that arise from mixing and matching input modalities on the mobile device with actions performed in the environment. We also discuss the underlying components required in handling these extended multimodal interactions and present an implementation of our ideas in a demonstrator called the Mobile ShopAssist. This demonstrator is then used as the basis for a recent usability study that we describe on user interaction within mobile contexts.

Item Type:	Refereed Conference Paper
Research Division:	Information and Computing Sciences
Research Group:	Library and information studies
Research Field:	Human information interaction and retrieval
Objective Division:	Information and Communication Services
Objective Group:	Information systems, technologies and services
Objective Field:	Application software packages
UTAS Author:	Wasinger, R (Dr Rainer Wasinger)
ID Code:	90167
Year Published:	2005
Deposited By:	Information and Communication Technology
Deposited On:	2014-03-27
Last Modified:	2016-11-04
Downloads:	0