eCite Digital Repository

Comparative analysis of intelligent personal agent performance


Herbert, D and Kang, B, Comparative analysis of intelligent personal agent performance, Lecture Notes in Computer Science, 26-27 August 2019, Cuvu, Fiji, pp. 127-141. ISBN 9783030306380 (2019) [Refereed Conference Paper]

Copyright Statement

Copyright 2019 Springer Nature

Official URL:

DOI: doi:10.1007/978-3-030-30639-7_11


Intelligent Personal Assistant (IPA) devices such as Google Home and Amazon Echo have become commodity hardware and are well-known in the public domain. Leveraging these devices as speech-based interfaces to bespoke conversation agent (CA) systems in vocabulary-specific domains exposes their underlying Automatic Speech Recognition (ASR) transcription error rates, which are usually hidden behind a probability matching of utterance to intent (slot filling). We present an evaluation of the two aforementioned IPA’s isolated word and phrasal recognition rates together with an improvement scheme associated with a Contextual Multiple Classification Ripple Down Rules (C-MCRDR) CA knowledge-base system (KBS). When measuring isolated-word word error rates (WER) for a human speaker, Google Home achieved an average WER of 0.082 compared to 0.276 for Amazon Echo. Computer-generated utterances unsurprisingly had much poorer recognition rates, with WER for Google Home and Amazon Echo of 0.155 and 0.502 respectively. For phrasal tests, Google Home had an average WER of 0.066 in comparison to the Amazon Echo WER of 0.242 when processing human-sourced sentences. We applied a rule-based transcription error-correcting scheme for isolated words and achieved correct recognition rates of 100% for the Google Home in five of the isolated word data sets, and across all isolated words datasets we improved the initial average WER of 0.082 to 0.0153, a significant decrease of 81.34%.

Item Details

Item Type:Refereed Conference Paper
Keywords:MCRDR, intelligent personal assistant, knowledge-base systems, conversational agent
Research Division:Information and Computing Sciences
Research Group:Information systems
Research Field:Information security management
Objective Division:Information and Communication Services
Objective Group:Information systems, technologies and services
Objective Field:Information systems, technologies and services not elsewhere classified
UTAS Author:Herbert, D (Mr David Herbert)
UTAS Author:Kang, B (Professor Byeong Kang)
ID Code:137276
Year Published:2019
Deposited By:Information and Communication Technology
Deposited On:2020-02-06
Last Modified:2020-05-18

Repository Staff Only: item control page