File(s) under permanent embargo
Comparative analysis of intelligent personal agent performance
conference contribution
posted on 2023-05-23, 14:27 authored by David HerbertDavid Herbert, Byeong KangByeong KangIntelligent Personal Assistant (IPA) devices such as Google Home and Amazon Echo have become commodity hardware and are well-known in the public domain. Leveraging these devices as speech-based interfaces to bespoke conversation agent (CA) systems in vocabulary-specific domains exposes their underlying Automatic Speech Recognition (ASR) transcription error rates, which are usually hidden behind a probability matching of utterance to intent (slot filling). We present an evaluation of the two aforementioned IPA’s isolated word and phrasal recognition rates together with an improvement scheme associated with a Contextual Multiple Classification Ripple Down Rules (C-MCRDR) CA knowledge-base system (KBS). When measuring isolated-word word error rates (WER) for a human speaker, Google Home achieved an average WER of 0.082 compared to 0.276 for Amazon Echo. Computer-generated utterances unsurprisingly had much poorer recognition rates, with WER for Google Home and Amazon Echo of 0.155 and 0.502 respectively. For phrasal tests, Google Home had an average WER of 0.066 in comparison to the Amazon Echo WER of 0.242 when processing human-sourced sentences. We applied a rule-based transcription error-correcting scheme for isolated words and achieved correct recognition rates of 100% for the Google Home in five of the isolated word data sets, and across all isolated words datasets we improved the initial average WER of 0.082 to 0.0153, a significant decrease of 81.34%.
History
Publication title
Lecture Notes in Computer ScienceVolume
11669Editors
K Ohara and Q BaiPagination
127-141ISBN
9783030306380Department/School
School of Information and Communication TechnologyPublisher
Springer NaturePlace of publication
SwitzerlandEvent title
Knowledge Management and Acquisition for Intelligent Systems - 16th Pacific Rim Knowledge Acquisition WorkshopEvent Venue
Cuvu, FijiDate of Event (Start Date)
2019-08-26Date of Event (End Date)
2019-08-27Rights statement
Copyright 2019 Springer NatureRepository Status
- Restricted