Apple's (AAPL) - Get Report Siri, for its part, delivered the largest improvement relative to the last time it was tested. And all four of the assistants that were tested on smart speakers did a meaningfully better job relative to their last go-around.
Loup's Smart Speaker Test
In the 2018 edition of its Smart Speaker IQ Test, research firm Loup Ventures asked smart speakers powered by Google Assistant, Siri, Amazon.com's (AMZN) - Get Report Alexa and Microsoft's (MSFT) - Get Report Cortana a common set of 800 questions. The questions were split into five categories: Local information, commerce, navigation, information in general and commands (for example, alarms or reminders).
Google Assistant, which was tested on a Google Home speaker, managed to answer 87.9% of the questions correctly, up from 81.1% a year ago. Siri, which was naturally tested on Apple's HomePod speaker, got 74.6% of the questions right, up from about 52% in February, when the HomePod first became available. Loup does note that Siri benefited from the fact that some Siri requests were passed on to a paired iOS device that handled them, rather than directly handled by the HomePod.
Alexa (tested on an Amazon Echo) saw its accuracy improve to 72.5% from 63.8%, and Cortana (tested on Harman Kardon's Invoke speaker) improved to 63.4% from 56.4%. Google Assistant was the only assistant to properly understand all 800 of the questions, but the others were able to understand 99% or more.
Of the five categories created by Loup, Google Assistant came first all of them save for commands, where it finished second to Siri. Interestingly, Google was well ahead of all rivals (Alexa included) in the commerce category, where it had an 86% accuracy rate to Siri's 56%, Alexa's 52% and Cortana's 31%.
Meanwhile, both Google Assistant and Siri were well ahead of their peers in the local info and navigation categories, and Google Assistant and Alexa were well ahead in the broader information category.
It's worth noting that Google Assistant and Siri also claimed the top two spots in Loup's July test of voice assistants running on smartphones. In that test, Google Assistant and Siri accurately fielded 85.5% and 78.5% of questions, respectively, while Alexa and Cortana did so for 61.4% and 52.4%.
Loup's data provides some context for why Google Assistant has been gaining ground on Alexa in the smart speaker wars, in spite of Alexa's head start and Amazon's ability to promote Echo devices to its giant customer base. Google's arguably unmatched AI/machine learning abilities have bolstered Assistant, and so has integration with Google Search and its massive Knowledge Graph.
One could add here that Google's ability to integrate Assistant with other Google services becomes an even bigger advantage on devices featuring a display, whether a mobile device or a smart display such as the Google Home Hub. On such hardware, Google can respond to a voice command or query by showing content such as search results, Knowledge Graph data, YouTube videos or images from a user's Google Photos collection.
Another takeaway from Loup's data is that Siri, which has received a lot of criticism over the years even from staunch Apple fans, does seem to be meaningfully improving. Apple, which earlier this week named AI chief and former Google exec John Giannandrea an SVP, has made no secret of the fact that it considers machine learning investments a major priority.
And looking broadly at Google and Apple's performance in both Loup's smart speaker test and the July smartphone assistant test, the numbers suggest the widespread use of Assistant and Siri on smartphones is -- by giving both algorithms and human decision-makers more data and experience to work with -- helping improve their competitiveness on mobile devices and smart speakers alike. On this count, it probably isn't a coincidence that Google Assistant and Siri easily beat Alexa and Cortana in the two voice assistant categories (local info and navigation) that are bound to see a disproportionate number of smartphone queries.
It can't be forgotten that in spite of growing rapidly the last two years, the global smart speaker installed base is (thanks to Google and Apple's control of the world's two dominant mobile platforms) still dwarfed by the installed base of smartphones with Google Assistant or Siri baked in. While Amazon has some strengths of its own in the voice assistant wars and is certainly investing aggressively to strengthen Alexa, the competitive advantages provided by Google Assistant and Siri's mobile popularity won't be going away anytime soon.
(Editor's Pick. Originally published Dec. 23.)