The Google Assistant's new artificial intelligence capabilities are groundbreaking, but they're also terrifying to some. 

During the Alphabet Inc.-owned  (GOOGL) - Get Report Google's annual developer conference on Tuesday, CEO Sundar Pichai led a demonstration of Duplex, its AI technology that can carry out natural conversations to complete tasks for users such as making appointments and reservations.

The crowd applauded as it heard a human-sounding call Google Assistant placed to make an appointments at both a hair salon and a restaurant, complete with human-sounding "um's" and "ah's". In both cases, the AI didn't identify that it was not human and the callers didn't appear to suspect they were talking to a computer, alarming some observers.

The growing market for smart speakers and other devices equipped with digital assistants has become an important way for tech giants such as Amazon.com Inc. (AMZN) - Get Report , Apple (AAPL) - Get Report and Google to enmesh consumers into their ecosystem of products, apps and services, Gartner analyst Werner Goertz said.

Image placeholder title

Goertz, who covers personal technologies, said that Amazon's Alexa, Google Assistant's main competitor, is on the path to ubiquity with Amazon's vast retail distribution and early success with smart home technology.Amazon's share of the smart speaker market stood at about 50% in the last quarter of last year, according to the research firm Strategy Analytics, whileGoogle increased its market share to over 35%.

The potential for digital assistants to listen and record conversations without users knowing it has created concerns for some in the past, and Google's technology adds a new fear to the mix. Goertz said that creating authentic-sounding speech could be dangerous for the humans on the other end who might not know they're speaking to an AI bot.

"It will be misused and there is potential for unethical or even criminal activity," Goertz said, likening the dilemma to the difficulty users have had discerning between real media reports and "fake news" on social media.

In response to questions about the Duplex demo from TheStreet, a Google spokesperson sent the following statement:

"We understand and value the discussion around Google Duplex -- as we've said from the beginning, transparency in the technology is important. We are designing this feature with disclosure built-in, and we'll make sure the system is appropriately identified. What we showed at I/O was an early technology demo, and we look forward to incorporating feedback as we develop this into a product.

IBISWorld tech industry analyst Dylan Miller said that in addition to admiring the technology, he felt alarmed by how human the speech sounded and saw his "fair share of allusions to 'The Terminator,'" and its merciless humanoid robots.  

"In tech, with the good comes the bad," Miller said. "While our scheduling and making mundane appointments may be automated to save more time, there's a tradeoff with that as well."

While the reactions to the new technology have been mixed, IHS Markit senior analyst Paul Erickson said consumers will not necessarily be turned off to using smart speakers and digital voice assistants.

"Once people get over the initial discomfort about how realistic AI can get, I think it will impress consumers -- or at least surprise consumers -- how far [they] can come," he said.

Alphabet, Amazon and Apple are holdings in Jim Cramer's Action Alerts PLUS Charitable Trust Portfolio. Want to be alerted before Cramer buys or sells GOOGL, AMZN or AAPL? Learn more now.