Every Step You Take (I’ll Be Watching You): Virtual Assistant Patents Raise Privacy ConcernsMay 26, 2022

An Intelligent Virtual Assistant (IVA) is an artificial intelligence system that emulates a human person in terms of intelligence, understanding, response, and adaptability. An IVA differs from traditional interactive voice response systems largely in its use of machine learning, which allows an IVA to predict user actions. The global IVA market is predicted to reach about $45 billion USD by 2027, expanding at a Compound Annual Growth Rate (CAGR) of 34%. A large portion of the IVA market is owned by Amazon and Google, although Apple has indicated its intent to launch a desktop IVA product to compete with Amazon’s Echo.

Virtual assistant trends indicate a growing desire by users to interact on a voice-first basis, i.e., interact with a device primarily by voice. Voiceprint recognition and pre-wake audio processing are therefore central to the next generation of IVAs. Big tech patent filings over the last several years demonstrate substantial advancements in IVA technology, while also raising significant privacy concerns.

In 2018 Amazon was granted a patent directed to “voice-based determination of physical and emotional characteristics of users.” According to the patent, the IVA will be able to “determine one or more voice features based at least in part on the speech or voice input from a user” such as “a gender of the user, an age or age range of the user, an ethnic origin or language accent of the user, an emotion of the user, a background noise of the environment in which the user is located, and other voice features.” When the IVA identifies a physical or emotional characteristic, it will generate a tag for that characteristic and link the tag to a data file of the voice input. The tags based on physical or emotional characteristics will allow the IVA to suggest targeted content to a user. For example, if the IVA detects that a user sounds bored, the IVA may query “are you in the mood for a movie?” Ultimately, this would permit companies to deliver targeted advertisements based on an individual’s mood at any given time.

In 2019, Amazon was also issued a patent directed to “pre-wake word speech processing.” A “wake word” is the waking command that activates the IVA, for example, “Alexa” or “Hey Alexa.” Pre-wake word activation of the IVA would in theory allow a user to make a request using more natural language, such as “play some music Alexa” versus “Alexa, play some music.” However, this feature requires more extensive audio processing: per Amazon’s 2019 patent, “[a]s the device detects audio it may process the audio (either before or after the audio is stored in the buffer) to determine if the audio includes a wake word. The device may continue to do so until it detects a wake word in the received audio.” Put another way, Echo would store and process all audio it hears and conduct an ex post facto assessment of whether that audio included a command.

Apple has made significant steps to develop its own IVA products and systems. For example, in February 2022 Apple filed a first patent application for a “voice-controlled electronic device” and a second application directed to a “desktop electronic device,” ostensibly to better compete with Amazon Echo and Google Assistant. Building on its prior speech-to-text systems (e.g., this patent for methods of inferring user intent from speech), Apple more recently filed a patent application directed to improved “methods and user interfaces for voice-based control of electronic devices.”

Although these developments in IVA voice recognition and analysis will improve user experience, they also introduce a number of potential privacy concerns. Pre-wake word speech processing may permit the IVA to analyze and store much more audio than intended by the user. It remains unclear exactly how much data is collected and stored via smart home devices, and for how long tech giants are retaining that data. Further, although the collection and use of voiceprint data have been anticipated, the implementation of targeted advertisements based specifically on emotional or physical conditions represents an unprecedented degree of commercializing personal information. As the technology supporting digital assistants continues to improve, lawmakers need to adapt existing privacy laws accordingly, and consumers should be aware of the extent to which their data are being used.

Sarah M.D. Luth is an Intellectual Property Attorney in the MVS Biotechnology & Chemical Practice Group. To learn more, visit our MVS website , or contact Sarah directly via email .

← Return to Filewrapper

Stay in Touch

Receive the latest news and updates from us and our attorneys.

Sign Up