Apple Watch Series 12 and Ultra 4 will bring two surprisingly futuristic new tricks: Live Rewind and Siri Recap, a pair of Audio Intelligence features that turn the smartwatch into an always-listening AI device.
Apple markets Live Rewind as a way to rewind the world around you. Actually, it can only turn the previous 15 seconds of a conversation into a text snippet. Still, it potentially could help people remember something that’s just been said.
Siri Recap, on the other hand, quietly takes notes during your conversations, then uses Apple Intelligence to create concise summaries you can refer to later. The goal is to help people stay focused on the moment.
The new Audio Intelligence features seem quite useful, but they make privacy advocates nervous. Fortunately, Apple really locked them down, hoping to prevent the Apple Watch features from being abused.
The benefits of Apple Watch Live Rewind
The new AI-powered Apple Watch features will arrive at a time when people are already nervous and/or angry about being monitored without their knowledge. From Ray-Ban Meta AI Glasses (aka “pervert glasses“) to Flock license plate cameras, a backlash is brewing.
Now Apple, which prides itself on being the most privacy-focused big tech company around, is turning its smartwatch into a device capable of capturing audio 24/7 for AI-powered quick replays and summaries. And that sounds a little creepy at first.
During the Apple Watch portion of the Surprise and Shine event on Wednesday, Apple demoed using Live Rewind at a restaurant. A server quickly rattled off a lengthy list of daily specials, and a customer then used their Apple Watch to turn that rush of words into text they could easily scroll through to make a decision (see image above).
“This is great to recall a book recommendation, a colleague’s brilliant idea or something you didn’t hear the first time,” said Ron Huang, Apple VP of sensing and connectivity.
Users can activate Live Rewind with a double press of the Digital Crown on a compatible Apple Watch to show the previous 15 seconds of a conversation as text.
Summarize your life with Siri Recap
Image: Apple
Siri Recap takes Apple’s new ambient-listening approach a step further. When enabled, Apple Watch Series 12 or Apple Watch Ultra 4 can listen during conversations and use Apple Intelligence to create a high-level summary, complete with a title and key points.
The idea is to let you stay focused on a meeting, appointment or conversation instead of worrying about taking notes.
“It uses ambient listening to take high-level notes of conversations like takeaways from a team meeting or parent-teacher conference,” said Huang.
Audio Intelligence features alarm privacy hawks
Because the user activates Live Rewind after the words it transcribes have already been spoken, the Apple Watch must constantly monitor everything said nearby.
The same goes for Siri Recap. In order to create the summaries, the watch must be listening.
It’s no surprise that privacy advocates are raising red flags about the Audio Intelligence features. But Apple built in quite a few privacy safeguards.
Legal experts quoted in a Bloomberg report Friday acknowledged that Apple is sailing into uncharted waters with Live Rewind and Siri Recap. Many expressed concerns about the legality of using the new features.
“We see a lot of circumstances where the technology is getting ahead of the law,” Colin Zick of Foley Hoag LLP told Bloomberg. “My advice would be, treat this as if it was going to be treated like an interception under the wiretapping statute, and therefore, ask permission.”
Apple locked down Live Rewind to prevent abuse
First off, the Apple Watch wearer must activate these features in Settings — they are off by default. So it’s up to the user to decide whether to use them.
For Live Rewind, the user cannot access the temporary audio recording, only a text transcription. And Apple can never access it.
The watch’s S11 chip includes a hardware-isolated Secure Exclave, which holds the temporary audio buffer. When Live Rewind is activated, the relevant audio is securely transferred to the Secure Exclave on the paired iPhone for transcription. The raw audio is then deleted. The process happens entirely on the Apple Watch and iPhone, not a remote server.
Plus, Apple also built a privacy warning into Live Rewind. Activating the feature produces an audible chime, even when the Apple Watch is in silent mode or connected to headphones. The display also shows an animation and microphone indicator so people nearby can see the feature is active.
Siri Recap also takes privacy into account
The process is similar for Siri Recap. Apple says it doesn’t create a permanent audio recording. The user can’t even get a full transcript of the conversation, only the summary.
The raw audio gets transferred from the Apple Watch’s Secure Enclave to be processed into bullet points inside the iPhone’s Secure Exclave. Then it’s deleted. The audio remains inaccessible to the user and Apple.
Neither of these Audio Intelligence features names the people involved in conversations. Siri Recap uses “Speaker A” and “Speaker B,” though the actual recap may contain names if they show up in the conversation. Live Rewind doesn’t do attribution at all.
The user can save Siri Recap summaries in the Siri app, but they automatically disappear after seven days unless the user chooses to keep them.
An example of Apple Watch Live Rewind and Siri Recap privacy
Let’s consider an example of someone trying to abuse Live Rewind and Siri Recap to see how the privacy protections work.
Cousin Daryl hears Cousin Larry call Aunt Hildegard “an old besom.” Daryl uses his Apple Watch to capture the phrase and goes off to tattle to their aunt.
But Daryl doesn’t have a recording of Larry’s voice, or anything that labels the insult as coming from Larry. All he has is some text on his Apple Watch screen, which Daryl could have easily faked. Not very convincing.
Siri Recap wouldn’t help Daryl bust Larry either. All it might say is that someone called Speaker A was criticizing Aunt Hildegard, but there’s no way to know who. That person might even be Daryl himself.
Requires the latest hardware
Apple promises that Siri Recap and Live Rewind arrive in beta later this year. But take note: They’ll only work on Apple Watch Series 12 or Apple Watch Ultra 4, not earlier models.
Plus, an iPhone that supports Apple Intelligence must be paired with the Apple Watch for either of these Audio Intelligence features to be available.
See also: Apple Watch Series 12 is the upgrade long-term users have been waiting for
Ed Hardy has been writing full-time about tech for 26 years, and using it for much longer than that. His intro to Apple was a Macintosh SE/30 (which he still has), but now he uses a 13-inch iPad Pro as his primary computer.
That’s because he’s a “tablet first” type of guy. Rather than use a Macbook, he connects a keyboard case to the iPad. And instead of a desktop Mac, he connects his tablet to a 27-inch display and full-size keyboard. (So don’t try to tell him that everyone has to use a Mac to be productive.)
Before coming to Cult of Mac, Ed wrote for NotebookReview, TabletPCReview and Brighthand, as well as other sites.
One response to “Apple Watch Live Rewind and Siri Recap sound creepy, but privacy is built in”
I do think that the chime should be every 15 seconds or so while it is recording.