ZDNET’s key takeaways
- Onscreen awareness is a Siri AI feature.
- The feature lets you chat with Siri AI.
- Get quick responses to onscreen content.
One of the many cool capabilities baked into the new and improved Siri AI is on-screen awareness. Here, Siri can “see” what’s on your iPhone’s screen to provide information, answer questions, perform requests, and chat with you about the content.
As well: How to claim up to $95 from Apple’s Siri AI settlement today
I mostly used this feature on my iPhone with iOS 27 installed. However, onscreen awareness is also available on an iPad with iPadOS 27 and a Mac with MacOS 27. You can even use it on an Apple Watch with WatchOS 27, but for now the use of the feature on the watch is limited. Only a few built-in apps support it, such as Messages, Notifications, and Calendar. But more monitoring apps should start jumping on the bandwagon.
The downside here is that Siri only “sees” what is visible. When you open a long email, website or file, the AI doesn’t see anything above or below the content on the screen. Keep that in mind when you need help with something that goes beyond the visible screen.
Begin
To get started on your iPhone, first make sure you’ve updated to iOS 27. Just go to Settings, select General, and tap Software Updates. Select the option to upgrade to iOS 27 and let the new version download and install.
You will then need to sign up to use the Siri AI beta. After your phone reboots, go to Settings and tap on the option to try the Siri AI beta. You may have to wait a few hours or even a day to be registered for the public beta. Then you can use Siri AI.
Use Siri’s onscreen awareness
To try out the new onscreen awareness, navigate to a specific screen on your phone. Activate Siri by saying “Hey Siri” or “Siri”. Alternatively, press the side button. With Siri appearing on the screen, you can ask your question or make a request. Siri can look at your current screen and provide the help or information you need.
How to use Siri’s onscreen awareness
Now, here are five ways I like to use Siri AI’s onscreen awareness.
1. Ask Siri for directions to a specific location
I often receive emails about plays, conventions and other events that I have signed up for. Depending on the location, I usually need directions.
With Siri AI, I simply open the email with the address (or often just the name of the place) and ask for directions. In response, Siri confirms the address and then takes me to the Maps app with turn-by-turn directions. I can do the same with a text message, website, image or other screen that shows a location name or address.
2. Ask Siri for help with a screen on my phone
I sometimes need to tweak the settings on my iPhone, both for personal and professional reasons. Although I am familiar with most of the settings, there are always a few that I may not fully understand or know how to set. But here I can register Siri.
Also: Want to try iOS 27’s upgraded Siri? Be prepared for usage limits
With the appropriate settings screen open, I can ask Siri, “Explain this screen and give me advice on how to set it up.” In return, Siri gives me a rundown of the settings on the screen and helps me determine how to set it. I can do this with any screen on my iPhone, though it’s especially useful when I need to figure out what to do with a particular setting.
Screenshot by Lance Whitney/ZDNET3. Ask Siri to share a photo or other item
When my wife and I are out and about taking pictures, she often asks me to send her a picture I just took. In the past, I had to open the Photos app or the Messages app, attach the photo, and then send it to her text messages. But now I can just ask Siri to do it.
With the photo on the screen, I say: “Siri, write this photo to my wife.” Siri confirms the action, and then shares the photo with her via text message. Of course, I can do this with any file, screen or other item on my phone.
Screenshot by Lance Whitney/ZDNET4. Add items to a list
I sometimes surf the internet for different recipes. If I find one I like, I’ll buy the necessary ingredients the next time I’m at the grocery store. That’s why I put them on a note or list.
As an example, I was looking for a recipe for Key Lime Pie. With the ingredients on the screen, I asked Siri to add them to my shopping list, which she did quickly and easily.
Screenshot by Lance Whitney/ZDNET5. Edit a photo or other file
As an avid photographer, I always take pictures on my phone. Sometimes a shot doesn’t look good. Sure, I can always tweak the image manually using the various tools in the Photos app. But often it’s faster to let Siri do the hard work and then review the results.
Also: After your iOS 27 upgrade, do these 5 things first
Here I open a photo on Photos and tell Siri, “Enlarge this photo to make it brighter.” Siri then makes the necessary adjustments automatically. Once I’m satisfied with the results, I can then save the revised version.
Screenshot by Lance Whitney/ZDNETThese are just five ways I’ve taken advantage of Siri’s onscreen awareness. There are many other questions, requests and tasks where Siri can come in handy. With the new AI flavor on your iPhone (or iPad or Mac), try using Siri to help you with a screen on your device and see how it works.
See full bio