Alongside seeding iOS 18.2 beta 2 today, Apple released a new App Intents API to prepare developers for Siri onscreen awareness. This will allow the virtual assistant and Apple Intelligence to understand a third-party app’s information on a user’s display.
Apple is making the new API available today to give developers ample time to prepare their apps for Siri’s new Apple Intelligence features intended to launch early next year.
Per Apple’s documentation, the new API allows third-party apps to send content currently on the user’s display to Siri and Apple Intelligence. This will allow the virtual assistant to take action on the displayed information per the user’s request. Siri onscreen awareness and the App Intents API also support sending onscreen data to a third-party service if explicitly asked by the user.
Discover new horizons, always connected with eSIM
Travel the world stress and hassle-free with the best eSIM service available. Enjoy unlimited data, 5G speeds, and global coverage for affordable prices with Holafly. And, enjoy an exclusive 5% discount.
When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like “Hey Siri, what’s this document about?”
Right now, the App Intents API supports a limited set of onscreen data domains. This includes Browser, Document reader, File management, Mail, Photos, Presentations, Spreadsheets, and Word Processor. For example, a user can ask Siri questions about a currently opened webpage, or they can request a description of objects in a photo. Siri onscreen awareness also enables more complex actions as suggested by Apple’s documentation, with one example calling for Siri to suggest a creative title for a presentation.
Siri onscreen awareness is one of many advanced features Apple promised for the troubled virtual assistant as part of Apple Intelligence. However, this feature, alongside others like precise in-app controls and personal context understanding, is not set to be available for iPhone users until next year, likely as part of iOS 18.4 around March or April.
