iOS 18 Siri will feature advanced in-app functionality – Here’s what’s new

Siri will be able to perform complex actions in core system apps

In one of our earlier exclusive reports, we said that Siri would get a big upgrade. We explained that the company’s virtual assistant would be able to open and edit specific photos, set the device wallpaper, and so on.

Since then, AppleInsider has received the exact details of Siri’s new functionality, as well as prompts Apple used to test the software. Coming from people familiar with Apple’s AI initiative, this new info gives us our most detailed look yet at the virtual assistant’s new in-app capabilities.

Although one might expect the virtual assistant only to follow direct instructions and orders like “Open Safari,” Apple has tested a wide variety of different prompts for the new AI. Many of these prompts contain “natural” language, as would be spoken by a real human being.

This means that Apple’s test prompts often contain descriptions of what the user is trying to do, along with some arguably superfluous details.

Instead of containing only direct commands like “Show me pictures of my cat,” the company’s testing prompts mention that the user wants to make a blog, or that they’re feeling lazy/nostalgic in some instances.

In doing so, Apple wants to get its AI accustomed to natural language, speech patterns, sentence structures, and instruction types Siri is likely to receive from end users — some of whom may not be all that familiar with new technology.

• Books
• Calendar
• Camera
• Contacts
• Files
• Freeform
• Generative Playground (internal)
• Keynote
• Mail
• Magnifier
• News
• Notes
• Photos
• Reminders
• Safari
• Stocks
• System Settings
• Voice Memos

Through Siri, users will be able to open specific books or sections of the app, flip through book pages, and so on.

Image Credit: Apple

These features will ultimately assist users who may be unfamiliar with the app’s user interface or who have difficulty navigating through the UI. Siri will make it significantly easier to find things like the audiobook store or the user’s PDF files and downloaded books.

Within the Books app, Siri will be able to:

• Open a specific book by saying its title
• Open recent books, the book you last read
• Change book themes – bold, focus, paper, original
• Open the user’s book library
• Flip to the next/previous page
• Pause audiobooks
• Switch to the library, search, reading now or books tab
• Ask for specific things in the Book Store – like audiobooks or manga
• Show finished books, downloaded books, PDF documents

Siri’s ability to control device cameras will make it a lot easier to take photos and videos. Users could tell Siri to set their camera to a certain mode, and set a timer.

Siri will gain the ability to take pictures, videos, and more with iOS 18

This could potentially benefit users who are unfamiliar with the various options and settings within Apple’s default Camera application. It would also give users the option to easily switch between the front-facing and rear cameras through Siri commands.

With the Camera application, Siri will gain the ability to:

• Toggle video recording
• Open the camera to a specific mode – photo, portrait, video, slow motion – and set a timer
• Flip to the front-facing camera or to the rear camera

Siri’s new Keynote-related features will benefit users in the education, marketing, or business sectors. Through Siri, users will easily be able to see who edited a Keynote presentation and will be able to make changes of their own.

Keynote is part of Apple’s iWork Suite of productivity applications

Educators and students will now have the option to insert audio, images, and video via Apple’s new-and-improved, LLM-powered virtual assistant. Based on what we’ve heard, researchers and analysts will likely find this feature useful, as it would make the creation of presentations significantly easier.

In Keynote, Apple’s new-and-improved visual assistant will gain the ability to:

• Add images, image galleries, audio, video, text boxes, or comments to specific slides – eg. “Add the image named cat to Slide 1”
• Show specific slides
• Create a new Keynote presentation with a specific theme
• Create a new slide in a Keynote presentation
• Play and stop Keynote presentations
• Display the Activity Stream of a Keynote Presentation – showing what was changed and by whom
• Add web videos, such as YouTube videos, from Safari to slides
• Set bullet points for specific slides
• Set the title for a specific slide

Apple’s Mail application will get a significant overhaul this year, as we revealed in our exclusive report on Project BlackPearl. The company will enhance its built-in email app with machine learning, which will allow it to automatically classify emails based on their text content.

Apple will upgrade the Mail app with machine learning as part of Project BlackPearl

This means that Mail will be able to separate emails into categories such as Commerce, News, Promotions, Social, Transactions, and Other. Siri will be able to use the information to perform different actions within the app.

For instance, Apple’s AI-powered virtual assistant will be able to identify emails with promo codes, or those from a specific company and label them as junk.

Siri will be able to perform a variety of actions within Mail, all of which will streamline email communication. With Siri, users will have the option to reply to all senders of an email.

For instance, users could say something like “Reply to everyone on this email with the body text ‘Yes, I’ll be there.'” The virtual assistant will also have the ability to forward emails, block email senders, and much more.

In the Mail app, Siri will be able to:

• Send an email message, optionally at a specific time or date
• Un-send an email (undo send)
• Save an email as a draft
• Delete an email draft
• Compose an email, and cancel email creation
• Block email senders
• Archive, move, forward emails
• Mark an email as junk (eg. Emails with promo codes or ads)
• Enable or disable flags for emails with specific subjects, with the option to set the flag color
• Mute and Delete emails from a specific sender, or about a specific subject/topic
• Remove the follow-up suggestion to an email
• Reply to emails, or to all recipients of an email
• Mark an email as “Read later” with an optional date (eg. Tomorrow) or remove the designation from an email
• Unsubscribe from emails, especially marketing-related ones
• Enable notification settings for emails from a specific sender or about a certain topic (eg. Delivery)
• Summarize emails and email threads
• Enable AI-generated “Smart Replies” to emails

Apple’s Mail application is also getting a feature known as Smart Replies, through which users will be able to send off an AI-generated response to an email. Apple’s response generation is handled by the on-device Ajax LLM and cloud-based processing in more complex situations.

All of this is sure to benefit business users and members of large companies, who routinely deal with massive amounts of emails. Customer service representatives would also likely make use of the new Siri functionality.

Apple’s Siri enhancements for Notes will make it significantly easier to navigate between individual notes, tag them, and move them around. Users will have the option to utilize key features through the company’s virtual assistant Siri.

Apple’s Notes application will receive in-app audio recording, audio transcription and LLM-powered summarization with iOS 18

Within Notes, Apple’s new LLM-powered Siri will have the ability to:

• Create, rename, and delete Note folders
• Open, create, and delete specific tags in Notes
• Remove tags from specific notes
• Add tags to individual notes (if not already present)
• Include/exclude existing tags for individual notes
• Open a specific account within Notes (eg. iCloud, Outlook, etc.)
• Open, move, delete, or pin/unpin a specific note
• Open the note or folder views

The built-in Notes app is also expected to feature in-app audio recording, audio transcription as well as LLM-powered summarization. This will allow users to store audio recordings, summaries, and transcriptions alongside other material such as text or images – all in one note.

Math Notes is another upgrade Apple has in store for the Notes app, which will offer integration with the all-new GreyParrot Calculator. With Math Notes, users will be able to include complex mathematical equations and graphs as part of individual notes.

Also in the works, Keyboard Math Predictions will allow Apple’s software to recognize mathematical expressions as part of text input. As a result, users will have the option to auto-complete mathematical equations through solutions generated by AI and machine learning.

As a whole, these enhancements will turn Notes into the go-to note-taking application, making it a true powerhouse for students as well as engineers and business professionals.

As we revealed in one of our earlier reports, Siri will have significantly enhanced functionality within the built-in Photos application. Within the app, users will have the option to edit, move, and hide photos, albums, collections and memories through Apple’s virtual assistant.

The Photos app is getting a new Clean Up tool powered by generative AI

Siri will also have the ability to identify specific objects, landscapes or people within a photo through Visual Look Up. This means that users could request to see photos of their loved ones, like friends and relatives – or they could ask to see things like sunsets and nature photography.

The latter would help users who might be looking to create a nature-themed blog or scrapbook. Users will also have the option to edit things like vacation photos, videos, screenshots – all through Apple’s virtual assistant, Siri.

In Photos, Siri will be able to:

• Search for objects in photos, both those locally and from the web – pets, landmarks, food, nature, landscapes, places, sunsets
• Rotate Photos – clockwise, counterclockwise
• Rename albums with user-provided titles
• Open photos of specific people or pets (relatives, friends, etc.)
• Duplicate photos
• Move photos to the Personal Library
• Open memories
• Create new memories, albums
• Add photos to albums
• Open photos and videos for editing
• Open albums, folders, or collections within the Photos app
• Hide photos of people, pets, objects, or photos from a specific time
• Favorite individual photos
• Enhance photos
• Apply filters to photos – noir, vivid, etc.

Users who recently moved from an Android or Windows device to a Mac or iPhone would possibly have difficulty navigating the user interface. This is where Siri’s in-app functionality would truly shine, as users would no longer need to memorize convoluted settings menus and various options.

Apple’s default Photos app will get some minor UI changes, along with new AI-powered editing features. The Clean Up tool will facilitate the removal of objects from an image through the use of generative AI.

Siri’s functionality within Reminders will assist users in creating things like lists of chores, errands, groceries, books to read, school supplies, travel arrangements, and so on. The virtual assistant will also be able to add or remove hashtags from specific reminders and generate overviews of reminders.

The Calendar and Reminders applications will become integrated with iOS 18 and macOS 15

This would ultimately help users who regularly deal with large volumes of information and who need to remember specific details. Business users could create reminders of people they need to meet, while students could make a list of supplies needed for the new school year.

In Reminders, the new AI-infused assistant will be able to:

• Create new Reminders lists
• Create new Reminders list sections
• Add or remove hashtags from specific reminders
• Display a list / overview of reminders

Reminders is also expected to receive integration with Apple’s default Calendar application, as AppleInsider exclusively revealed. This means that users will be able to create and edit reminders directly from the built-in Calendar app.

Having Siri read or summarize the on-screen page contents would assist users who have poor eyesight or visual impairments. This would also help those who are dyslexic, or have other difficulties with reading and reading comprehension.

Safari 18 will introduce a new Intelligent Browsing feature along with a built-in content blocker called Web Eraser

Managing tab groups through Apple’s upgraded virtual assistant would make organizing and navigating significantly easier. Users would be able to more easily conduct research, plan travel arrangements, and open specific websites.

As is consistent with Apple’s focus on privacy, Siri will also have the option to create private tabs and private tab groups. This ensures that the Safari browser does not keep a record of browsing activity for those tabs and windows.

In Safari 18, Siri will gain the ability to:

• Read or summarize an article within a webpage
• Create new tab groups
• Open a new tab in the current tab group, or the Private tab group

Safari 18 will also receive an all-new Intelligent Browsing feature, which will enhance the search functionality and provide summaries of articles. The next-gen browser could also receive a built-in content blocker called “Web Eraser.”

Web Eraser has generated controversy, with the UK’s News Media Assocation and a group of French publishers both sending Apple complaints about the feature. Although AppleInsider was told the feature is complete and functional, Apple may opt to remove it ahead of WWDC to avoid controversy.

The in-app capabilities of Siri within the Stocks app will greatly benefit less tech-savvy long-time investors. Through Siri, users could request an overview of business-related news or create watchlists with specific stocks.

Siri will be able to perform actions within the Stocks app on iOS 18 and macOS 15

This would help users keep track of specific stock types (eg. Blue chips) while also keeping them updated in regard to the latest events in the investment world. It would also aid those who are unfamiliar with the app’s user interface, or who may not have the time to scroll through different menus and settings.

In Stocks, Apple’s virtual assistant will have the ability to:

• Display business news
• Create Watchlists with specific names and specific stocks

Apple’s upgraded virtual assistant will offer much-needed functionality within the Voice Memos app. Through Siri, users will be able to create, pause, delete and move individual recordings or folders of recordings.

Voice Memos is getting a new audio transcription feature in iOS 18 and macOS 15

This would ultimately serve to benefit a truly wide array of users, ranging from those in the business sector to students, engineers, and creatives. Users will be able to record meetings and lectures, shopping and to-do lists, singing and much more – all through Siri.

With the Voice Memos application, Siri will be able to:

• Open individual voice memos / recordings
• Create new recordings inside Voice Memos
• Pause recordings
• Create, open, and delete folders within Voice Memos
• Delete specific recordings by their name, date of creation, and audio content
• Search for voice memo recordings

The new in-app functionality will give users a way of more easily locating specific files they might be looking for – as Siri will be able to find recordings by name, date, or location. The Voice Memos app is also expected to receive in-app audio transcription.

Siri will also be able to open different areas of System Settings, such as Brightness Settings, Home Screen, and App Library Settings, along with Multi-tasking and Gesture Settings. The System Settings application is also expected to receive a reorganized layout with iOS 18 and macOS 15.

The System Settings application is getting reorganized with iOS 18 and macOS 15

Apple’s virtual assistant will be able to open new boards in Freeform, scan documents with the Files app, and move specific files to the trash. In the Contacts application, Siri will have the ability to edit contact details.

In the Magnifier app, Siri will be able to describe objects, plus identify people and doors in view of the camera. This would presumably work similarly to how products like Rabbit R1 identify objects, while making use of Apple’s existing frameworks and features like Visual Lookup.

Siri is also capable of generating a variety of different images via Apple’s internal Generative Playground application, though it remains unclear if this functionality will become available to the end user.

As part of Project Greymatter, Siri will receive the ability to generate summaries of articles, emails and email threads, messages, notifications, and notifications stacks as well as posts. The Greymatter Catch-Up feature will provide users with a recap of their most recent notifications.

Siri, Spotlight, Mail, and Messages will all get significant upgrades thanks to Apple’s Ajax LLM and generative AI software

Apple’s AI software is capable of generating different levels of summarization, ranging from just three topic words all the way up to three sentences. These different summarization levels will likely be used for specific UI elements within Apple’s new operating systems.

Siri, Spotlight, and Mail will all receive advanced search functionality and greater contextual understanding through Apple’s AI software. When creating replies, Siri will be able to take into consideration different entities like people and companies, places, and calendar events.

According to people familiar with the matter, generative AI software will also allow for the instant creation of Smart Replies, which will be available in Mail, Siri and Messages. These AI-generated responses will make sending messages significantly quicker.

As far as non-AI upgrades are concerned, Apple will revamp the Calculator application on macOS 15. The new-and-improved Calculator will also be available on iPad for the very first time, as it’s a “universal application”

The calculator app is getting a redesign for macOS 15 as part of Project GreyParrot

Developed under the codename Project GreyParrot, the new application will enable an iOS-style Calculator on Mac, with rounded buttons and darker shades of black. It will feature a new history tape sidebar and an improved unit-conversion system.

Apple’s Maps application is also expected to receive an upgrade which will allow users to add notes to specific locations. With these new place notes, users can add the key information about a place that could only be relevant to them.

AppleInsider was told that these notes would only be accessible to the users who created them, within the Maps application. This would make them different from something like the built-in Notes app, which features iCloud compatibility.

It is important to remember, however, that not all features make it to release. The information in this report comes from people familiar with pre-release versions of Apple’s operating systems and test environments, and may not reflect the final release version.

Apple is expected to debut its next generation of operating systems at its annual Worldwide Developers Conference on June 10, which is when we’ll likely see some of the improvements mentioned here.