visionOS 27 Launches with Siri AI, Spatial Panoramas, and Tracked Accessory Support

Apple launched visionOS 27 for Vision Pro headsets this week, marking the fourth major operating system version for the spatial computing device. Despite limited announcement fanfare, the update represents one of the largest upgrades yet, introducing preconfigured held object tracking, an infrared LED tracked accessories framework, high-quality unfoveated screen recording, and numerous other improvements.

visionOS 27 is available for both the M2 and M5 models of Apple Vision Pro.

Siri AI and Visual Intelligence

Siri AI, Apple's large language model-powered overhaul of its digital assistant, is now rolling out in beta on visionOS 27 for both M2 and M5 Vision Pro headsets. Users must join a waitlist in the visionOS Settings to access the feature, with Apple noting that wait times vary and the feature will download automatically once access is granted.

Siri AI is currently available only in English when device and Siri languages are set to the same supported English option. Apple says Siri AI is powered by next-generation Apple Foundation Models, which were co-developed with Google and based on its Gemini models. Some tasks run on an on-device model, while more complex operations are handled by Apple's Private Cloud Compute system.

Apple describes Siri AI as a "profoundly more capable and conversational assistant" than traditional Siri, with "personal context understanding, broad world knowledge, and more." In visionOS, Siri AI appears as a persistent orb that users can pin to a physical spot in their room. Users can start speaking by simply looking at the orb, bypassing the need to say "Siri", though the traditional wake word remains an option.

Siri AI on visionOS also supports Apple's Visual Intelligence feature, enabling users to "search, ask questions, and take action" relating to physical and virtual surroundings. A dedicated Siri app lets users access previous conversations and serves as a preinstalled alternative to third-party AI chatbot applications including ChatGPT, Gemini, Claude, Grok, and Meta AI.

Additional features powered by Apple Foundation Models include significantly more realistic image generation in Image Playground, automatic bookmarks and Reading List organization in Safari, the ability to upgrade saved password strength with a single tap (coming in a future visionOS 27 version), and personalized Smart Replies in Messages. On the M5 Vision Pro, Siri also features a more realistic and expressive voice that users can customize.

Spatial Panorama Environments

visionOS 2 introduced synthetic stereoscopy for 2D images, and visionOS 26 enabled conversion of 2D images into limited-perspective volumetric spatial scenes using Gaussian splatting. visionOS 27 extends this spatial scene concept to iPhone-captured panoramas, which users can also set as their visionOS home environment to serve as backgrounds for windows and volumes in the Shared Space.

Higher Frequency Object Tracking

visionOS 2 added object tracking for real-world 3D objects that app developers prespecify by providing 3D models to Create ML on Mac. The original system was designed for static objects and responded slowly to movement.

visionOS 27 introduces a high-frame-rate object tracking mode suitable for moving and held objects. While Apple does not specify the exact update rate, initial reports from developers suggest it may operate around 30Hz. Apple recommends Extended Training mode for tracking held objects, which significantly improves tracking performance but also substantially increases machine learning model training time. Developer Brad Lynch found that training a fish toy took approximately 18 hours. Despite improvements, this tracking method still maintains a significantly lower update rate and higher latency than hand tracking or tracked accessories.

IR LED Tracked Accessories

visionOS 26 featured hardcoded support for PlayStation VR2 Sense controllers and Muse digital stylus, both featuring constellations of infrared LEDs. visionOS 27 opens this capability to any company by adding necessary APIs for developing infrared LED tracked accessories for Vision Pro, similar to SteamVR Lighthouse Tracking accessories on PC.

Optical tracking runs at 90Hz and is fused with IMU accelerometer and gyroscope data from the accessory, enabling tracking quality comparable to PlayStation VR2 Sense controllers. Hardware makers can choose between two software support paths. The Companion App path allows releasing a visionOS companion app that registers the accessory shape and IR LED configuration, suitable for consumer devices. The Built Into App path allows developers to build accessory details into apps that utilize the accessory, ideal for enterprise applications.

DFRobot and MikroE announced reference hardware and developer kits for this capability, with DFRobot hardware available to purchase now. Companies can use these directly or as a basis for their own accessories, lowering the barrier to entry compared to building from scratch.

Thórsmörk Environment

visionOS 27 introduces a new environment option for the Shared Space: Iceland's Thórsmörk valley, known as Thor's Valley and widely considered one of the most scenic places on the planet. The environment features day and night modes, with the nighttime variant displaying the aurora borealis. Thórsmörk is the tenth virtual environment for visionOS, following Jupiter from Amalthea added in visionOS 26.

The complete visionOS environment list now includes Haleakalā, Yosemite, Joshua Tree, Mount Hood, White Sands, The Moon, Lake Vrangla, Bora Bora, Jupiter, and Thórsmörk.

Unfoveated Screen Recording

visionOS 27 introduces high-quality unfoveated screen recording. Apple Vision Pro typically uses always-on eye-tracked foveated rendering in the Shared Space and many Full Space apps, rendering only the region currently in focus. The new recording capability captures full-quality video without this foveation limitation.