With the Quest 3 and Vision Pro, we are shifting our focus to Mixed Reality. What are the main development challenges when blending digital objects with the real world? Specifically, how do you handle "occlusion" so that virtual objects don't just float in front of real furniture, and how do you manage lighting consistency?
3 answers
The biggest hurdle is Depth Occlusion. Without a proper depth map, a virtual character standing behind a real sofa will appear to "overlay" it, which ruins the illusion. On the Quest 3, you should use the Scene Mesh provided by the Insight SDK to create invisible physical colliders for the room. For lighting, you need to use Ambient Light Estimation. This API allows your virtual objects to change color and intensity based on the real-world light data captured by the cameras. Without this, your virtual objects will look "glowy" and disconnected. Implementing "shadow planes" that catch virtual shadows on real floors is also a small technical detail that makes a massive difference in grounding the object in reality.
Are you struggling more with the Spatial Anchors or the actual visual fidelity of the passthrough graininess affecting your UI design?
We found that Physics Interaction is the hardest part. Making a virtual ball bounce realistically off a real-world table requires a very accurate spatial mesh.
Emily is right. Even a 5cm offset in the mesh makes the physics feel "off" and reminds the user they are looking at a simulation.
Jeffrey, the graininess is a real issue for readability! Brian, for MR UI, I recommend using larger fonts and high-contrast colors. Regarding the Spatial Anchors, make sure you implement a "Relocalization" check. If the user moves to a different room and comes back, your virtual objects need to stay exactly where they were placed. This requires saving the anchor data to the local headset storage and refreshing it on startup. If you don't, the user has to "re-set" their virtual office every single time they put the headset on, which is a terrible user experience.