Microsoft designer and game developer Oscar Saladin recently shared a post on Medium. The post showcases some new capabilities for the tech giant’s HoloLens. In the post, Saladin shows how the HoloLens—an augmented reality (AR) pair of smart glasses—allows users to manipulate holographic objects as if they were real ones. As well as fling them around and summon them with “telekinesis” like some kind of digital-world Jedi.
Saladin notes he and his colleagues came up with the HoloLens’ new capability in order to make AR interactions feel more natural. To do so, the designer says the HoloLens team looked into how they could make interacting with objects a more realistic, tactile experience. One that, despite the lack of touch, still feels like a real interaction.
“As articulated hand tracking with HoloLens 2 gives us the pose of a hand, we can construct a physically simulated twin of your hand in the virtual world by replicating the positions, velocities, and momentum of different parts of the hand,” Saladin writes in his post. “When this virtual hand interacts with virtual objects, [our] physics engine simulates the outcome, applying momentum, collision, friction, and gravity.”
Just as video games use physics engines to handle open-ended interactions between objects, so too does the HoloLens. This means the smart glasses can faithfully model the “physical” interactions between a user’s virtual hand and the virtual objects they’re manipulating. Including when the objects are floating in midair.
Speaking of which, Saladin and his colleagues created a “telekinesis” gesture that allows users to whip objects about without “touching” them. Kind of like Yoda on a Force rampage. In the videos above and below, a user demonstrates the ability. The user takes holographic cubes and flings them into the air or onto surfaces, then pulls them back to their palms. (With the classic open-hand Force user gesture, of course.)
As for the results? It seems Saladin et al. are right on target. Playing with the cubes seems natural. And even when users telekinetically manipulate them, their movements have an intuitive sense. The fact that cubes light up in response to touch (purposefully) also enhances the sense that these holographic objects are real.
Indeed, Saladin says the holographic objects are so real, people were climbing on IRL ones to touch them. Hence the reason for the telekinesis power. Some users even describe their skin as feeling warm when they hold particularly bright, glowing objects. Which is a testament to the power of suggestion. And, probably, a hint at what’s coming if we ever physically link our brains to the virtual world.
