Eyes & hands in AR: A sci-fi-inspired mobile UI research

One-handed interaction is ubiquitous in our daily lives, primarily through the use of smartphones. Smartglasses and head-mounted displays (HMDs) become increasingly prevalent, with big players like Apple, Microsoft, and Meta taking center stage in the race for the next generation of spatial computing devices and AR/VR. In our research, we aimed to take a step beyond the home and explore how we can design user interfaces (UIs) that are as mobile as smartphones. Interestingly, science fiction (e.g., Star Wars, Marvel) has introduced numerous innovative concepts that offer a “freehand” alternative to a smartphone. <img alt="" src="https://miro.medium.com/v2/resize:fit:1382/1*e5WETe9FUoCb3GLBSyzlvQ.png" style="height:1035px; width:292px" /> <img alt="" src="https://miro.medium.com/v2/resize:fit:399/1*P4OTJF7JTYLMzPxw_UlKGg.png" style="height:296px; width:295px" /> <img alt="" src="https://miro.medium.com/v2/resize:fit:1280/1*YWEU3iVEquSyLlmjmSYq3A.gif" style="height:676px; width:414px" /> Image Credit: Disney, <a href="https://youtu.be/Oed_RffEjvE" rel="noopener ugc nofollow" target="_blank">youtube</a> (right) Partly, it’s possible with current HMDs. For instance, Microsoft’s Mixed Reality Toolkit (MRTK) offers the <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-uxcomponents/packages/uxcomponents/hand-menu" rel="noopener ugc nofollow" target="_blank">Hand Menu</a>, a virtual UI element that attaches to and moves with the user’s hand, for anytime access. Earlier, the HoloLens 1 employed a <a href="https://learn.microsoft.com/en-us/dynamics365/mixed-reality/guides/hl1" rel="noopener ugc nofollow" target="_blank">“bloom” gesture</a> to summon the device’s home menu at once, positioning it in a world-anchored space. Meta Quest devices offer the <a href="https://www.meta.com/help/quest/articles/getting-started/getting-started-with-quest-2/universal-menu-for-quest-2-and-quest/#:~:text=You%20can%20access%20the%20universal,and%20record%20video%20in%20VR." rel="noopener ugc nofollow" target="_blank">Universal Menu</a>, which spawns at the hand and is then world-anchored, for one-time actions via pinch dragging. <a href="https://medium.com/@ken.pfeuffer/eyes-hands-in-ar-a-sci-fi-inspired-mobile-ui-research-7f75b69b15f2">Learn More</a>