Microsoft’s Mixed Reality Toolkit (MRTK) is a cross-platform toolkit that accelerates cross-platform MR app development for Virtual Reality (VR) and Augmented Reality (AR). Our Immersive Experience team explains three ways we leverage it to save clients time and money.

We’re still in the fledgling era of XR application design and we’re always looking for ways to save time and money. One of the major time/money saving tools we’ve been behind, since the beginning, is Microsoft’s Mixed Reality Toolkit (MRTK). The beauty of MRTK, for our team and many others, is the ease at which we can get an idea up and running, polish it based on testing, and put out a finished application; all within a very short timeframe.

First there is Rapid Prototyping, which is made possible by the many User Interface (UI) and User Experience (UX) tools, prefabs (templates of objects, like buttons), and examples that MRTK comes with, out of the box such as:

• Near and Far Interactions for more intuitive user navigation.
• Subtle use of Solvers to keep pertinent information within the user’s field of view without overwhelming them.
• Consolidated Hand Menus to free up precious XR screen space without losing functionality.

Second is faster UI/UX Experience Polish with access to MRTK’s “basic” building blocks and core systems. This allows our team to work in a “Design, Test, Polish, Repeat” pipeline that opens up a large amount of flexibility while keeping the base of the application solid and stable.

Finally, the time savings in ensuring your application works on multiple devices and operating systems. Finishing applications is always the hardest part; the last 10% takes 90% of the time. This is even more difficult when attempting to create Cross-Platform applications. Being able to use a single device, the HoloLens 2 for instance, to test for a large portion of the core development process saves time. When we’re satisfied with the core functionality, we use MRTK’s cross-platform support to publish to multiple devices, for final polish and testing. This process has saved us countless hours of troubleshooting and headache.

MRTK Features That Enable Rapid Prototyping

Near and Far Interactions – One of the most natural feeling interactions on the Hololens 2, is directly manipulating objects with hands. The MRTK team has put a lot of time and energy into allowing developers to give the users a wide variety of feedback which helps make the interactions even more intuitive. Three of the best forms of user feedback are sight, touch, and sound.

Holographic buttons can be poked, pressed, or slid; Objects can be grabbed, moved, rotated, and scaled. Everything can also give subtle, secondary, feedback at the same time. For example, holograms that light up when close to a user’s hand and buttons that animate, make a sound, highlight or change color, move, and change size by simply being pointed or gazed at.

All of these subtle queues add to the natural feel of interactions in a great XR experience and come stock in MRTK.

Solvers – Think of Solvers as little modules that allow you to blend real and XR worlds. There are a few we would consider critical in creating a seamless user experience:

1. Surface Magnetism Solvers allow users to stick holograms to real world objects or other holograms. A great example is interior decorating or retail apps that allow the user to see a product in their own physical space. For example, attaching a holographic TV to a physical wall or holographic furniture to a physical floor to get a better feel for how they will look in real life.
2. Orbital Solvers are a great way to keep menus or models in view because they can be locked to a user’s position and tasked to orbit around them, always staying within a certain angle of that user’s view. A simple example is a step-by-step tutorial message that follows the user while they repair a physical object.
3. Hand Constraints are another amazing way to interact with the real world and they allow developers to attach anything to the user’s hands based off position, orientation, and hand gestures. Hand Menus are one of the many elements derived from this and they are phenomenally useful.

Hand Menus – There are a lot of new problems to solve creatively in the XR space. Screen real estate is at a premium and the last thing anyone wants to do is waste it with unnecessary menus. This is where Hand Menus shine. If you’ve ever used a Virtual Reality application, you’re no stranger to this element. For example turning your hand palm up and a menu appears or turning your hand down and it disappears, leaving you able to see the full view again, unimpeded by a menu you no longer need. This tool, especially in combination with the Near Interactions, allows for fluid and intuitive menus that a user only sees when needed.

User Interface and Experience Polishing Capabilities for Designer Autonomy

Our “Design, Test, Polish, Repeat” mentality has allowed us to quickly pivot from project to project. Having base MRTK assets to work from, especially within the realm of UI/UX, has given us the ability to be more agile than ever. The way MRTK’s building blocks are created and structured help designer autonomy. Building from these blocks when prototyping, we’re able to quickly bring each application from a work in progress prototype to a polished experience.

For example, using MRTK’s button templates, we can quickly create our own branded style with much less effort than starting from scratch. Or create custom Hand Menus as a base for future applications and integrate simple scripts into MRTK’s already robust pipeline to create something truly unique.

Designer autonomy is a huge part of our success and speed. Artists and Designers can work within Unity and MRTK to create an entire UI/UX flow without much need of a Unity Developer. This is a total win for everyone involved. Designers aren’t dependent and waiting on Developers to have time to hook something up, while Developers get to focus on other tasks instead of constantly being randomized when a new interaction is requested. Like a lot of companies, we will create a prototype and then test the heck out of it! We can then rapidly adjust and update our applications based on what we find during testing.

Building Cross-Platform Experiences with MRTK

A huge selling point for MRTK is that their tools work with nearly all XR platforms, it’s highly customizable, and Microsoft is heavily invested in MRTK for the long haul.

A major time saver for us (in the beginning) was the ability to create an experience for HoloLens 1 and immediately port it to HoloLens 2 when they became available. As with all upgraded hardware, we still needed to hook up the additional functionality of the HoloLens 2 like eye tracking, Near Interactions, and Hand Menus. However, we were able to prototype everything before it came out in Unity. When it did, we simply “flipped a switch” and deployed to ARM64 and the HoloLens 2... and it WORKED!

We’ve had similarly great results working with larger cross-platform experiences. Sharing the same core codebase and very similar UI/UX flow across HoloLens (1 and 2), IOS, and Android devices is remarkable and saves massive amounts of time/money/testing.

MRTK is also very customizable. We’ve been able to add and remove functionality with the click of a checkbox, update our buttons, hand menus, and solvers with simple prefab and script updates, and hook into MRTK’s scripts to make it do completely different things if needed.

Seeing the great collaboration and constant updates and work the MRTK team and community have put into this tool makes it easy to rally behind it. The team is eager to hear from studios and indie developers alike to learn how they can make the toolkit even better in the future.

Our Immersive Team at Valorem Reply has saved countless hours of headache and rework while adding layers of polish to our immersive applications by utilizing MRTK. Despite the initial learning curve, MRTK is a must for any development team working in the XR space. It has endowed our team with the ability to:

• Quickly iterate on individual prototypes when dealing with Interactions, Menus, and Solvers.
• Greatly increase the speed at which we can get through the “Design, Test, Polish, Repeat” cycle.
• Achieve even greater reach and stability with cross-platform deployment capabilities and a community supported by the MSFT MRTK Team.

If you are considering a mixed reality application or experience for your organization, our experts are here to help wherever you are on the journey. From identifying mixed reality opportunities and road mapping your path to design and deployment, we can add scale and speed to your extended reality journey. To speak with one of our Immersive Experience or 3D Design specialists, reach out to marketing@valorem.com.

*Valve 3D Model graphics compliments of Turbosquid.