The recent release of Microsoft HoloLens 2 Research Mode enriches the immersive mixed reality experience with access to IMU sensor data and articulated hand-tracking and eye-tracking capabilities. Valorem's Director of Global Innovation, Research & Incubation explains how these advantages have made HoloLens 2 an industry leader in computer vision research devices.

The HoloLens 2 is by far the most sophisticated Spatial Computing device on the market. A powerful mobile device, packed with sensors and advanced AI computer vision algorithms that can spatially process the world around us in real-time, fully untethered.

The HoloLens 2 built-in computer vision algorithms like the 3D reconstruction / spatial mapping, articulated hand tracking and more are optimized for the HoloLens’ holographic processing unit (HPU). Extending those was not possible by default, but thanks to the newly added Research Mode feature it is now possible to access the raw sensor data also on the HoloLens 2. This enables custom computer vision algorithm development with processing on the HoloLens itself, remotely on a PC or in the cloud.

Features of the Microsoft HoloLens Research Mode

Research Mode was available on the HoloLens 1 but was more limited than the new HoloLens 2 version. Combined with the latest and greatest sensor and processing units, HoloLens 2 is enabling unparalleled and powerful immersive mixed reality experiences for custom computer vision solutions.

Using Research Mode in HoloLens 2, you can enhance the immersive experience with access the following sensor data:

Visible Light Environment Tracking cameras: 4 grayscale cameras used by the system for head tracking and map building. Two of the grayscale cameras are configured as a stereo rig, capturing the area in front of the device so that the absolute depth of tracked visual features can be determined. Two additional grayscale cameras angled at the side provide a wider, better feature tracking. The cameras are synchronized with global-shutter and are more sensitive to light than the color camera, capturing images at a rate of up to 30 fps.

Depth camera: 1 Time-of-Flight / active infrared illumination camera to determine depth. This ToF camera is similar to the depth camera of the Azure Kinect, where the HoloLens 2 depth camera supports two operation modes:

• High-framerate (45 fps) near-depth sensing commonly used for hand tracking. Gives pseudo-depth with phase wrap beyond 1 meter. For example objects at 1.5m distance will be reported at 0.5m distance and so on.
• Low-framerate (1-5 fps) far-depth sensing, currently used by spatial mapping.

In both modes the depth camera also provides actively illuminated infrared (IR) images which are mostly unaffected by ambient light and can be useful on their own.

Inertial Measurement Unit (IMU) / motion sensor consisting of:

• Accelerometer used by the system to determine linear acceleration along the X, Y and Z axes and gravity.
• Gyro used by the system to determine rotations.
• Magnetometer used by the system to estimate absolute orientation.

RGB color camera: This is the default color camera used for Mixed Reality Captures and can also be accessed without Research Mode enabled via various APIs like Windows.MediaCapture or Unity. It is an auto-focus photo/video camera with auto white balance, auto exposure, and full image processing pipeline. It has various modes of operation, ranging from high resolution and 60 fps to more lower resolution camera profiles. The full list is available here.

The HoloLens 2 Research Mode allows applications to also simultaneously leverage the results of the built-in computer vision, such as SLAM, for motion tracking of the device as well as the spatial mapping algorithms for environmental 3D reconstruction, articulated hand-tracking and eye tracking. Access to the raw eye tracking cameras is not provided for privacy and security reasons but existing APIs can access the processed eye-gaze direction.

Setting Up HoloLens 2 Research Mode

HoloLens 2 Research Mode is available as a public preview beginning with Windows build 19041.1356- which is provided via the Insider Preview program at the moment- and will be available as a regular update in the future. Be aware that enabling Research Mode uses more battery power and can also lower the overall security of the device due to application sensor data misuse, therefore make sure to only run trusted applications leveraging Research Mode APIs.

Once the correct Windows build is running on the HoloLens, the Research Mode needs to be enabled via the Device Portal by checking “Allow access to sensor streams” and rebooting the device:

Research Mode’s Built-In Sensor Streams

The various sensor streams can be accessed via Microsoft’s Media Foundation API layer. To get started it’s best to use one of the four samples developed by Microsoft’s Mixed Reality research team released as open source on GitHub:

• SensorVisualization: visualizes camera streams live on the HoloLens device.
• CalibrationVisualization: visualizes depth and gray-scale camera coordinate frames live.
• CameraWithCVAndCalibration: shows how to process Research Mode streams live on the HoloLens 2, leveraging the popular computer vision library OpenCV to detect arUco markers in the two frontal gray-scale cameras and triangulate the detections.
• StreamRecorder: simultaneously captures depth, gray-scale cameras, head, hand, and eye tracking data and saves the streams on the HoloLens device. Additionally, Python scripts are provided for offline processing of the recorded streams on a PC or in the cloud.

In the below video I walk-through the SensorVisualization sample and show how to use it end-to-end, including a use case allowing night vision!

Modernizing Your Workplace with HoloLens 2 and Research Mode

The HoloLens 2 Research Mode allows us to research custom computer vision solutions for specialized use cases and is not aimed for end-user deployments. We at Valorem Reply are here to help with our experienced team of computer vision and HoloLens developers, artists, and designers. If you have a Spatial Computing project in mind (with or without Research Mode), we would love to hear about it! Reach out to us as marketing@valorem.com to schedule time with one of our industry experts.