This project enables the rendering of tracked hands on HoloLens 2 and streams them to a Unity scene on PC.
You can find the project code and assets in the following GitHub repository: MixedRealityToolkit-Unity/UnityProjects/MRTKDevTemplate/ - This project is based on the official MRTK3 template, specifically the hand interaction scene.
Unity 2021.3.29f1.
A slight modification has been made to the PUN2 source code. The boolean IsMine in PhotonView is made public. Note: This change needs to be redone if reimporting Photon.
ConnectToServer.cs: This script enables the connection of both HoloLens and the PC to the Photon Unity Networking (PUN) cloud. Once both are connected, rigged hand models are instantiated across the network.
Rigged hand models must be stored in the resource folder: Resource Folder. Each joint of the hand model is equipped with PhotonView, PhotonTransformView, and Owner scripts to synchronize the pose over the network. The Owner script determines who can set and who can read the pose. For instance, the HoloLens sets all poses, while the PC only reads them. Hand materials are stored here as well, e.g., if you want to adjust the HoloLens hands to be rendered white on the PC side and invisible but blocking on the HL side, this can be done here.
PUNRiggedHandMeshVisualizer.cs: This script retrieves joint poses from HoloLens hand tracking. It also synchronizes the hand's scale and visibility across the network (i.e., whether they are currently tracked and should be visible). This script also allows for setting whether hands are rendered or not on HoloLens via showHandsOnTransparentDisplays.
Both the HoloLens scene and the PC scene load the same hand model, but the ownership properties differ:
- PC Scene:
Ownerproperty must be set tofalse. - HoloLens Scene:
Ownerproperty must be set totrue.
SceneOpenedHandler.cs: This script automatically toggles the Owner property in the prefabs each time a respective scene is opened, ensuring proper synchronization and control based on the active platform.
Click to expand!
- Clone the repository from GitHub.
- Open the project with Unity 2021.3.29f1
- Deploy the HL_Hand_Streaming scene to HoloLens 2.
- Run the PC_Hand_Streaming scene on your PC to start receiving hand tracking data (click editor occassionally otherwise it stops updating).
MRTK3 Unity Documentation
MRTK3 is the third generation of the Mixed Reality Toolkit for Unity. It's an open source project designed to accelerate cross-platform mixed reality development in Unity. MRTK3 is built on top of Unity's XR Interaction Toolkit (XRI) and OpenXR. This new generation of MRTK is intended to be faster, cleaner, and more modular, with an easier cross-platform development workflow enabled by OpenXR and the Unity Input System.
- Built on Unity XR Interaction Toolkit and the Unity Input System.
- Dedicated to OpenXR, with flexibility for other XRSDK backends
- Open-ended and extensible interaction paradigms across devices, platforms, and applications
- Rewrote and redesigned most features and systems, from UX to input to subsystems.
- Zero per-frame memory allocation.
- Tuned for maximum performance on HoloLens 2 and other resource-constrained mobile platforms.
- New interaction models (gaze-pinch indirect manipulation).
- Updated Mixed Reality Design Language.
- Unity Canvas + 3D UX: production-grade dynamic auto-layout.
- Unified 2D & 3D input for gamepad, mouse, and accessibility support.
- Data binding for branding, theming, dynamic data, and complex lists.
MRTK3 requires Unity 2021.3.21 or higher. In addition, you need the Mixed Reality Feature Tool for Unity to find, download, and add the packages to your project.
Follow the documentation for setting up MRTK3 packages as dependencies in your project here. Alternatively, you can clone this repo directly to experiment with our template project. However, we strongly recommend adding MRTK3 packages as dependencies through the Feature Tool, as it makes updating, managing, and consuming MRTK3 packages far easier and less error-prone.
| Platform | Supported Devices |
|---|---|
| OpenXR devices | Microsoft HoloLens 2 Magic Leap 2 Meta Quest 1/2 Windows Mixed Reality (experimental) SteamVR (experimental) Oculus Rift on OpenXR (experimental) Varjo XR-3 (experimental) If your OpenXR device already works with MRTK3, let us know! |
| Windows | Traditional flat-screen desktop (experimental) |
| And more coming soon! |
In previous versions of MRTK (HoloToolkit and MRTK v2), all packages were released as a complete set, marked with the same version number (ex: 2.8.0). Starting with MRTK3 GA, each package will be individually versioned, following the Semantic Versioning 2.0.0 specification. (As a result, the '3' in MRTK3 is not a version number!)
Individual versioning will enable faster servicing while providing improved developer understanding of the magnitude of changes and reducing the number of packages needing to be updated to acquire the desired fix(es).
For example, if a non-breaking new feature is added to the UX core package, which contains the logic for user interface behavior the minor version number will increase (from 3.0.x to 3.1.0). Since the change is non-breaking, the UX components package, which depends upon UX core, is not required to be updated.
As a result of this change, there is not a unified MRTK3 product version.
To help identify specific packages and their versions, MRTK3 provides an about dialog that lists the relevant packages included in the project. To access this dialog, select Mixed Reality > MRTK3 > About MRTK from the Unity Editor menu.
Some parts of MRTK3 are at earlier stages of the development process than others. Early preview packages can be identified in the Mixed Reality Feature Tool and Unity Package Manager by the Early Preview designation in their names.
As of June 2022, the following components are considered to be in early preview.
| Name | Package Name |
|---|---|
| Accessibility | org.mixedrealitytoolkit.accessibility |
| Data Binding and Theming | org.mixedrealitytoolkit.data |
The MRTK team is fully committed to releasing this functionality. It is important to note that the packages may not contain the complete feature set that is planned to be released or they may undergo major, breaking architectural changes before release.
We very much encourage you to provide any and all feedback to help shape the final form of these early preview features.
This project welcomes contributions, suggestions, and feedback. All contributions, suggestions, and feedback you submitted are accepted under the Project's license. You represent that if you do not own copyright in the code that you have the authority to submit it under the Project's license. All feedback, suggestions, or contributions are not confidential.
For more information on how to contribute Mixed Reality Toolkit for Unity Project, please read CONTRIBUTING.md.
For information on how the Mixed Reality Toolkit for Unity Project is governed, please read GOVERNANCE.md.