c# - Hololens 2 | Microsoft - Has anyone gotten Scene Understanding to work in realtime?

Question

I've been trying to find proper documentation or tutorials on how to work with the Scene Understanding SDK (in Unity). I'm not new at programming, but not quite proficient either. I've been trying to get Scene Understanding SDK to work through their own documentation page. But it simply doesn't explain well how to get it running with realtime data from the Spatial Awareness Observer/mesh, instead of pre-made scenes.

The example scene from the SDK Github page works to some extent in the Editor, but deploying it onto my Hololens results in the scene not creating actual results when pressing the Scene Understanding panel buttons..

I could post code snippets of what I've tried, but I don't think it will be very helpful, I would be very happy to see a code snipper or project with a basic scene which I can take apart and deploy onto my Lens to see how it works. If anybody has a comprehensible documentation page or tutorial for me, that would be fantastic too!

Answer 1

To compute a live scene from the HoloLens' spatial awareness mesh initialize the scene with SceneObserver.ComputeAsync (to compute from live data) instead of with ComputeSerializedAsync (to load from a saved scene).

There is a code snippet demonstrating this in the Initialization section of the Scene understanding docs:

// Create Query settings for the scene update
SceneQuerySettings querySettings;

querySettings.EnableSceneObjectQuads = true;                                       // Requests that the scene updates quads.
querySettings.EnableSceneObjectMeshes = true;                                      // Requests that the scene updates watertight mesh data.
querySettings.EnableOnlyObservedSceneObjects = false;                              // Do not explicitly turn off quad inference.
querySettings.EnableWorldMesh = true;                                              // Requests a static version of the spatial mapping mesh.
querySettings.RequestedMeshLevelOfDetail = SceneMeshLevelOfDetail.Fine;            // Requests the finest LOD of the static spatial mapping mesh.

// Initialize a new Scene
Scene myScene = SceneObserver.ComputeAsync(querySettings, 10.0f).GetAwaiter().GetResult();

You should be able to update the SceneUnderstandingManager.cs file from the https://github.com/microsoft/MixedReality-SceneUnderstanding-Samples to call ComputeAsync instead of ComputeSerializedAsync if you want the samples to run on the world rather than on the saved scenes.

If you need more specific help you'll need to provide more specific details about what exactly you're doing, what you've tried, and how the behaviour you get differs from the behaviour you are trying to get.