Overview
This Hololens 2 project explores an interaction to reimagine the creation tools for Mixed Reality. Inspired by an old television show, "Shaka laka boom boom" where Sanju has a pencil that makes things come to life.
Tools
Hololens 2, MRTK, Unity, Pytorch,YOLOv8, Quickdraw dataset
Timeline
Aug 2022
Design
The design process involves creating a Mixed Reality visualization agent that can interpret sketches on paper, using the Hololens 2 for its advanced display capabilities. Initial planning included brainstorming use cases and storyboarding potential applications. The Google Quickdraw dataset was used as a resource to train machine learning models to interpret sketches. The design also included the integration of Sketchfab's API, a comprehensive 3D model database, to generate 3D models in real-time. The ultimate goal is to create an interactive prototype that can recognize hand-drawn objects and display corresponding 3D models.
Development
The app is built for Hololens 2, using real-time object detection from sketches to create 3D models in the user's vision field. It is trained on the Google Quickdraw dataset, supporting around 350 common objects. The 3D models are sourced from the SketchFab API, filtered to meet Hololens 2's specifications, downloaded, and then rendered for the user. The app also uses MRTK3 to enable hand interactions with the 3D models, allowing users to scale, rotate or move the holographic objects.