LEGO Projection Mapping

LEGO Projection Mapping is an experimental work on exploring the possibility of projection mapping on LEGO sets with automated calibration. Whilst inspired by the rich third-party market of LED lighting kits for official sets, projection mapping offers the potential of dynamically overlaying exciting effects and contents on LEGO sets. Imagine being able to overlay festive themes, animate characters in LEGO cities or re-colour LEGO buildings! The breakthrough work here is the development of an algorithm that allows the automatic calibration of the projection system using nothing more than a smartphone and a household projector. 

The algorithm is inspired by how remote sensing algorithm solves for the 3D spatial relationship amongst the sun, the atmosphere and the satellite, here translated to the projector, the LEGO set and the smartphone camera respectively. This is coupled with state-of-the-art machine learning-based scene segmentation and object detection to ultimately provide a seamless projection mapping experience. The research and development are still in its early days. If you want to get involved or have any ideas, please get in touch!

Below are some demo photos of an early prototype for the Disney Train and Station set #71044.