Open source 3D spatial dataset for training embodied AI

Open source 3D spatial dataset for training embodied AI

Market news |
3D spatial data company Matterport has announced a collaboration with Facebook's AI Research lab (FAIR) through which it will make the largest-ever dataset of 3D indoor spaces available for academic, non-commercial uses.
By Rich Pell


Through the collaboration, the companies hope to enable researchers to advance Habitat – Facebook AI’s simulation platform for training embodied agents (virtual robots) – to help robots better understand and interact with the physical world. The resulting Habitat-Matterport 3D Research Dataset (HM3D) is a collection of 1,000 of Matterport’s high-resolution digital twins made up of residential, commercial, and civic spaces generated precisely from real-world environments.

The companies say they expect HM3D to play a significant role in advancing embodied AI research, which seeks to teach robots and virtual AI assistants to understand and interact with the complexities of the physical world.

Dhruv Batra, Research Scientist at Facebook AI Research says, “Until now, this rich spatial data has been glaringly absent in the field, so HM3D has the potential to change the landscape of embodied AI and 3D computer vision. Our hope is that the 3D dataset brings researchers closer to building intelligent machines – to do for embodied AI what pioneers before us did for 2D computer vision and other areas of AI.”

HM3D is free and available now for academic, non-commercial research. Researchers can use it with FAIR’s Habitat simulator to train embodied agents, such as home robots and AI assistants, at scale. HM3D, say the companies, is a foundational step towards helping these agents navigate through real-world environments and better understand the variations of spaces such as bedrooms, bathrooms, kitchen and hallways, as well as the different configurations of those rooms within every structure.

It can also assist robots in recognizing how objects within rooms are typically arranged so that instructions are correctly understood. This research could one day be used in production applications like robots that can retrieve medicine from a bedroom nightstand or AR glasses that can help people remember where they left their keys.

Conway Chen, Vice President of Business Development and Alliances at Matterport says, “We are excited to collaborate with Facebook as we provide the academic and research communities access to this unique spatial dataset that is sure to impact how we work and live. With more than five million spaces captured with the Matterport platform, we are the only company that can offer a diverse library of high-resolution, data-rich digital twins of various styles, sizes, and complexities from across the world. HM3D can also be used more broadly by academia, and we can’t wait to see what innovations emerge.”

For more, see the Facebook AI blog post “Introducing the Habitat-Matterport 3D research data set for training embodied AI.”

Facebook AI Research

Related articles:
3D space capture company going public via SPAC deal
AI imaging app turns smartphones into 3D capture devices
Computer vision platform creates intelligent 3D twin of any building
New Microsoft IoT service makes physical spaces virtual

Linked Articles