Language for generalizable representations

• Created a virtual playground for embodied AI to learn interpretable, common-sense representations of its environment.
• Built a simulated environment in Unreal Engine 4 that couples language and vision in a scene graph.
• Devised a dataset of images, language labels, and object segmentation.
• Ongoing: a representation learning algorithm that uses language labels to decompose a latent encoding of the environment.

Collaborators: Xu Xie, John Dang, Samuel Hunter, Paymon Haddad, Yixin Zhu, Song-Chun Zhu