text: 414,40,1331614517634224132,322460769,Yoav Artzi,"New NLP+robotics+vision paper: Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following @ CoRL 2020 <LINK> Work done by Valts Blukis Three core contributions make this happen, let’s upack them ... <LINK> First contribution: a general few-shot language-conditioned object segmentation method. We align references to objects in language to their observations in the world via a database of exemplars. Need to reason about new objects? Add exemplars to the database! <LINK> Training requires quite a bit of data to generalize well to unseen objects, *but* the nice part: the whole thing is trained from automatically generated augmented reality data using synthetic ShapeNet objects <LINK> Second contribution: given the alignment, we formalize and build a learned object context map. The map masks over object identities and easily handles new objects. It explicitly encodes text context information into positions in the spatial environment. <LINK> Third contribution: integrating the map into a complete policy (natural language instructions, raw observations) -&gt; continuous velocity control. The policy generates plans on top of our context maps as visitation distributions. <LINK> First stage mapping is through a series of projections and using our LingUNet architecture to predict the visitation distributions on top of the map (we first introduced this method in our CoRL 2018 paper) <LINK> Agent never saw all these objects during learning, just a few exemplars. Bonus 1: modeling of concepts like objects and trajectories is strong inductive bias —&gt; outperform systems that learn with hundreds of demonstrations with new objects. Let’s look at the complete video again <LINK> Bonus2: interpretability via explicit alignments! Another example ... <LINK> There’s a much longer talk with more details <LINK> from the recent SpLU workshop and also a brief overview from CoRL <LINK>",https://arxiv.org/abs/2011.07384,"We study the problem of learning a robot policy to follow natural language instructions that can be easily extended to reason about new objects. We introduce a few-shot language-conditioned object grounding method trained from augmented reality data that uses exemplars to identify objects and align them to their mentions in instructions. We present a learned map representation that encodes object locations and their instructed use, and construct it from our few-shot grounding output. We integrate this mapping approach into an instruction-following policy, thereby allowing it to reason about previously unseen objects at test-time by simply adding exemplars. We evaluate on the task of learning to map raw observations and instructions to continuous control of a physical quadcopter. Our approach significantly outperforms the prior state of the art in the presence of new objects, even when the prior approach observes all objects during training. ","Few-shot Object Grounding and Mapping for Natural Language Robot