Research / Stanford University

Towards Natural Language Reasoning for Unified Robotics Description Format Files

Neil Nie, Austin Patel, Aakash Mishra

Stanford University · Mar 2024

PDFCite
Pipeline: URDF files are rendered, captioned by a vision-language model, and turned into question-answer pairs for fine-tuning
Pipeline: URDF files are rendered, captioned by a vision-language model, and turned into question-answer pairs for fine-tuning

Teaches a language model to answer questions about robot description files, using renders captioned by a vision-language model as training data. Joint-counting accuracy rises from 6% to 50%.

Abstract

We develop a language model that is capable of semantic reasoning with Universal Robot Description Format (URDF) files. These files describe the geometry and articulation of 3D objects. Leveraging the SAPIEN dataset of household items, we develop a model that can interpret URDF files alongside natural language queries to answer questions about object articulation, dynamics, and appearance.

Our approach involves generating image renders of URDF objects, obtaining natural language captions via a VLM, and then creating question-answer pairs for model training. We experiment with fine-tuning pre-trained large language models with different datasets while also exploring the synergy between textual and geometric data representations. We demonstrate quantitative and qualitative results showing the improvement of our fine-tuned model on both a general Q&A dataset and a joint counting task that highlights the model's ability to reason with URDFs.

Cite

@techreport{nie2024urdf,
  title = {Towards Natural Language Reasoning for Unified Robotics Description Format Files},
  author = {Neil Nie and Austin Patel and Aakash Mishra},
  year = {2024},
  institution = {Stanford University}
}