FIDO the Robot Dog

Learning Dynamics Models via In-the-Wild Fetch

Michelle Pan1, Maximilian Du1, Sriram Yenamandra1, Ajay Sridhar1, Mark Zolotas2, Dorsa Sadigh1, Jennifer Grannen1

1 Stanford University 2 Toyota Research Institute

Paper

Abstract

Learning generalizable dynamics models requires diverse, grounded training data. Existing approaches trained on internet-sourced data are constrained by the availability of high-quality samples, while the diversity of real-world data collection efforts is limited by infrastructure and operator demands. In our work, we propose collecting dynamics data through natural, embodied robot interactions. We present Fetch Interaction for Dynamics Observation (FIDO), a robot quadruped that collects in-the-wild dynamics data by playing fetch. FIDO first observes a ball trajectory thrown by a user. Then, it retrieves the ball and returns it, creating a naturally engaging and repeatable interaction that yields grounded observations of object dynamics. We show that a dynamics model trained on this data is directly useful for both unseen domain-adjacent sports tasks and robot planning, suggesting that in-the-wild interaction diversity is a key driver of learning generalizable physical dynamics. [expand]

Fetch Interaction  

The fetch interaction is intuitive and naturally repetitive, lowering the barrier to user participation. Critically, it requires no specialized training or purpose-built infrastructure, enabling in-the-wild data collection across diverse users and environments.


In-the-wild Deployment

We deploy this system to collect data across interactions with 18 users in three environments. During each round of fetch, the robot gathers video data of the dynamics of the ball as it falls through the air and bounces on the ground.


Dynamics Video Data

The resulting data captures a greater diversity of physical conditions than a laboratory setting allows, such as variety in backgrounds, lighting, surface types, and throw styles.


Results: Sports Trajectory Prediction    

We also show that FIDO outperforms a pre-trained VLM baseline by 33.8% on shuttlecock trajectory prediction in badminton and by 15.4% on ball trajectory prediction in baseball. Model performance improves monotonically with the diversity of data, demonstrating the need for diverse data collection conditions to learn generalizable dynamics.


Results: Robot Balloon Catching  

We consider the task of catching a thrown balloon in a basket attached to the head of the robot, testing the generalization capabilities of FIDO to a novel dynamics setup. Because of the limited precision of the robot while moving at high speeds, the robot strongly benefits from using the dynamics model predictions to proactively move to where the balloon will land. Using a dynamics model fine-tuned with FIDO data yields a 29.4% improvement in balloon catching over baselines trained without FIDO data.