FIDO the Robot Dog
Learning Dynamics Models via In-the-Wild Fetch
1 Stanford University 2 Toyota Research Institute
PaperAbstract
Learning generalizable dynamics models requires diverse, grounded training data. Existing approaches trained on internet-sourced data are constrained by the availability of high-quality samples, while the diversity of real-world data collection efforts is limited by infrastructure and operator demands. In our work, we propose collecting dynamics data through natural, embodied robot interactions. We present Fetch Interaction for Dynamics Observation (FIDO), a robot quadruped that collects in-the-wild dynamics data by playing fetch. FIDO first observes a ball trajectory thrown by a user. Then, it retrieves the ball and returns it, creating a naturally engaging and repeatable interaction that yields grounded observations of object dynamics. We show that a dynamics model trained on this data is directly useful for both unseen domain-adjacent sports tasks and robot planning, suggesting that in-the-wild interaction diversity is a key driver of learning generalizable physical dynamics.
Learning generalizable dynamics models requires diverse, grounded training data. Existing approaches trained on internet-sourced data are constrained by the availability of high-quality samples, while the diversity of real-world data collection efforts is limited by infrastructure and operator demands. In our work, we propose collecting dynamics data through natural, embodied robot interactions. We present Fetch Interaction for Dynamics Observation (FIDO), a robot quadruped that collects in-the-wild dynamics data by playing fetch. FIDO first observes a ball trajectory thrown by a user. Then, it retrieves the ball and returns it, creating a naturally engaging and repeatable interaction that yields grounded observations of object dynamics, including the arc of throws, the effect of gravity, and the variability of real-world conditions. We collect FIDO data across N=18 users in three in-the-wild environments. A projectile dynamics model trained on this data exhibits strong transfer to unseen domain-adjacent tasks. FIDO outperforms a pretrained VLM baseline by 29.1% on shuttlecock trajectory prediction in badminton and by 26.8% on ball trajectory prediction in baseball, with both results improving monotonically with the diversity of collected fetch data. Our learned dynamics model is also directly useful for robot planning: a robot plans through these learned dynamics to catch a balloon before it hits the ground, where the addition of FIDO data improves catching performance by 20.0%. These results suggest that in-the-wild interaction diversity is a key driver of learning generalizable physical dynamics.
Fetch Interaction 
The fetch interaction is intuitive and naturally repetitive, lowering the barrier to user participation. Critically, it requires no specialized training or purpose-built infrastructure, enabling in-the-wild data collection across diverse users and environments.
In-the-wild Deployment
We deploy this system to collect data across interactions with 18 users in three environments. During each round of fetch, the robot gathers video data of the dynamics of the ball as it falls through the air and bounces on the ground.
Dynamics Video Data
The resulting data captures a greater diversity of physical conditions than a laboratory setting allows, such as variety in backgrounds, lighting, surface types, and throw styles.
Results: Sports Trajectory Prediction
We also show that FIDO outperforms a pre-trained VLM baseline by 33.8% on shuttlecock trajectory prediction in badminton and by 15.4% on ball trajectory prediction in baseball. Model performance improves monotonically with the diversity of data, demonstrating the need for diverse data collection conditions to learn generalizable dynamics.
Results: Robot Balloon Catching 
We consider the task of catching a thrown balloon in a basket attached to the head of the robot, testing the generalization capabilities of FIDO to a novel dynamics setup. Because of the limited precision of the robot while moving at high speeds, the robot strongly benefits from using the dynamics model predictions to proactively move to where the balloon will land. Using a dynamics model fine-tuned with FIDO data yields a 29.4% improvement in balloon catching over baselines trained without FIDO data.