Notably, a large part of the promise of autonomous robots depends on long-term operation in domains shared with humans and other agents. These environments are typically highly complex, semantically rich, and highly dynamic with agents frequently moving through and interacting with the scene.
This talk presents an autonomy pipeline combining perception, prediction, and planning to address these challenges. We first present methods to detect and represent complex semantics, short-term motion, and long-term changes for real-time robot perception in a unified framework called Khronos. We then show how Dynamic Scene Graphs DSGs can represent semantic symbols in a task-driven fashion and facilitate reasoning about the scene, such as the prediction of likely future outcomes based on the data the robot has already collected.
Lastly, we show how robots as embodied agents can leverage these actionable scene representations and predictions to complete tasks such as actively gathering data that helps them improve their world models, perception, and action capabilities fully autonomously over time.
The presented methods are demonstrated on-board fully autonomous aerial, legged, and wheeled robots, run in real-time on mobile hardware, and are available as open-source software. Roland Siegwart, where he also obtained his M. Marc Pollefeys. His research focuses on active and passive perception of complex, dynamic, and changing environments for robotic interaction and augmented reality. This includes research on dense geometric and semantic scene representations, scene abstraction and understanding, as well as detection and prediction of moving and changing entities to enable continuous improvement of a robot's scene model and perception capabilities.
This talk will explore the robustness challenges in deep learning and large language models LLMs , demonstrating how seemingly minor perturbations can lead to critical failures.
