An Event-Based Perception Pipeline for a Table Tennis Robot

Table tennis robots gained traction over the last years and have become a popular research challenge for control and perception algorithms. Fast and accurate ball detection is crucial for enabling a robotic arm to rally the ball back successfully. So far, most table tennis robots use conventional, frame-based cameras for the perception pipeline. While frame-based cameras have great advantages, they do suffer from motion blur if the frame rate is not high enough for fast-moving objects. Event-based cameras, on the other hand, do not suffer from this limitation since pixels report changes in intensity asynchronously and independently, leading to an event stream with a temporal resolution in µs. To the best of our knowledge, we present the first real-time perception pipeline for a table tennis robot that uses only event-based cameras. We show that compared to a frame-based pipeline, event-based perception pipelines give an order of magnitude more position estimates of the ball, which is beneficial for the estimation and prediction of the ball's position, velocity, and spin, resulting in lower errors and uncertainties. This is an advantage for the robot control, which has to be fast, given the short time a table tennis ball is flying until the robot has to hit back.

Special thanks to Sony AI for partially funding this project. We would also like to thank Mario Laux for his help with the C++ implementation of the ball detector. Thanks to Bernd Pfrommer for his open source contributions to many (event-based vision) ROS packages, two of them used in this work.