An Intelligent Human–Robot Interactive Mobile Grasping Robot Based on Visual Gesture Recognition

📅 2018 🔖 Competition work

To address the limitations of hand gesture interaction methods in practical scenarios, such as the complexity of sensor-based interaction and the high cost and distance constraints of 3D vision reconstruction, this work developed an intelligent human–robot interactive mobile grasping robot based on monocular vision–based hand gesture detection and recognition.

In terms of the hardware system, a tracked mobile platform is employed as the carrier, integrating an Arduino control board, a servo-driven mechanical gripper, a PcDuino with a camera module for real-time visual monitoring, and wireless communication technology. The Arduino is responsible for executing motion commands of the tracked vehicle and controlling the opening and closing of the mechanical gripper. The PcDuino captures environmental video and transmits the video stream to the PC via mjpg-streamer, thereby enabling remote monitoring of the robot. On the PC side, corresponding hand gestures are performed based on the remotely monitored environment and then recognized, after which the information is transmitted wirelessly to the Arduino board to interactively control the robot's movement and grasping.

In terms of hand gesture detection and recognition, a lightweight SSD-based network is adopted to achieve real-time detection and recognition of multiple hand gestures, including oplam (open_palm), cplam (close_palm), fist, and ok. Specifically, the cplam gesture is used to control the motion of the tracked mobile robot, where the relative position of the gesture center with respect to the image center determines the movement direction. The oplam, fist, and ok gestures are employed to control the opening and closing of the mechanical gripper and to stop the motion of the mobile robot, respectively.

The following video is a test demo, and the original source code has been released on github.

Acknowledgements

Thanks to the support of the Institute of Robotics and Intelligent Systems, Wuhan University of Science and Technology, and Associate Professor Lei Bin. The team members include Gao Xumin, Xiao Chao, Wang Huayu, and Lei Bin.