Lai Sum Yim1, Quang TN Vo2, Ching-I Huang1, Chi-Ruei Wang1,
Wren McQueary2, Hsueh-Cheng Wang1, Haikun Huang2, and Lap-Fai Yu2
________________________________________________
L.S. Yim and Q.T. Vo contributed equally to this work.
1Department of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan.
2Department of Computer Science, George Mason University, USA.
Abstract
This paper presents an easy-to-deploy, virtual reality-based teleoperation system for controlling a robot arm. The proposed system is based on a consumer-grade virtual reality device (Oculus Quest 2) with a low-cost robot arm (a LoCoBot) to allow easy replication and set up. The proposed Work-from-Home Virtual Reality (WFH-VR) system allows the user to feel an intimate connection with the real remote robot arm. Virtual representations of the robot and objects to be manipulated in the real-world are presented in VR by streaming data pertaining to orientation and poses. The user studies suggest that 1) the proposed telerobotic system is effective under conditions both with and without network latency, whereas a method that simply streams video does not. This design enables the system implemented at an arbitrary distance from the actual work site. 2) The proposed system allows novices to perform manipulation tasks requiring higher dexterity than traditional keyboard controls can support, such as setting tableware. https://arg-nctu.github.io/projects/vr-robot-arm.html.
Video
System Overview
The proposed system comprises two operations: controlling the robot arm and synchronizing virtual objects with real-world objects. We employed Unity PUN2 for all communication tasks over the internet and an Oculus Quest 2 VR headset to visualize the virtual environment while manipulating the robot arm. Within the virtual environment, real-world objects are replicated via 3D reconstruction or CAD modeling. The actual robot arm and gripper are controlled by guiding an identical virtual robot arm (i.e., an arm with the same joint configuration and dimensions as the real arm). These movements are synchronized for both the VR user (client) and the physical robot's environment (host) in the virtual scene. The interface also provides a video signal streamed from the workplace via Unity PUN2.
VR Robot Teleoperation across the Globe
We tested the proposed VR teleoperation platform between two universities across the globe. The VR side was set up at George Mason University in Virginia, USA. The real robot side was set up at National Yang Ming Chiao Tung University in Taiwan. The user in USA put on a VR headset and controlled a LoCoBot robot arm in Taiwan.
Conclusion
1. Presents a advanced approach to the VR teleoperation of a robot arm.
2. Proposed system allows even 52 novices to perform tasks of high dexterity
3. Carry out 3 Human-robot-interaction Experiments
More details you can find in the paper.
Bibtex
@inproceedings{yim2022wfhvr ,
title={WFH-VR: Teleoperating a Robot Arm to set a Dining Table across the Globe via Virtual Reality},
author={Yim, Lai Sum and Vo, Quang TN and Huang, Ching-I and Wang, Chi-Ruei and McQueary, Wren and Wang, Hsueh-Cheng and Huang, Haikun and Yu, Lap-Fai},
booktitle={2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
year={2022},
organization={IEEE}
}
Acknowledgments
The research was supported by Taiwan's National Science and Technology Council (grants 111-NU-E-A49-001-NU, 110-2221-E-A49-124, and 111R10093Y-2). This work was funded in part by Qualcomm through the Taiwan University Research Collaboration Project. This work was supported by an NSF CAREER award (award\#: 1942531) and an NSF FTW-HTF-R grant (award\#: 2128867).
