Telerobotic Satellite Servicing

The Satellite Servicing Projects Division (SSPD) at the NASA Goddard Space Flight Center (GSFC) is developing technology for on-orbit robotic servicing of satellites, using human operators located on Earth to control the remote robot. Much of our effort has focused on one crucial step in on-orbit refueling, which is to gain access to the satellite’s fuel ports by removing a portion of the multilayer insulation (MLI) that protects the outside of the satellite body. We adopted a model-based approach (see Figure), where the operator works with a simulated model and the results of the simulation are streamed into space. The robot in space uses sensor-based control to attempt to recreate the simulation, and contains a task monitor to detect when it has failed.

Model-based architecture; operator at da Vinci master console interacts with simulated model, receiving haptic feedback from virtual fixtures. Teleoperation commands are streamed (with time delay) to remote robot (WAM robot in this setup), which uses hybrid position/force control based on task frame model. Remote robot uses sensor feedback to update model and to detect task failures.

We demonstrated this approach on a ground-based mockup, where we use the master console of an open-source research version of the da Vinci Surgical Robot to teleoperate the remote robot, which is
a Whole Arm Manipulator (WAM) robot or Universal Robot UR5 at JHU or a Motoman robot at GSFC or the West Virginia Robotic Technology Center (WVRTC). We performed several multi-user studies with the JHU testbed, with Institutional Review Board (IRB) approval, as described below.

Our published work is based on SSPD’s initial plan to use a sharp blade (similar to a seam ripper) to cut the Kapton tape that holds the MLI patch over the access panel. Thus, we created an augmented reality user interface, initially presented at IROS 2012, to enable the operator to define a virtual fixture plane on the master console to represent the satellite surface. In later work, presented at ICRA 2015, we developed a method to enable an operator to more accurately align the virtual plane by dynamically texturing the virtual plane with the purpose of adding visual distortion until the virtual plane is accurately aligned with the real one. Once the virtual plane is defined, it provides haptic feedback to the operator and prevents the operator from moving beyond the plane or from changing the orientation of the cutter with respect to the plane normal (rotation within the plane; i.e., around the plane normal, may be permitted). On the remote robot, the user-specified plane determines the task frame for hybrid position/force control; specifically, the plane normal defines the direction for force control. Thus, while the operator interacts with the simulated environment, the remote robot uses sensor-based control to attempt to reproduce this simulation, as presented at ICRA 2013. We validated this approach in our first multi-user study, which was presented at ICRA 2015. We also developed a method to update the task frame (i.e., alignment of the plane model) used for hybrid position/force control based on the actual trajectory followed by the robot, which is necessary to ensure that the cutter remains correctly oriented with the plane. This required a method to estimate the surface compliance to compensate for variations in the normal force. The developed method and results were presented at ICRA 2016.

Virtual fixtures (VF) on plane: Line VF (left), Non-Holonomic VF (right)

In addition to the virtual fixture plane, we tested several virtual fixtures to better constrain motion on the plane. The left figure shows a line virtual fixture where the operator adjusts the line by pressing the clutch pedal (to temporarily disengage teleoperation) and then uses the master manipulator to change the orientation of the line with respect to the plane (i.e., a “masters as mice” interaction). We also created a non-holonomic constraint (NHC), which is similar to the line VF except that the operator adjusts the line direction by “steering” (similar to driving a car) and a non-holonomic virtual fixture (NHVF), where the operator is able to deviate from the line, but the system applies a virtual fixture to help steer back toward the line. The right figure shows a visualization of the NHVF, where the curved blue line depicts the VF guidance path back toward the line. We performed multi-user trials of these virtual fixtures and presented results at IROS 2015.

As shown in the top figure, we developed a Task Monitor to enable the remote robot to detect cutting failures to avoid having to wait for the operator to recognize the problem in the delayed video feedback and provide corrective action via time-delayed control. We experimentally determined that the expected force in the direction of cutting could be sufficiently described by kinetic friction plus a constant force due to the cutting process. This simple model should be feasible to evaluate even with the limited computational resources available in space. The concept is that the on-orbit robot system would use the model to estimate the expected force in the direction of cutting, and stop motion if the measured force is significantly higher or lower than the expected force. Initial results were published in Haptics 2014, followed by an improved method that included an on-line estimation of the model parameters (i.e., friction and cutting force) presented at IROS 2015.

In parallel to the research in model-based control, we investigated alternative visualization strategies. All of the above experiments utilized an augmented reality visualization where virtual objects, such as graphical representations of the virtual fixtures or feedback from the remote task monitor, are overlayed on the delayed video images. In addition to the problems due to the time delay, this approach restricts the interface to one of the available camera views, which is often sub-optimal and unintuitive because it is frequently provided by a camera mounted on the robot end-effector (i.e., an “eye in hand” configuration). Thus, we developed an augmented virtuality visualization, where the operator primarily visualizes the 3D model of the scene, which can be presented in stereo and from any perspective. This model is augmented by projections of the live (delayed) video onto the 3D model. For our development and testing, we are using a small-scale satellite model that was built at JHU.

Augmented Virtuality Visualization
Augmented Virtuality Visualization. Visualization of 3D model created from image survey of satellite, with robot tool camera image projected onto model.


Vagvolgyi, Balazs; Niu, Wenlong; Chen, Zihan; Wilkening, Paul; Kazanzides, Peter

Augmented Virtuality for Model-Based Teleoperation Inproceedings

In: IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS), Vancouver, Canada, 2017.


Li, Xiao ; Kazanzides, Peter

Task Frame Estimation during Model-Based Teleoperation for Satellite Servicing Inproceedings

In: IEEE Intl. Conf on Robotics and Automation (ICRA), pp. 2834-2839, Stockholm, Sweden, 2016.


Li, Xiao ; Kazanzides, Peter

Parameter Estimation and Anomaly Detection while Cutting Insulation during Telerobotic Satellite Servicing Inproceedings

In: IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pp. 4562-4567, Hamburg, Germany, 2015.


Vozar, Steve ; Chen, Zihan ; Kazanzides, Peter ; Whitcomb, Louis L

Preliminary Study of Virtual Nonholonomic Constraints for Time-Delayed Teleoperation Inproceedings

In: IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS), pp. 4244-4250, Hamburg, Germany, 2015.


Vozar, Steve; Leonard, Simon; Kazanzides, Peter; Whitcomb, Louis

Experimental Evaluation of Force Control for Virtual-Fixture-Assisted Teleoperation for On-Orbit Manipulation of Satellite Thermal Blanket Insulation Inproceedings

In: IEEE Intl. Conf on Robotics and Automation (ICRA), pp. 4424-4431, Seattle, WA, 2015.


Kandaswamy, Isha ; Xia, Tian ; Kazanzides, Peter

Strategies and models for cutting satellite insulation in telerobotic servicing missions Inproceedings

In: IEEE Haptics Symposium, pp. 467-472, Houston, TX, 2014.


Xia, T; Leonard, S; Kandaswamy, I; Blank, A; Whitcomb, L L; Kazanzides, P

Model-Based Telerobotic Control with Virtual Fixtures For Satellite Servicing Tasks Inproceedings

In: IEEE Intl. Conf. on Robotics and Automation (ICRA), Karlsruhe, Germany, 2013.


Xia, Tian ; Leonard, Simon ; Deguet, Anton ; Whitcomb, Louis L; Kazanzides, Peter

Augmented Reality Environment with Virtual Fixtures for Robotic Telemanipulation in Space Inproceedings

In: IEEE/RSJ Intl. Conf. on Intell. Robots and Systems (IROS), pp. 5059-5064, Vilamoura, Portugal, 2012.