- 1 HMD for the First Assistant in robotic surgery
- 2 HMD for Training of Medical Procedures
- 3 Calibration of Optical See-Through HMDs
- 4 Sensor Fusion to Overcome Occlusions in HMD Applications
- 5 HMD for Image-Guided Neurosurgery
HMD for the First Assistant in robotic surgery
In robotic-assisted laparoscopic surgery, the first assistant (FA) stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. We are developing ARssist, an augmented reality application based on an optical see-through head-mounted display, to aid the FA. ARssist provides visualization of the robotic instruments and endoscope “inside” the patient body, and various ways to render the stereo endoscopy on the head-mounted display, as shown in the Figure below.
ARAMIS: Augmented Reality Assistance for Minimally Invasive Surgery Using a Head-Mounted Display Inproceedings
In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 74-82, 2019.
Augmented Reality Assisted Instrument Insertion and Tool Manipulation for the First Assistant in Robotic Surgery Inproceedings
In: IEEE Intl. Conf. on Robotics and Automation (ICRA), pp. 5173-5179, 2019.
In: IET Healthcare Technology Letters, vol. 5, no. 5, pp. 194-200, 2018.
HMD for Training of Medical Procedures
We have been researching the application of augmented reality overlay on an HMD to support training of medical procedures. Our initial project was to support training of combat medics; for example, to perform a needle decompression procedure in response to a diagnosed tension pneumothorax. We developed application software, using the Unity3D framework, that reads an augmented workflow from a JSON file. More recently, we are applying this to training of neurosurgical procedures, such as ventriculostomy.
Can Mixed-Reality Improve the Training of Medical Procedures? Inproceedings
In: IEEE Engin. in Medicine and Biology Conf. (EMBC), pp. 4065-4068, 2018.
Evaluation of Optical See-Through Head-Mounted Displays in Training for Critical Care and Trauma Inproceedings
In: IEEE Virtual Reality (VR), pp. 96-97, 2018.
Calibration of Optical See-Through HMDs
For an optical see-through HMD, it is necessary to calibrate the display to the user’s eyes, so that augmented overlays are displayed in the correct locations with respect to the physical world. We have developed new methods for calibration of both mono and stereo display systems.
Robust Optical See-Through Head-Mounted Display Calibration: Taking Anisotropic Nature of User Interaction Errors into Account Inproceedings
In: IEEE Virtual Reality, pp. 219-220, Los Angeles, CA, 2017.
Modeling Physical Structure as Additional Constraints for Stereoscopic Optical See-Through Head-Mounted Display Calibration Inproceedings
In: IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp. 154-155, Merida, Mexico, 2016.
Reduction of Interaction Space in Single Point Active Alignment Method for Optical See-Through Head-Mounted Display Calibration Inproceedings
In: IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp. 156-157, Merida, Mexico, 2016.
Sensor Fusion to Overcome Occlusions in HMD Applications
Augmented reality overlays require a method to track the position of the user’s head (HMD) with respect to objects in the environment. For example, when overlaying tumor margins for image-guided neurosurgery, it is necessary to know the location of the HMD with respect to the patient. This tracking is generally provided by an optical tracking system (i.e., one or more cameras), which can be mounted on the HMD (“inside-out” tracking) or externally (“outside-in” tracking). In either case, it is possible for the camera view to be fully or partially occluded. We developed a method to handle these occlusions by performing sensor fusion of measurements from the optical tracker and from an inertial measurement unit (IMU). The method estimates the bias of the inertial sensors when the optical tracker provides full 6 degree-of-freedom (DOF) pose information. As long as the position of at least one marker can be tracked by the optical system, the 3-DOF position can be combined with the orientation estimated from the inertial measurements to recover the full 6-DOF pose information. When all the markers are occluded, the position tracking relies on the inertial sensors which are bias-corrected by the optical tracking system. Our experiments demonstrate that this approach can effectively handle long periods (at least several minutes) of partial occlusion and relatively short periods (up to a few seconds) of total occlusion.
An Inertial and Optical Sensor Fusion Approach for Six Degree-of-Freedom Pose Estimation Journal Article
In: Sensors, vol. 15, no. 7, pp. 16448-16465, 2015.
Fusion of Inertial Sensing to Compensate for Partial Occlusions in Optical Tracking Systems Inproceedings
In: MICCAI Workshop on Augmented Environments for Computer-Assisted Interventions (AE-CAI), pp. 60-69, Springer LNCS 8678, Boston, MA, 2014.
HMD for Image-Guided Neurosurgery
In surgery, the goal is to enable a surgeon, wearing the HMD, to perform image-guided surgery while keeping hands and eyes focused on the patient. The developed system does not display high-resolution preoperative images on the HMD, but rather shows simple graphics derived from the navigation information. For example, the navigation information can include models of the patient anatomy that are obtained from preoperative images, such as biopsy target points and tumor outlines. The system presents one or more “picture-in-picture” virtual views of the preoperative data and shows the positions of tracked instruments with respect to these views.
An Interactive Mixed Reality Platform for Bedside Surgical Procedures Inproceedings
In: Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 65-75, 2020.
Interactive Navigation System in Mixed-Reality for Neurosurgery Inproceedings
In: IEEE Virtual Reality (VR), pp. 783-784, 2020.
Interactive Training and Operation Ecosystem for Surgical Tasks in Mixed Reality Inproceedings
In: MICCAI Workshop on OR 2.0 Context-Aware Operating Theaters, pp. 20-29, 2018.
Surgical navigation with a head-mounted tracking system and display Inproceedings
In: Medicine Meets Virtual Reality (MMVR), San Diego, CA, 2013.
Augmented reality goggles with an integrated tracking system for navigation in neurosurgery Inproceedings
In: IEEE Virtual Reality, pp. 123-124, Orange County, CA, 2012.
JHU Hub Article, ‘Mixed Reality’ Makes for Better Surgeons, May 17, 2019
WUSA, Channel 9 (Washington DC), TV News Story, July 31, 2019