Augmented Reality HMD Research

HMD for Training of Combat Medics

We have been researching the application of augmented reality overlay on an HMD to support training of combat medics; for example, to perform a needle decompression procedure in response to a diagnosed tension pneumothorax. We developed application software, using the Unity3D framework, that reads an augmented workflow from a JSON file.

Calibration of Optical See-Through HMDs

For an optical see-through HMD, it is necessary to calibrate the display to the user’s eyes, so that augmented overlays are displayed in the correct locations with respect to the physical world. We have developed new methods for calibration of both mono and stereo display systems.


Azimi, Ehsan ; Qian, Long ; Kazanzides, Peter ; Navab, Nassir

Robust Optical See-Through Head-Mounted Display Calibration: Taking Anisotropic Nature of User Interaction Errors into Account Inproceedings

In: IEEE Virtual Reality, pp. 219-220, Los Angeles, CA, 2017.


Qian, Long ; Winkler, Alexander ; Fuerst, Bernhard ; Kazanzides, Peter ; Navab, Nassir

Modeling Physical Structure as Additional Constraints for Stereoscopic Optical See-Through Head-Mounted Display Calibration Inproceedings

In: IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp. 154-155, Merida, Mexico, 2016.


Qian, Long ; Winkler, Alexander ; Fuerst, Bernhard ; Kazanzides, Peter ; Navab, Nassir

Reduction of Interaction Space in Single Point Active Alignment Method for Optical See-Through Head-Mounted Display Calibration Inproceedings

In: IEEE Intl. Symp. on Mixed and Augmented Reality (ISMAR), pp. 156-157, Merida, Mexico, 2016.


Sensor Fusion to Overcome Occlusions in HMD Applications

Augmented reality overlays require a method to track the position of the user’s head (HMD) with respect to objects in the environment. For example, when overlaying tumor margins for image-guided neurosurgery, it is necessary to know the location of the HMD with respect to the patient. This tracking is generally provided by an optical tracking system (i.e., one or more cameras), which can be mounted on the HMD (“inside-out” tracking) or externally (“outside-in” tracking). In either case, it is possible for the camera view to be fully or partially occluded. We developed a method to handle these occlusions by performing sensor fusion of measurements from the optical tracker and from an inertial measurement unit (IMU). The method estimates the bias of the inertial sensors when the optical tracker provides full 6 degree-of-freedom (DOF) pose information. As long as the position of at least one marker can be tracked by the optical system, the 3-DOF position can be combined with the orientation estimated from the inertial measurements to recover the full 6-DOF pose information. When all the markers are occluded, the position tracking relies on the inertial sensors which are bias-corrected by the optical tracking system. Our experiments demonstrate that this approach can effectively handle long periods (at least several minutes) of partial occlusion and relatively short periods (up to a few seconds) of total occlusion.


He, Changyu ; Kazanzides, Peter ; Sen, Hasan Tutkun ; Kim, Sungmin ; Liu, Yue

An Inertial and Optical Sensor Fusion Approach for Six Degree-of-Freedom Pose Estimation Journal Article

In: Sensors, 15 (7), pp. 16448-16465, 2015.


He, Changyu ; Sen, Tutkun H; Kim, Sungmin ; Sadda, Praneeth ; Kazanzides, Peter

Fusion of Inertial Sensing to Compensate for Partial Occlusions in Optical Tracking Systems Inproceedings

In: MICCAI Workshop on Augmented Environments for Computer-Assisted Interventions (AE-CAI), pp. 60-69, Springer LNCS 8678, Boston, MA, 2014.


HMD for Image-Guided Neurosurgery

In surgery, the goal is to enable a surgeon, wearing the HMD, to perform image-guided surgery while keeping hands and eyes focused on the patient. The developed system does not display high-resolution preoperative images on the HMD, but rather shows simple graphics derived from the navigation information. For example, the navigation information can include models of the patient anatomy that are obtained from preoperative images, such as biopsy target points and tumor outlines. The system presents one or more “picture-in-picture” virtual views of the preoperative data and shows the positions of tracked instruments with respect to these views.


Sadda, Praneeth ; Azimi, Ehsan ; Jallo, George ; Doswell, Jayfus ; Kazanzides, Peter

Surgical navigation with a head-mounted tracking system and display Inproceedings

In: Medicine Meets Virtual Reality (MMVR), San Diego, CA, 2013.


Azimi, Ehsan; Doswell, Jayfus; Kazanzides, Peter

Augmented reality goggles with an integrated tracking system for navigation in neurosurgery Inproceedings

In: IEEE Virtual Reality, pp. 123-124, Orange County, CA, 2012.