WHC2015

Half-day workshop on
"Cutaneous Feedback for Teleoperation in Medical Robotics"

The workshop took place on Monday, June 22, 2015 during the 2015 IEEE World Haptics Conference in Chicago, USA.

Abstract

Telerobotic surgical systems involve a slave robot, which interacts with the patient, and a master console, operated by the human surgeon. The slave robot reproduces the hand movements of the surgeon, who in turn needs to observe the operative environment with which the robot is interacting. The latter can be achieved by a combination of visual and haptic cues that flow from the operating table to the surgeon. Visual feedback is already available in commercial robotic surgery systems (e.g., the Intuitive Surgical da Vinci Si), but current surgical robots have very limited haptic feedback.

This omission is due to multiple reasons, from the negative effect that haptic feedback has on the stability of the system to the difficulty of including accurate sensors in the robotic instruments due to sterilization and cost requirements. However, haptic feedback is still widely considered a valuable addition to teleoperated surgical systems.

For this reason, it is paramount to study and develop new systems to provide surgeons with haptic feedback from the operating tools, while guaranteeing the safety of the patient. In this sense, cutaneous feedback has recently received great attention from researchers; delivering ungrounded haptic cues to the surgeon’s skin conveys rich information and does not affect the stability and the safety of the teleoperation system.

The aim of this workshop is to bring together researchers from haptics and surgical robotics to discuss current research and future directions, to bring haptics in the operating room and improve the performance of current surgical robotic systems.

Audience

Topics include, but are not limited to, cutaneous haptic sensing and rendering systems, robotic teleoperation, safety and ethical issues in robot-assisted surgery and related medical scenarios. The workshop is open to any student, researcher as well as developer and end user interested in the design, development, and use of cutaneous feedback for teleoperated medical procedures.

We also intend to provide ample opportunity for round-table discussions where invited speakers and workshop participants will be encouraged to propose questions and ponder the next greatest challenges for cutaneous feedback in medical robotics.

Program and Slides

9.00 – 9.05 Welcome by the organizers
Slides
9.05 – 9.25 Dr. Antonio Gangemi (Univ. of Illinois Medical Center), “Robotic Training for General Surgery Residents at UIC”
Abstract and Slides

Abstract
Introduction: The working hours restrictions have decreased the operative experience of general surgery residents (GSRs). The surgical skills laboratory (SSL) might overcome this training limitation by allowing GSRs to achieve basic laparoscopic (lap) and robotic (rob) surgery competency in a risk-free environment. With this priority in mind, we implemented a standardized, proficiency-based curriculum for GSRs.
Methods: PGY1s and PGY3s must pass a criterion-referenced test on the “dry” laparoscopic station (DLS) and the robotic simulator (RS) respectively before advancing to the training on the live porcine model. PGY2s and PGY4s are evaluated in the “wet” lap and rob lab settings with the “OSATS” and “CRSA Score” respectively. An anonymous survey was administered to get GSRs feedback on the proficiency-based curriculum and the overall quality of the program.
Results: 2/6 PGY1s failed the “precision cutting” task and required an additional testing session to achieve full proficiency. 3/6 PGY2s passed the final test (average OSATS score 30/35) whilst other 3 required an additional testing session to achieve full proficiency. The 6 PGY3s required an average of 4.2 sessions to achieve proficiency on at least one of the 5 exercises included in the RS curriculum. The highest and lowest passing rate was 85.7% for “camera target 1” and 28.5% for the “tubes 1” exercise. 5/6 PGY4 passed the test on the live porcine model (average CRSA Score 35/50; min 18, max 49). 80.7% of the GSRs responded to the anonymous survey and 71% rated the overall quality of the program good, very good or excellent whilst 85% requested more lap and rob exposure.
Conclusions: The standardized, proficiency-based curriculum is time consuming and logistically demanding but overall feasible and well perceived by the GSRs. Further studies are needed to investigate the predictive validity of our findings in terms in the clinical setting.

Slides of Dr. Antonio Gangemi References
9.25 – 9.45 Dr. Lawton Verner and Dr. Ryan Steger (Intuitive Surgical, Inc.), “Challenges to Adding Haptic Fedback to Surgical Robots”
Abstract and Slides

Abstract
The addition of haptic feedback to teleoperated surgical systems is of interest to both the academic research community and commercial industry. While some success has been realized with adding haptic feedback to teleoperators in the academic setting, commercial use of these technologies has been limited. This presentation will highlight the challenges in commercializing a haptic feedback system for surgical teleoperators. More specifically, we will detail some of the clinical requirements and design constraints (such as cost, safety, reliability) and how they pertain to incorporating haptic feedback into a surgical teleoperator.

Link http://www.intuitivesurgical.com/

Slides of Dr. Lawton Verner and Dr. Ryan Steger
9.45 – 10.05 Prof. Cagatay Basdogan (Koç Univ.), “Challenges in Characterization of Soft Tissue Material Properties”
Abstract and Slides

Abstract
The current systems designed for tele-operated medical surgery lack haptic feedback. In other words, the surgeon operating in front of the master console utilizes visual cues to command slave robots operating on a patient. However, due to the complex nature of soft tissue material properties, the visual information is insufficient and can be misleading. On the other hand, characterization of soft tissue material properties, especially in-vivo, is highly challenging. In this talk, I will focus on the characterization of soft tissue material properties and attempt to convince the audience that haptic feedback to the surgeon is not only helpful, but also necessary.

References
  • Ayyildiz, M., Aktas, R.G., Basdogan, C., 2014, “Effect of Solution and Post-Mortem Time on Mechanical and Histological Properties of Liver During Cold Preservation”, Biorheology, Vol. 51, No.1, pp. 47-70.
  • Yarpuzlu, B., Ayyildiz, M., Tok, O.E., Aktas, R.G., Basdogan, C., 2014, “Correlation Between the Mechanical and Histological Properties of Liver Tissue”, Journal of the Mechanical Behavior of Biomedical Materials, Vol. 29, pp. 403-416.
  • Ayyildiz, M., Guclu, B., Yildiz, M.Z., Basdogan, C., 2013, "An Opto-Electro-Mechanical Tactile Sensor for Detection of Breast Lumps", IEEE Transactions on Haptics, Vol. 6, No. 2, pp. 145-155.
  • Gokgol, C., Basdogan, C., Canadinc, D., 2012, “Estimation of Fracture Toughness of Liver Tissue: Experiments and Validation”, Medical Engineering and Physics, Vol. 34, No. 7, pp. 882-891.
  • Basdogan, C., 2012, “Dynamic Material Properties of Human and Animal Livers”, Soft Tissue Biomechanical Modeling for Computer Assisted Surgery, Springer Series on Studies in Mechanobiology, Tissue Engineering, and Biomaterials, Vol. 11, pp. 229-241, Ed: P. Yohan.
  • Ozcan, M. U., Ocal, S., Basdogan, C., Dogusoy, G., Tokat, Y., 2011, “Characterization of Frequency-Dependent Material Properties of Human Liver and its Pathologies Using an Impact Hammer”, Medical Image Analysis, Vol. 15, No.1, pp. 45-52.
  • Peterlik, I., Sedef, M., Basdogan, C., Matyska, L., 2010, “Real-time Visio-Haptic Interaction with Static Soft Tissue Models Having Geometric and Material Nonlinearity”, Computers and Graphics, Vol. 34, No.1, pp. 43-54.
  • Samur, E., Sedef, M., Basdogan, C., Avtan, L., Duzgun, O., 2007, “A Robotic Indenter for Minimally Invasive Measurement and Characterization of Soft Tissue Behavior”, Medical Image Analysis, Vol. 11, No.4, pp. 361-373.
Slides of Prof. Cagatay Basdogan
10.05 – 10.15 Panel discussion
10.15 – 10.30 Coffee Break
10.30 – 10.50 Dr. Claudio Pacchierotti (Italian Inst. of Technology) and Prof. Domenico Prattichizzo (Univ. of Siena and Italian Inst. of Technology),
“Cutaneous Feedback of Fingertip Deformation and Vibration for Palpation in Robotic Surgery”
Abstract and Slides

Abstract
Despite its expected clinical benefits, current teleoperated surgical robots do not provide the surgeon with haptic feedback largely because grounded forces can destabilize the system’s closed-loop controller. This work presents an alternative approach that enables the surgeon to feel fingertip contact deformations and vibrations while guaranteeing the teleoperator's stability. We implemented our cutaneous feedback solution on an Intuitive Surgical da Vinci Standard robot by mounting a SynTouch BioTac tactile sensor to the distal end of a surgical instrument and a custom cutaneous display to the corresponding master controller. As the user probes the remote environment, the contact deformations, DC pressure, and AC pressure (vibrations) sensed by the BioTac are directly mapped to input commands for the cutaneous device’s motors using a model-free data-driven algorithm. The cutaneous display continually moves, tilts, and vibrates a flat plate at the operator’s fingertip to optimally reproduce the tactile sensations experienced by the BioTac. We tested the proposed approach by having eighteen subjects use the augmented da Vinci robot to palpate a heart model with no haptic feedback, only deformation feedback, and deformation plus vibration feedback. Fingertip deformation feedback significantly improved palpation performance by reducing the task completion time, the pressure exerted on the heart model, and the subject's absolute error in detecting the orientation of the embedded plastic stick. Vibration feedback significantly improved palpation performance only for the seven subjects who dragged the BioTac across the model, rather than pressing straight into it.

References
  • C. Pacchierotti, D. Prattichizzo, K. J. Kuchenbecker. Cutaneous Feedback of Fingertip Deformation and Vibration for Palpation in Robotic Surgery. IEEE Transactions on Biomedical Engineering, 2015.
  • C. Pacchierotti, P. Shirsat, J. K. Koehn, D. Prattichizzo, K. J. Kuchenbecker. Cutaneous Feedback of Planar Fingertip Deformation and Vibration on a da Vinci Surgical Robot. In Workshop on The Role of Human Sensorimotor Control in Surgical Robotics. IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS). Chicago, USA, 2014.
  • C. Pacchierotti, D. Prattichizzo, K. J. Kuchenbecker. A data-driven approach to remote tactile interaction: from a BioTac sensor to any fingetip cutaneous device. In Haptics: Neuroscience, Devices, Modeling, and Applications. Eurohaptics 2014, Lecture Notes in Computer Science, Pages 418-424, Versailles, France, 2014.
  • L. Meli, C. Pacchierotti, D. Prattichizzo. Sensory subtraction in robot-assisted surgery: fingertip skin deformation feedback to ensure safety and improve transparency in bimanual haptic interaction. IEEE Transactions on Biomedical Engineering, 61(4):1318-1327, 2014.
Slides of Dr. Claudio Pacchierotti
10.50 – 11.10 Prof. Allison M. Okamura (Stanford Univ.), “Tactile Skin Deformation Feedback for Conveying Environment Forces in Teleoperation”
Abstract and Slides

Abstract
This talk will describe the design, development, and experimental validation of a class of tactile devices that provide feedback using local fingerpad skin deformation. Skin deformation tactile feedback can provide force and torque information through skin deformation cues on multiple fingers in a manner consistent with our natural interaction with external objects. We have used skin deformation feedback for sensory substitution and augmentation of force and/or torque information during manipulation of objects during teleoperation, and compared it with other forms of feedback including vibration and graphics. Experiments in teleoperation with a pair of Phantom Premium haptic devices and a da Vinci Research Kit with novice and surgeon users demonstrate the modes of skin deformation feedback and types of tasks in which skin deformation tactile feedback is most useful. This talk includes work done primarily by Zhan Fan Quek and Sam Schorr, in collaboration with Allison Okamura, Will Provancher, and Ilana Nisky.

References
  • Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher and A. M. Okamura. Sensory Substitution of Force and Torque using 6-Degree-of-Freedom Tangential and Normal Skin Deformation Feedback. IEEE International Conference on Robotics and Automation, 2015. In press.
  • Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher and A. M. Okamura. Sensory Substitution and Augmentation using 3-Degree-of-Freedom Skin Deformation Feedback. IEEE Transactions on Haptics, 2015. In press.
  • Z. F. Quek, S.B. Schorr, I. Nisky, W. R. Provancher and A. M. Okamura. Sensory Substitution using 3-Degree-of-Freedom Tangential and Normal Skin Deformation Feedback. Haptics Symposium, pages 27-33, 2014.
  • Z. F. Quek, S. B. Schorr, I. Nisky, W. R. Provancher and A. M. Okamura. Augmentation of Stiffness Perception with a 1-Degree-of-Freedom Skin Stretch Device. IEEE Transactions on Human-Machine Systems, 44(6):731-742, 2014.
  • S. B. Schorr, Z. F. Quek, R. Y. Romano, I. Nisky, W. R. Provancher and A. M. Okamura. Sensory Substitution via Cutaneous Skin Stretch Feedback. In IEEE International Conference on Robotics and Automation, pages 2333-2338, 2013.
  • Z. F. Quek, S. B. Schorr, I. Nisky, A. M. Okamura and W. R. Provancher. Sensory Augmentation of Stiffness using Fingerpad Skin Stretch. In IEEE World Haptics Conference, pages 467-472, 2013.
Slides of Prof. Allison Okamura
11.10 – 11.30 Prof. Dong-Soo Kwon (KAIST), “A Novel Surgical Pen-type Master Device using Vibrotactile Feedback”
Abstract and Slides

Abstract
We have been developing medical robots aiming to minimize invasiveness for patients, and maximize convenience for surgeons: the Natural Orifice Transluminal Endoscopic Surgery Robot (K-NOTES robot), a single port surgery robot system, and a portable endoscopic tool handler (PETH). To control these robots efficiently, we have proposed a lightweight, intuitive and user-friendly pen-type master device.
The proposed pen-type master device provides 6 D.O.F. commands. The 3 dof position of the surgical instrument's end effector is obtained by using a tablet and a vision system. The remaining 3 dof orientation of the surgical instrument's wrist is obtained by IMU Sensor and a vision system. The pen-type master device may need to provide force feedback with vibrotactile feedback.
In this presentation, we will show preliminary results of vibrotactile feedback effect of the pen-type master device by comparing number of suture's cut off times with/without presence of the vibrotactile feedback. The most frequently occurring problem in surgery is breaking the suture via excessive force during suturing task. We also present effectiveness of the vibrotactile feedback for the suturing task that has been tested with a simple 1 dof tension measurement experimental equipment. The proposed pen-type master device is still in research stage verifying the stability and the efficiency in robotic surgery through computer simulations.
In the further work, our robotic surgery systems will include the vibrotactile feedback pen-type master device and a miniature disposable force sensor on the surgical instrument that has been developed in Sungkyunkwan Univ.

Slides of Prof. Dong-Soo Kwon
11.30 – 11.50 Prof. Katherine J. Kuchenbecker (Univ. of Pennsylvania), “Tactile Feedback of Tool Vibrations in Robotic Surgery”
Abstract and Slides


Abstract
Although commercial robotic surgery systems such as the Intuitive da Vinci are approved for use on human patients, they provide the surgeon with very little touch feedback. Measuring and recreating tool-tissue interaction forces is technically challenging, especially while satisfying the safety and sterility requirements of surgery. My team at Penn has invented and studied a method that enables the surgeon to feel high-frequency instrument vibrations, which indicate important transitions in manipulation contact state that are often difficult to discern visually. We mount three-axis high-bandwidth accelerometers to the robot arms under the sterile draping; their outputs drive one-axis voice-coil actuators positioned on the surgeon hand controllers. Categorizing the tool vibrations recorded during in vivo animal surgeries showed that 82% of surgical actions caused measurable vibration cues; recordings during live human surgeries also indicate the clinical feasibility of this approach. Other studies have demonstrated that surgeons doing manipulation tasks significantly prefer receiving this additional vibration feedback over the standard configuration of the da Vinci. This feedback significantly improves the surgeon's ability to detect off-camera instrument collisions, potentially improving patient safety. We have also found that experienced robotic surgeons cause significantly smaller tool vibrations than novice surgeons; this unexpected discovery led us to examine the role instrument vibration feedback can play in training residents and in estimating trainee skill.

References
  • K. J. Kuchenbecker, J. Gewirtz, W. McMahan, D. Standish, P. Martin, J. Bohren, P. J. Mendoza, and D. I. Lee. VerroTouch: High-frequency acceleration feedback for telerobotic surgery. In Astrid M. L. Kappers, Jan B. F. van Erp, Wouter M. Bergmann Tiest, and Frans C. T. van der Helm, editors, Haptics: Generating and Perceiving Tangible Sensations, Proc. EuroHaptics, Part I, volume 6191 of Lecture Notes in Computer Science, pages 189–196, Amsterdam, Netherlands, July 2010. Springer.
  • W. McMahan, J. Gewirtz, D. Standish, P. Martin, J. Kunkel, M. Lilavois, A. Wedmid, D. I. Lee, and K. J. Kuchenbecker. Tool contact acceleration feedback for telerobotic surgery. IEEE Transactions on Haptics, 4(3):210–220, July– September 2011.
  • K. Bark, E. D. Gomez, C. Rivera, W. McMahan, A. Remington, K. Murayama, D. I. Lee, K. Dumon, N. Williams, and K. J. Kuchenbecker. Surgical instrument vibrations are a construct-valid measure of technical skill in robotic peg transfer and suturing tasks. In Proc. Hamlyn Symposium on Medical Robotics, pages 50–51, London, England, 2012.
  • K. Bark, W. McMahan, A. Remington, J. Gewirtz, A. Wedmid, D. I. Lee, and K. J. Kuchenbecker. In vivo validation of a system for haptic feedback of tool vibrations in robotic surgery. Surgical Endoscopy, 27(2):656–664, February 2013. dynamic article (paper plus video), available at http://www.springerlink.com/content/417j532708417342.
  • E. D. Gomez. The Role of Haptic Tool Vibrations in Skill Acquisition and Assessment in Minimally Invasive Robotic Surgery. Thesis, Masters in Translational Research, University of Pennsylvania, May 2013.
  • W. McMahan. Providing Haptic Perception to Telerobotic Systems via Tactile Acceleration Signals. Dissertation, Doctorate of Philosophy in Mechanical Engineering and Applied Mechanics, University of Pennsylvania, May 2013.
  • W. McMahan, E. D. Gomez, L. Chen, K. Bark, J. C. Nappo, E. I. Koch, D. I. Lee, K. Dumon, N. Williams, and K. J. Kuchenbecker. A practical system for recording instrument interactions during live robotic surgery. Journal of Robotic Surgery, 7(4):351–358, December 2013.
  • J. K. Koehn and K. J. Kuchenbecker. Surgeons and non-surgeons prefer haptic feedback of instrument vibrations during robotic surgery. Accepted for publication in Surgical Endoscopy.
Slides of Prof. Katherine Kuchenbecker
11.50 – 12.00 Panel discussion

Video Recordings

The video recordings of the workshop will be soon made available to all the members of the Technical Committee on Haptics.

Organizers

Dr. Claudio Pacchierotti, Italian Institute of Technology

Claudio received the B.S. and M.S. degrees cum laude in computer engineering from the University of Siena (Italy) in 2009 and 2011, respectively. He was a visiting student at the Karlstad University (Sweden) in 2010, at the University of Padua (Italy) in 2013, at the University of Twente (The Netherlands) in 2014, and at the University of Pennsylvania (USA) in 2014. He received the Ph.D. degree in Robotics and Automation from the University of Siena and the Italian Institute of Technology in 2014. His research deals with robotics and haptics, focusing on cutaneous force feedback techniques, wearable devices, and haptics for robotic surgery.
Email: pacchierotti@diism.unisi.it



Prof. Domenico Prattichizzo, University of Siena and Italian Institute of Technology

Domenico received the M.S. degree in Electronics Engineering and the Ph.D. degree in Robotics and Automation from the University of Pisa in 1991 and 1995, respectively. He has been Associate Professor of Robotics at the University of Siena since 2002 and Scientific Consultant at Istituto Italiano di Tecnologia, Genova, Italy since 2009. In 1994, he was Visiting Scientist at the MIT AI Lab. He co-authored the Grasping chapter of Handbook of Robotics Springer, 2008, which was awarded two PROSE Awards by the American Association of Publishers. From 2003 to 2014, he has been Associate Editor in Chief of the IEEE Transactions on Haptics. From 2003 to 2007, he was Associate Editor of the IEEE Trans. on Robotics and IEEE Trans. on Control Systems Technologies. He was vice-chair for Special Issues of the IEEE Technical Committee on Haptics (2006-2010); chair of the Italian Chapter of the IEEE RAS (2006-2010), awarded with the IEEE 2009 Chapter of the Year Award; and co-editor of two books by STAR, Springer Tracks in Advanced Robotics, Springer (2003, 2005). His research interests are in haptics, grasping, visual servoing, mobile robotics and geometric control. He has authored more than 200 papers in these fields.
Email: prattichizzo@diism.unisi.it



Prof. Katherine J. Kuchenbecker, University of Pennsylvania

Katherine received the B.S., M.S., and Ph.D. degrees in mechanical engineering from Stanford University, Stanford, CA, in 2000, 2002, and 2006, respectively. She completed a Postdoctoral Research Fellowship at the Johns Hopkins University, Baltimore, MD, in 2006–2007. She is currently an Associate Professor in Mechanical Engineering and Applied Mechanics at the University of Pennsylvania, Philadelphia. Her research centers on the design and control of haptic interfaces and robotic systems, and she directs the Penn Haptics Group, which is part of the General Robotics, Automation, Sensing, and Perception (GRASP) Laboratory. Prof. Kuchenbecker was the recipient of the 2009 National Science Foundation CAREER Award, the 2008 and 2011 Citations for Meritorious Service as a Reviewer for the IEEE Transactions on Haptics, and the 2012 IEEE Robotics and Automation Society Academic Early Career Award. She is co-chairing the IEEE Haptics Symposium in 2016 and 2018, and she is presently a co-chair of the IEEE RAS Technical Committee on Haptics.
Email: kuchenbe@seas.upenn.edu