SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments
0
Sign in to get full access
Overview
- SurgTrack is a method for 3D tracking of real-world surgical instruments without requiring CAD models.
- It uses a Signed Distance Field (SDF) representation to model the 3D shape of instruments and a Posture Memory Pool (PMP) to efficiently store and retrieve instrument poses.
- SurgTrack achieves real-time performance by optimizing instrument poses using a Posture Graph.
Plain English Explanation
SurgTrack is a new technique that allows doctors to track the 3D position and orientation of surgical tools during an operation, without needing 3D computer models of the tools. Instead, it creates a special 3D representation of the tool shape called a Signed Distance Field, which can be quickly matched to the camera images to figure out the tool's pose.
To make this tracking efficient, SurgTrack also uses a Posture Memory Pool to store and efficiently retrieve previous poses of the tools. This allows it to quickly estimate the new pose of the tool based on how it has moved in the past. Finally, SurgTrack optimizes the tool pose using a Posture Graph, which helps it find the best match between the camera image and the 3D tool model in real-time.
The key advantage of SurgTrack is that it can track surgical tools without requiring detailed 3D models, which are often difficult to obtain. This makes it more practical to implement in real surgical settings. By providing accurate 3D tracking of tools, SurgTrack could help surgeons better understand the spatial relationships between instruments and the patient's anatomy during procedures.
Technical Explanation
SurgTrack uses a Signed Distance Field (SDF) to represent the 3D shape of surgical instruments without requiring CAD models. The SDF encodes the 3D geometry as a function that gives the distance to the nearest surface, with negative values inside the object and positive values outside.
To efficiently store and retrieve instrument poses, SurgTrack employs a Posture Memory Pool (PMP) that maintains a history of previously observed poses. This allows SurgTrack to quickly estimate the new pose of an instrument based on how it has moved in the past.
SurgTrack optimizes the instrument poses using a Posture Graph that represents the space of possible poses. By optimizing over this graph, SurgTrack can find the pose that best matches the camera observations in real-time.
The key technical innovations in SurgTrack are the use of the SDF representation, the PMP for efficient pose retrieval, and the Posture Graph optimization. Together, these components enable accurate 3D tracking of surgical instruments without requiring CAD models.
Critical Analysis
The authors acknowledge that SurgTrack has some limitations. For example, it may struggle to track instruments that undergo significant deformation during use. The paper also does not address how SurgTrack would handle occlusions or track multiple instruments simultaneously.
Additionally, the evaluation is primarily focused on tracking accuracy, with less emphasis on the real-world practicality and usability of the system. More research may be needed to understand how SurgTrack would perform in a live surgical setting and integrate with existing surgical workflows.
Overall, SurgTrack represents an interesting and promising approach to 3D instrument tracking. However, further development and validation would be needed to demonstrate its full potential for improving surgical procedures.
Conclusion
SurgTrack is a novel method for 3D tracking of surgical instruments without requiring CAD models. By using a Signed Distance Field representation, a Posture Memory Pool, and Posture Graph optimization, SurgTrack can achieve real-time performance and accurate pose estimation. This could be a valuable tool for improving the spatial awareness and efficiency of surgical procedures. While the research shows promise, additional work is needed to fully validate the system and address its current limitations.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
SurgTrack: CAD-Free 3D Tracking of Real-world Surgical Instruments
Wenwu Guo, Jinlin Wu, Zhen Chen, Qingxiang Zhao, Miao Xu, Zhen Lei, Hongbin Liu
Vision-based surgical navigation has received increasing attention due to its non-invasive, cost-effective, and flexible advantages. In particular, a critical element of the vision-based navigation system is tracking surgical instruments. Compared with 2D instrument tracking methods, 3D instrument tracking has broader value in clinical practice, but is also more challenging due to weak texture, occlusion, and lack of Computer-Aided Design (CAD) models for 3D registration. To solve these challenges, we propose the SurgTrack, a two-stage 3D instrument tracking method for CAD-free and robust real-world applications. In the first registration stage, we incorporate an Instrument Signed Distance Field (SDF) modeling the 3D representation of instruments, achieving CAD-freed 3D registration. Due to this, we can obtain the location and orientation of instruments in the 3D space by matching the video stream with the registered SDF model. In the second tracking stage, we devise a posture graph optimization module, leveraging the historical tracking results of the posture memory pool to optimize the tracking results and improve the occlusion robustness. Furthermore, we collect the Instrument3D dataset to comprehensively evaluate the 3D tracking of surgical instruments. The extensive experiments validate the superiority and scalability of our SurgTrack, by outperforming the state-of-the-arts with a remarkable improvement. The code and dataset are available at https://github.com/wenwucode/SurgTrack.
Read more9/5/2024
🗣️
0
New!SLAM assisted 3D tracking system for laparoscopic surgery
Jingwei Song, Ray Zhang, Wenwei Zhang, Hao Zhou, Maani Ghaffari
A major limitation of minimally invasive surgery is the difficulty in accurately locating the internal anatomical structures of the target organ due to the lack of tactile feedback and transparency. Augmented reality (AR) offers a promising solution to overcome this challenge. Numerous studies have shown that combining learning-based and geometric methods can achieve accurate preoperative and intraoperative data registration. This work proposes a real-time monocular 3D tracking algorithm for post-registration tasks. The ORB-SLAM2 framework is adopted and modified for prior-based 3D tracking. The primitive 3D shape is used for fast initialization of the monocular SLAM. A pseudo-segmentation strategy is employed to separate the target organ from the background for tracking purposes, and the geometric prior of the 3D shape is incorporated as an additional constraint in the pose graph. Experiments from in-vivo and ex-vivo tests demonstrate that the proposed 3D tracking system provides robust 3D tracking and effectively handles typical challenges such as fast motion, out-of-field-of-view scenarios, partial visibility, and organ-background relative motion.
Read more9/19/2024
0
SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos
Chinedu Innocent Nwoye, Nicolas Padoy
Accurate tool tracking is essential for the success of computer-assisted intervention. Previous efforts often modeled tool trajectories rigidly, overlooking the dynamic nature of surgical procedures, especially tracking scenarios like out-of-body and out-of-camera views. Addressing this limitation, the new CholecTrack20 dataset provides detailed labels that account for multiple tool trajectories in three perspectives: (1) intraoperative, (2) intracorporeal, and (3) visibility, representing the different types of temporal duration of tool tracks. These fine-grained labels enhance tracking flexibility but also increase the task complexity. Re-identifying tools after occlusion or re-insertion into the body remains challenging due to high visual similarity, especially among tools of the same category. This work recognizes the critical role of the tool operators in distinguishing tool track instances, especially those belonging to the same tool category. The operators' information are however not explicitly captured in surgical videos. We therefore propose SurgiTrack, a novel deep learning method that leverages YOLOv7 for precise tool detection and employs an attention mechanism to model the originating direction of the tools, as a proxy to their operators, for tool re-identification. To handle diverse tool trajectory perspectives, SurgiTrack employs a harmonizing bipartite matching graph, minimizing conflicts and ensuring accurate tool identity association. Experimental results on CholecTrack20 demonstrate SurgiTrack's effectiveness, outperforming baselines and state-of-the-art methods with real-time inference capability. This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries.
Read more5/31/2024
0
Monocular pose estimation of articulated surgical instruments in open surgery
Robert Spektor, Tom Friedman, Itay Or, Gil Bolotin, Shlomi Laufer
This work presents a novel approach to monocular 6D pose estimation of surgical instruments in open surgery, addressing challenges such as object articulations, symmetries, occlusions, and lack of annotated real-world data. The method leverages synthetic data generation and domain adaptation techniques to overcome these obstacles. The proposed approach consists of three main components: (1) synthetic data generation using 3D modeling of surgical tools with articulation rigging and physically-based rendering; (2) a tailored pose estimation framework combining object detection with pose estimation and a hybrid geometric fusion strategy; and (3) a training strategy that utilizes both synthetic and real unannotated data, employing domain adaptation on real video data using automatically generated pseudo-labels. Evaluations conducted on videos of open surgery demonstrate the good performance and real-world applicability of the proposed method, highlighting its potential for integration into medical augmented reality and robotic systems. The approach eliminates the need for extensive manual annotation of real surgical data.
Read more7/18/2024