SceneTracker: Long-term Scene Flow Estimation Network

Introduction

The paper introduces a new task called long-term scene flow estimation (LSFE), which aims to capture fine-grained and long-term 3D motion of objects in a scene. Existing methods like scene flow estimation (SFE) and 3D object tracking (3D OT) partially address this issue but have limitations in capturing long-term 3D trajectories or fine-grained deformations within objects.

The proposed method, SceneTracker, is the first LSFE method that takes an RGB-D video and initial 3D coordinates of a target as input and produces the target's 3D trajectory across the frames. SceneTracker adopts an iterative approach to approximate the optimal trajectory and overcome large displacements between frames. It dynamically constructs appearance and depth correlation features to enhance localization of the target's 3D position. The Transformer architecture is employed to explore long-range connections within and between trajectories, improving long-term estimation accuracy.

Compared to baseline methods, SceneTracker effectively addresses 3D occlusion and depth noise, achieving significant improvements in 3D error reduction. The key contributions are introducing the LSFE task, proposing the SceneTracker method for effective per-point 3D trajectory estimation within RGB-D videos, and validating the approach through qualitative and quantitative experiments.

Related Works

The paper discusses several tasks related to estimating motion and tracking points in videos and 3D scenes:

Optical Flow Estimation aims to estimate pixel-wise 2D motion between consecutive video frames. Methods like FlowNet, PWC-Net, GMFlow, and RAFT have tackled this using deep neural networks and techniques like iterative refinement and global matching.

Scene Flow Estimation estimates 3D motion and depth for each pixel across video frames. Approaches include operating directly on point clouds (FlowNet3D), coarse-to-fine refinement (CamLiFlow), and using optical expansion (OpticalExp). Handling occlusions is a key challenge addressed by methods like RigidMask and RAFT-3D.

Tracking Any Point focuses on tracking the global motion trajectory of points over multiple frames. Methods like PIPs, TAP-Net, MFT, and Cotracker use techniques like cost volumes, iterative inference, and modeling uncertainty.

The paper also discusses reconstruction-based offline methods like 3D-GS, Dynamic 3D Gaussians, and 4D-GS that model dynamic 3D scenes using Gaussian representations learned from multi-view videos. In contrast, the Live Scene Flow Estimation (LSFE) task tackled in this paper focuses on challenging scenarios with limited observations and real-time constraints.

Proposed Method

The paper describes an approach called SceneTracker for tracking 3D points throughout a 3D video sequence. The overall architecture involves dividing the video into overlapping sliding windows and iteratively updating the trajectories and template features for a set of query points within each window.

Key components include:

1. Feature Encoder and Downsampling: Extracts multi-scale image features and downsampled depth maps from the input RGB frames and depth maps.

2. Trajectory Initialization: Initializes the 3D trajectories for the query points in each sliding window based on the previous window's estimates.

3. Flow Iteration Module: Iteratively updates the template features and 3D trajectories by correlating them with the image features and depth maps using a transformer-based architecture. It constructs correlation features and motion features to model interactions.

4. Transformer Cell Network: Uses alternating cross-time and cross-space transformer blocks along with sinusoidal positional encodings to explore long-range dependencies within and between trajectories. Predicts residuals to update the trajectories and features.

5. Training: Supervises the network using ground truth 3D trajectories, computing a 3D loss that measures errors in predicted positions and inverse depths.

The method aims to handle large inter-frame motions by iteratively refining the trajectories within each sliding window in a coarse-to-fine manner, leveraging transformer architectures to capture long-range spatio-temporal dependencies.

Experiments

The paper describes the datasets, experimental setup, evaluation metrics, implementation details, and training processes used for the proposed long sequence flow estimation (LSFE) method called SceneTracker.

Datasets:

• LSFE lacks a dedicated dataset, so an augmented dataset called LSFOdyssey is constructed based on the PointOdyssey dataset for trajectory annotation prediction (TAP).
• LSFOdyssey has 127,437 training samples and 90 testing samples, with each sample containing a 24/40-frame 3D video and trajectories of 256 non-occluded query points.

Evaluation Metrics:

• 2D metrics: average 2D position accuracy, median 2D trajectory error, 2D survival rate.
• 3D metrics: percentages of points within distance thresholds, median/end-point 3D trajectory errors, 3D survival rate.

Implementation Details:

• PyTorch implementation, AdamW optimizer, one-cycle learning rate scheduler.
• Specific hyperparameter values for the network architecture.

Training Process:

• Odyssey process trains the model from scratch on LSFOdyssey with a 2e-4 learning rate for 200K steps.

The paper then visualizes SceneTracker's results on the test set and compares it against scene flow (SF) and TAP baselines, showing significant performance improvements. Ablation studies analyze the effects of the inference window size and number of updates in the iterative update module.

Conclusion

The paper proposes a new task called long-term scene flow estimation (LSFE) to better capture fine-grained and long-term 3D motion. To effectively tackle this task, the authors introduce a novel approach named SceneTracker. Experiments demonstrate that SceneTracker excels at handling 3D spatial occlusion and depth noise interference, making it well-suited for the LSFE task's requirements. The work proves the feasibility of simultaneously capturing fine-grained and long-term 3D motion, positioning LSFE as a promising direction for future research.