track head missing

[finninmunich]

First of all, thank you for your excellent work.

In the paper, you mention that “we introduce a transformer-based motion head, which jointly processes 2D images and 3D points extracted from Gaussian maps”, and that this head can be used to generate per-pixel displacement vectors.

From the codebase, it seems that the TrackHead within the VGGT class corresponds to this motion head.

However, I could not find where this motion head is invoked in the inference pipeline. Am I missing some part of the code, or is this component only used during training?

[Ziyi-Xiong1]

Hi. Thank you for your feedback.
The TrackHead is used for interpolation-related logic. If you are only reconstructing the scene, this head is not involved. During interpolation, the TrackHead is required to track the point cloud as well as the objects.

[finninmunich]

Hi. Thank you for your feedback. The TrackHead is used for interpolation-related logic. If you are only reconstructing the scene, this head is not involved. During interpolation, the TrackHead is required to track the point cloud as well as the objects.

Thanks for your reply 🙂

In inference.py, I noticed that point cloud tracking during interpolation relies on a third-party model, TAPIP3D.

This makes me wonder about the motion head described in the paper (i.e., using learned queries to estimate per-pixel displacement vectors). I couldn’t find any place in the codebase where predictions["track"] is actually used.

Could you clarify whether the motion head is still part of the released pipeline, or if it has been replaced by TAPIP3D in the inference stage?

[Ziyi-Xiong1]

Thank you for your feedback.
In the current pipeline, tapip is used as the point-tracking head.