SAM coming to video: SAM-track | | | | Turtles AI

SAM coming to video: SAM-track
  A team of researchers from the #ReLER #Lab at #Zhejiang #University's #College of #Computer #Science and #Technology have developed an open-source project called Segment and Track Anything (#SAM-Track) that can segment and track any object in videos using both automatic and interactive methods. The team includes @Wenguan Wang, @Yuanyou Xu, @Yangming Cheng, @Liulei Li, @Zongxin Yang, and @Yi Yang, and is supervised by the ReLER Lab at Zhejiang University's College of Computer Science and Technology. The SAM-Track pipeline uses the SAM (Segment Anything Models) algorithm for automatic/interactive key-frame segmentation and the #DeAOT (Decoupling features in Associating Objects with Transformers) algorithm for efficient multi-object tracking and propagation. The SAM-Track pipeline can detect and segment new objects dynamically and automatically using SAM, while DeAOT is responsible for tracking all identified objects. To showcase the capabilities of SAM-Track, the team has released a versatile demo video that demonstrates its segmentation and tracking capabilities in various scenarios, such as street views, AR, cells, animations, aerial shots, and more. In addition, the team has created a feature that enables users to interactively modify the mask for the initial video frame according to their needs. The interactive segmentation capabilities of Segment-and-Track-Anything are demonstrated in Demo1 and Demo2. Demo1 showcases SAM-Track's ability to interactively segment and track individual objects, while Demo2 showcases its ability to interactively add specified objects for tracking. To get started with SAM-Track, users must clone and rename the Segment-Anything and AOT-benchmark repositories, check the dependency requirements, and use the script to install the necessary libraries. Users can also download the default SAM and DeAOT/AOT models and run the demo.ipynb file to generate results. The team has also developed a user-friendly visual interface that allows users to easily obtain the results of their experiments. Users can upload the video directly on the UI and use Segtracker to track all objects within that video. The project is open-source, and licenses for borrowed code can be found in the file.