Basic Information

This is the fusion stage training and inference of the VIPL-SLP submission for CV-ISLR challenge.

Data Preparation

Download the estimated keypoints, extracted RGB and depth features from Google Drive, and put them under the root of project:

• estimated keypoints
• extracted RGB features
• extracted depth features

Download the processed information dicts from Google Drive, and put them in the ./data

Feel free to contact us if the link is invalid.

Environment Configuration

conda env create -f environment.yml

Evaluation

Download the pretrained weights from Google Drive and put them in the ./weights

RGB Track

• For RGB data:

python main.py --config ./configs/test_single_rgb.yaml --load-weights weights/single_rgb.pt

Expected performance: Average Topk-1 : 34.55%

• For skeleton data:

python main.py --config ./configs/test_single_skeleton.yaml --load-weights weights/sk_phase2.pt

Expected performance: Average Topk-1 : 46.00%

• For RGB+Skeleton data:

python main.py --config ./configs/test_fusion_rgbd.yaml --load-weights ./weights/fusion_rgbd.pt

Expected performance: Average Topk-1 : 56.87%

RGB-D Track

• For Depth data:

python main.py --config ./configs/test_single_depth.yaml --load-weights ./weights/single_depth.pt

Expected performance: Average Topk-1 : 28.84%

• For RGB+Skeleton+Depth data:

python main.py --config ./configs/test_fusion_rgbd.yaml --load-weights ./weights/fusion_rgbd.pt

Expected performance: Average Topk-1 : 57.98%

Training

RGB Track

python main.py --config ./configs/train_fusion_rgb.yaml

RGB-D Track

python main.py --config ./configs/train_fusion_rgbd.yaml