Code for paper "Self-Support Few-Shot Semantic Segmentation"

Code for paper "Self-Support Few-Shot Semantic Segmentation"
Abstract: Existing few-shot segmentation methods have achieved great progress based on the support-query matching framework. But they still heavily suffer from the limited coverage of intra-class variations from the few-shot supports provided. Motivated by the simple Gestalt principle that pixels belonging to the same object are more similar than those to different objects of same class, we propose a novel self-support matching strategy to alleviate this problem, which uses query prototypes to match query features, where the query prototypes are collected from high-confidence query predictions. This strategy can effectively capture the consistent underlying characteristics of the query objects, and thus fittingly match query features. We also propose an adaptive self-support background prototype generation module and self-support loss to further facilitate the self-support matching procedure. Our self-support network substantially improves the prototype quality, benefits more improvement from stronger backbones and more supports, and achieves SOTA on multiple datasets. Codes are at \url{this https URL}.

Self-Support Few-Shot Semantic Segmentation

Qi Fan, Wenjie Pei, Yu-Wing Tai, Chi-Keung Tang

The codebase contains the official code of our paper Self-Support Few-Shot Semantic Segmentation, ECCV 2022.

中文解读: ECCV 2022 | SSP: 自支持匹配的小样本任务新思想

PWC
PWC
PWC
PWC
PWC
PWC

Data preparation

Download

Pretrained model: ResNet-50 | ResNet-101

Dataset: Pascal images and ids | Semantic segmentation annotations

File Organization

You can follow HSNet to prepare the MS COCO and FSS-1000 datasets.

    ../                         # parent directory
    ├── ./SSP                   # current (project) directory
    |   ├── codes               # various codes
    |   └── ./pretrained        # pretrained model directory
    |            ├── resnet50.pth
    |            └── resnet101.pth
    └── Datasets_SSP/
        ├── VOC2012/            # PASCAL VOC2012 devkit
        │   ├── Annotations/
        │   ├── ImageSets/
        │   ├── ...
        │   └── SegmentationClassAug/
        ├── COCO2014/           
        │   ├── annotations/
        │   │   ├── train2014/  # (dir.) training masks (from Google Drive) 
        │   │   ├── val2014/    # (dir.) validation masks (from Google Drive)
        │   │   └── ..some json files..
        │   ├── train2014/
        │   └── val2014/
        └── FSS-1000/           # (dir.) contains 1000 object classes
            ├── abacus/   
            ├── ...
            └── zucchini/

Run the code

You can adapt the scripts of all.sh, test.sh and hsnet_test.sh (for the HSNet evaluation protocol) to train and evaluate your models.

CUDA_VISIBLE_DEVICES=0,1 python -W ignore main.py \
  --dataset pascal --data-root [Your Pascal Path] \
  --backbone resnet50 --fold 0 --shot 1

You may change the backbone from resnet50 to resnet101, change the fold from 0 to 1/2/3, or change the shot from 1 to 5 for other settings.

You can add/remove --refine to enable/disable the self-support refinement.

Performance and Trained Models

Pascal VOC

Method Setting Backbone SSP Refine Fold 0 Fold 1 Fold 2 Fold 3 Mean
Baseline 1-shot ResNet-50 No 54.9 66.5 61.7 48.3 57.9
Baseline 1-shot ResNet-101 No 57.2 68.5 61.3 53.3 60.1
Baseline 5-shot ResNet-50 No 61.6 70.3 70.5 56.4 64.7
Baseline 5-shot ResNet-101 No 64.2 74.0 71.5 61.3 67.8
SSP (Ours) 1-shot ResNet-50 Yes 61.4 67.8 66.5 50.9 61.7
SSP (Ours) 1-shot ResNet-101 Yes 63.2 70.4 68.5 56.3 64.6
SSP (Ours) 5-shot ResNet-50 Yes 67.5 72.3 75.2 62.1 69.3
SSP (Ours) 5-shot ResNet-101 Yes 70.9 77.1 78.9 66.1 73.3

MS COCO

Method Setting Backbone Eval Protocol Fold 0 Fold 1 Fold 2 Fold 3 Mean
SSP (Ours) 1-shot ResNet-50 Ours 46.4 35.2 27.3 25.4 33.6
SSP (Ours) 1-shot ResNet-101 Ours 50.4 39.9 30.6 30.0 37.7
SSP (Ours) 5-shot ResNet-50 Ours 53.9 42.0 36.0 33.7 41.4
SSP (Ours) 5-shot ResNet-101 Ours 57.8 47.0 40.2 39.9 46.2
SSP (Ours) 1-shot ResNet-50 HSNet 35.5 39.6 37.9 36.7 37.4
SSP (Ours) 1-shot ResNet-101 HSNet 39.1 45.1 42.7 41.2 42.0
SSP (Ours) 5-shot ResNet-50 HSNet 40.6 47.0 45.1 43.9 44.1
SSP (Ours) 5-shot ResNet-101 HSNet 47.4 54.5 50.4 49.6 50.2

Acknowledgement

This codebase is built based on MLC's baseline code and we borrow HSNet's evaluation protocol for the MS COCO dataset. We thank MLC and other FSS works for their great contributions.

Other related repos

Few-shot image/video object detection: FewX

Other related papers

@inproceedings{fan2021fsvod,
  title={Few-Shot Video Object Detection},
  author={Fan, Qi and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={ECCV},
  year={2022}
}
@inproceedings{fan2020cpmask,
  title={Commonality-Parsing Network across Shape and Appearance for Partially Supervised Instance Segmentation},
  author={Fan, Qi and Ke, Lei and Pei, Wenjie and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={ECCV},
  year={2020}
}
@inproceedings{fan2020fsod,
  title={Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector},
  author={Fan, Qi and Zhuo, Wei and Tang, Chi-Keung and Tai, Yu-Wing},
  booktitle={CVPR},
  year={2020}
}

Citation

@inproceedings{fan2022ssp,
  title={Self-Support Few-Shot Semantic Segmentation},
  author={Fan, Qi and Pei, Wenjie and Tai, Yu-Wing and Tang, Chi-Keung},
  journal={ECCV},
  year={2022}
}

Download Source Code

Download ZIP

Paper Preview

Aug 19, 2022

Related