Convpass

Source code of "Convolutional Bypasses Are Better Vision Transformer Adapters"

Arxiv preprint 07/2022

Requirements

Python == 3.8

torch == 1.10.0

torchvision == 0.11.1

timm == 0.4.12

avalanche-lib == 0.1.0

Data Preparation

To download the datasets, please refer to https://github.com/ZhangYuanhan-AI/NOAH/#data-preparation (thank ZhangYuanhan-AI for their code). Then move the dataset folders to ./vtab/data/.

Usage

Pretrained Model

Download the pretrained ViT-B/16 to ./ViT-B_16.npz

Train & Save

cd vtab
bash run.sh

Only the trainable parameters are saved (1.4MB/task on average).

Test

cd vtab
python test.py --method convpass --dataset <dataset-name>
python test.py --method convpass_attn --dataset <dataset-name>

Performance on VTAB-1k (seed = 42)

Checkpoints can be found here. Extract models.zip to ./vtab/.

Citation

@inproceedings{jie2022convpass,
      title={Convolutional Bypasses Are Better Vision Transformer Adapters}, 
      author={Shibo Jie and Zhi-Hong Deng},
      year={2022},
      archivePrefix={arXiv},
}

Acknowledgments

Part of the code is borrowed from NOAH and timm.

Download Source Code