matting¶
Summary¶
Number of checkpoints: 9
Number of configs: 10
Number of papers: 3
ALGORITHM: 3
GCA (AAAI’2020)¶
Task: Matting
Abstract¶
Over the last few years, deep learning based approaches have achieved outstanding improvements in natural image matting. Many of these methods can generate visually plausible alpha estimations, but typically yield blurry structures or textures in the semitransparent area. This is due to the local ambiguity of transparent objects. One possible solution is to leverage the far-surrounding information to estimate the local opacity. Traditional affinity-based methods often suffer from the high computational complexity, which are not suitable for high resolution alpha estimation. Inspired by affinity-based method and the successes of contextual attention in inpainting, we develop a novel end-to-end approach for natural image matting with a guided contextual attention module, which is specifically designed for image matting. Guided contextual attention module directly propagates high-level opacity information globally based on the learned low-level affinity. The proposed method can mimic information flow of affinity-based methods and utilize rich features learned by deep neural networks simultaneously. Experiment results on Composition-1k testing set and this http URL benchmark dataset demonstrate that our method outperforms state-of-the-art approaches in natural image matting.
Results and models¶
| Model | Dataset | SAD | MSE | GRAD | CONN | Training Resources | Download |
|---|---|---|---|---|---|---|---|
| baseline (our) | Composition-1K | 34.61 | 0.0083 | 16.21 | 32.12 | 4 | model | log |
| GCA (our) | Composition-1K | 33.38 | 0.0081 | 14.96 | 30.59 | 4 | model | log |
| baseline (with DIM pipeline) | Composition-1K | 49.95 | 0.0144 | 30.21 | 49.67 | 4 | model | log |
| GCA (with DIM pipeline) | Composition-1K | 49.42 | 0.0129 | 28.07 | 49.47 | 4 | model | log |
Quick Start¶
Train
Train Instructions
You can use the following commands to train a model with cpu or single/multiple GPUs.
## cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/gca/gca_r34_4xb10-200k_comp1k.py
## single-gpu train
python tools/train.py configs/gca/gca_r34_4xb10-200k_comp1k.py
## multi-gpu train
./tools/dist_train.sh configs/gca/gca_r34_4xb10-200k_comp1k.py 8
For more details, you can refer to Train a model part in train_test.md.
Test
Test Instructions
You can use the following commands to test a model with cpu or single/multiple GPUs.
## cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth
## single-gpu test
python tools/test.py configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth
## multi-gpu test
./tools/dist_test.sh configs/gca/gca_r34_4xb10-200k_comp1k.py https://download.openmmlab.com/mmediting/mattors/gca/gca_r34_4x10_200k_comp1k_SAD-33.38_20220615-65595f39.pth 8
For more details, you can refer to Test a pre-trained model part in train_test.md.
Citation¶
@inproceedings{li2020natural,
title={Natural Image Matting via Guided Contextual Attention},
author={Li, Yaoyi and Lu, Hongtao},
booktitle={Association for the Advancement of Artificial Intelligence (AAAI)},
year={2020}
}
IndexNet (ICCV’2019)¶
Task: Matting
Abstract¶
We show that existing upsampling operators can be unified with the notion of the index function. This notion is inspired by an observation in the decoding process of deep image matting where indices-guided unpooling can recover boundary details much better than other upsampling operators such as bilinear interpolation. By looking at the indices as a function of the feature map, we introduce the concept of learning to index, and present a novel index-guided encoder-decoder framework where indices are self-learned adaptively from data and are used to guide the pooling and upsampling operators, without the need of supervision. At the core of this framework is a flexible network module, termed IndexNet, which dynamically predicts indices given an input. Due to its flexibility, IndexNet can be used as a plug-in applying to any off-the-shelf convolutional networks that have coupled downsampling and upsampling stages.
Results and models¶
| Model | Dataset | SAD | MSE | GRAD | CONN | Training Resources | Download |
|---|---|---|---|---|---|---|---|
| M2O DINs (our) | Composition-1k | 45.6 | 0.012 | 25.5 | 44.8 | 1 | model | log |
| M2O DINs (with DIM pipeline) | Composition-1k | 50.1 | 0.016 | 30.8 | 49.5 | 1 | model | log |
The performance of training (best performance) with different random seeds diverges in a large range. You may need to run several experiments for each setting to obtain the above performance.
Quick Start¶
Train
Train Instructions
You can use the following commands to train a model with cpu or single/multiple GPUs.
## cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py
## single-gpu train
python tools/train.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py
## multi-gpu train
./tools/dist_train.sh configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py 8
For more details, you can refer to Train a model part in train_test.md.
Test
Test Instructions
You can use the following commands to test a model with cpu or single/multiple GPUs.
## cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth
## single-gpu test
python tools/test.py configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth
## multi-gpu test
./tools/dist_test.sh configs/indexnet/indexnet_mobv2_1xb16-78k_comp1k.py https://download.openmmlab.com/mmediting/mattors/indexnet/indexnet_mobv2_1x16_78k_comp1k_SAD-45.6_20200618_173817-26dd258d.pth 8
For more details, you can refer to Test a pre-trained model part in train_test.md.
Citation¶
@inproceedings{hao2019indexnet,
title={Indices Matter: Learning to Index for Deep Image Matting},
author={Lu, Hao and Dai, Yutong and Shen, Chunhua and Xu, Songcen},
booktitle={Proc. IEEE/CVF International Conference on Computer Vision (ICCV)},
year={2019}
}
DIM (CVPR’2017)¶
Task: Matting
Abstract¶
Image matting is a fundamental computer vision problem and has many applications. Previous algorithms have poor performance when an image has similar foreground and background colors or complicated textures. The main reasons are prior methods 1) only use low-level features and 2) lack high-level context. In this paper, we propose a novel deep learning based algorithm that can tackle both these problems. Our deep model has two parts. The first part is a deep convolutional encoder-decoder network that takes an image and the corresponding trimap as inputs and predict the alpha matte of the image. The second part is a small convolutional network that refines the alpha matte predictions of the first network to have more accurate alpha values and sharper edges. In addition, we also create a large-scale image matting dataset including 49300 training images and 1000 testing images. We evaluate our algorithm on the image matting benchmark, our testing set, and a wide variety of real images. Experimental results clearly demonstrate the superiority of our algorithm over previous methods.
Results and models¶
| Model | Dataset | SAD | MSE | GRAD | CONN | Training Resources | Download |
|---|---|---|---|---|---|---|---|
| stage1 (our) | Composition-1k | 53.8 | 0.017 | 32.7 | 54.5 | 1 | model | log |
| stage2 (our) | Composition-1k | 52.3 | 0.016 | 29.4 | 52.4 | 1 | model | log |
| stage3 (our) | Composition-1k | 50.6 | 0.015 | 29.0 | 50.7 | 1 | model | log |
| stage1 (online merge) | Composition-1k | - | - | - | - | - | - |
NOTE
stage1: train the encoder-decoder part without the refinement part.
stage2: fix the encoder-decoder part and train the refinement part.
stage3: fine-tune the whole network.
The performance of the model is not stable during the training. Thus, the reported performance is not from the last checkpoint. Instead, it is the best performance of all validations during training.
The performance of training (best performance) with different random seeds diverges in a large range. You may need to run several experiments for each setting to obtain the above performance.
Quick Start¶
Train
Train Instructions
You can use the following commands to train a model with cpu or single/multiple GPUs.
DIM is trained with three stages.
Stage 1: train the encoder-decoder part without the refinement part.
## cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py
## single-gpu train
python tools/train.py configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py
## multi-gpu train
./tools/dist_train.sh configs/dim/dim_stage1-v16_1xb1-1000k_comp1k.py 8
Stage 2: fix the encoder-decoder part and train the refinement part.
## cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py
## single-gpu train
python tools/train.py configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py
## multi-gpu train
./tools/dist_train.sh configs/dim/dim_stage2-v16-pln_1xb1-1000k_comp1k.py 8
Stage 3: fine-tune the whole network.
## cpu train
CUDA_VISIBLE_DEVICES=-1 python tools/train.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py
## single-gpu train
python tools/train.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py
## multi-gpu train
./tools/dist_train.sh configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py 8
For more details, you can refer to Train a model part in train_test.md.
Test
Test Instructions
You can use the following commands to test a model with cpu or single/multiple GPUs.
## cpu test
CUDA_VISIBLE_DEVICES=-1 python tools/test.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth
## single-gpu test
python tools/test.py configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth
## multi-gpu test
./tools/dist_test.sh configs/dim/dim_stage3-v16-pln_1xb1-1000k_comp1k.py https://download.openmmlab.com/mmediting/mattors/dim/dim_stage3_v16_pln_1x1_1000k_comp1k_SAD-50.6_20200609_111851-647f24b6.pth 8
For more details, you can refer to Test a pre-trained model part in train_test.md.
Citation¶
@inproceedings{xu2017deep,
title={Deep image matting},
author={Xu, Ning and Price, Brian and Cohen, Scott and Huang, Thomas},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2970--2979},
year={2017}
}